Sesame HTTP: MySQL Storage

As long as your Scrapy Field field name is the same as the database field name. Then congratulations, you can copy this SQL splicing script. Perform MySQL warehousing processing.

The specific splicing code is as follows:

    def process_item(self, item, spider):
        if isinstance(item, WhoscoredNewItem):
            table_name = item.pop('table_name')
            col_str = ''
            row_str = ''
            for key in item.keys():
                col_str = col_str + " " + key + ","
                row_str = "{}'{}',".format(row_str, item[key] if "'" not in item[key] else item[key].replace("'", "\\'"))
                sql = "insert INTO {} ({}) VALUES ({}) ON DUPLICATE KEY UPDATE ".format(table_name, col_str[1:-1], row_str[:-1])
            for (key, value) in six.iteritems(item):
                sql += "{} = '{}', ".format(key, value if "'" not in value else value.replace("'", "\\'"))
            sql = sql[:-2]
            self.cursor.execute(sql) #Execute SQL
            self.cnx.commit()# write operation

 This SQL splicing is implemented. If the same data exists in the database, it will be updated, and the SQL statement inserted if it does not exist.

The specific implementation is the first for loop, get key as MySQL field name, VALUES as SQL VALUES (spliced ​​into an inserted SQL statement)

The second for loop implements the concatenation of field name = VALUES.

And the sql in the first for loop constitutes insert into XXXXX on duplicate key update. If it exists, update the SQL statement that does not exist or insert.