Sesame HTTP: Scrapy Tips - MySQL Storage

In the past two days, I took over from work, and the crawler left by others found a very interesting splicing of SQL scripts.

As long as your Scrapy Field field name is the same as the database field name. Then congratulations, you can copy this SQL splicing script. Perform MySQL warehousing processing.

The specific splicing code is as follows:

    def process_item(self, item, spider):
        if isinstance(item, WhoscoredNewItem):
            table_name = item.pop('table_name')
            col_str = ''
            row_str = ''
            for key in item.keys():
                col_str = col_str + " " + key + ","
                row_str = "{}'{}',".format(row_str, item[key] if "'" not in item[key] else item[key].replace("'", "\\'"))
                sql = "insert INTO {} ({}) VALUES ({}) ON DUPLICATE KEY UPDATE ".format(table_name, col_str[1:-1], row_str[:-1])
            for (key, value) in six.iteritems(item):
                sql += "{} = '{}', ".format(key, value if "'" not in value else value.replace("'", "\\'"))
            sql = sql[:-2]
            self.cursor.execute(sql) #Execute SQL
            self.cnx.commit()# write operation

This SQL splicing is implemented. If

the inserted statement)

The second for loop implements the concatenation of field name = VALUES.

And the sql in the first for loop constitutes insert into XXXXX on duplicate key update. If it exists, update the SQL statement that does not exist or insert.

I can only think of the little brother who wrote this splicing. Pretty generic.

I don't know if you have thought of this method, but I didn't.