Python crawler frame Scrapy 4-11 Distributed Data table design type and date converted by the date type str

On one talked about how to use the pipeline to save the item of data to json file, but the main thing is how to save data to the mysql. And before you save, we first need to make is the design of the database.

Database design here is relatively simple, because the data we need to save is the article, a table can be, and the fields and item fields in the same table. Now the database is relatively simple, we can use directly navicat be built form, the latter if the database is complicated, we can use the database migration django version of the management.

Create a database named Spider, the establishment of a cnblogs_article table, designed as follows:

Some settings is not null, we can also design for these fields below defaults.

create_date design is date type, but the type is str in the script, we need to convert it to a date type.

Introduced in cnblogs.py in:

import datetime

Before create_time into the item, we add the following logic:

try:
    create_date = datetime.datetime.strptime(create_date, "%Y-%m-%d").date()
except Exception as e:
    create_date = datetime.datetime.now().date()

Debugging, I found create_date become a date type.