White school Python reptiles (38): Reptile framework Scrapy entry basis (six) Item Pipeline

# -*- coding: utf-8 -*-
from scrapy import Spider, Request
from first_scrapy.items import ImageItem

class MziTuSpider(Spider):
    name = 'MziTuSpider'
    allowed_domains = ['www.mzitu.com']
    start_urls = ['https://www.mzitu.com/mm/']

    def start_requests(self):
        headers = {
            'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.117 Safari/537.36',
            'referer': 'https://www.mzitu.com/'
        }
        yield Request('https://www.mzitu.com/mm/', self.parse, headers = headers)

    def parse(self, response):
        imageList = response.css('.postlist ul li')
        for image in imageList:
            item = ImageItem()
            item['id'] = image.css('a::attr("href")').extract_first().split('/')[3]
            item['url'] = image.css('a::attr("href")').extract_first()
            item['title'] = image.css('a img::attr("alt")').extract_first()
            item['thumb'] = image.css('a img::attr("data-original")').extract_first()
            yield item

其次我们再创建一个 Item 用来提取信息，示例代码如下：

import scrapy

class ImageItem(scrapy.Item):
    collection = table = 'image'
    id = scrapy.Field()
    url = scrapy.Field()
    title = scrapy.Field()
    thumb = scrapy.Field()

在前面的示例中我们已经创建了一个 MongoPipeline 用来讲数据保存在 MongoDB 中，本次我们再加一个 MysqlPipeline 将数据保存在 Mysql 中，示例代码如下：

前提需本地正常安装 pymysql 。

import pymysql


class MysqlPipeline():
    def __init__(self, host, database, user, password, port):
        self.host = host
        self.database = database
        self.user = user
        self.password = password
        self.port = port

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            host=crawler.settings.get('MYSQL_HOST'),
            database=crawler.settings.get('MYSQL_DATABASE'),
            user=crawler.settings.get('MYSQL_USER'),
            password=crawler.settings.get('MYSQL_PASSWORD'),
            port=crawler.settings.get('MYSQL_PORT'),
        )

    def open_spider(self, spider):
        self.db = pymysql.connect(self.host, self.user, self.password, self.database, charset='utf8',
                                  port=self.port)
        self.cursor = self.db.cursor()

    def close_spider(self, spider):
        self.db.close()

    def process_item(self, item, spider):
        print(item['title'])
        data = dict(item)
        keys = ', '.join(data.keys())
        values = ', '.join(['%s'] * len(data))
        sql = 'insert into %s (%s) values (%s)' % (item.table, keys, values)
        self.cursor.execute(sql, tuple(data.values()))
        self.db.commit()
        return item

这里和前面一样，我们需要在 settings 中配置一些有关 Mysql 的内容，如下：

MYSQL_HOST = 'localhost'
MYSQL_DATABASE = 'test'
MYSQL_USER = 'root'
MYSQL_PASSWORD = '123456'
MYSQL_PORT = 3306

还需在 settings 中增加我们的 MysqlPipeline 的相关配置，如下：

ITEM_PIPELINES = {
    'first_scrapy.pipelines.MongoPipeline': 400,
    'first_scrapy.pipelines.MysqlPipeline': 401,
}

这样，我们就创建好了我们的示例程序，现在通过命令行来启动我们的 Spider 。

scrapy crawl MziTuSpider

执行结果：

可以看到，我们的 MongoDB 和 Mysql 数据都正常的存入。

示例代码

本系列的所有代码小编都会放在代码管理仓库 Github 和 Gitee 上，方便大家取用。

示例代码-Github

示例代码-Gitee

White school Python reptiles (38): Reptile framework Scrapy entry basis (six) Item Pipeline

引言

自定义 Item Pipeline

process_item(self, item, spider)

open_spider(self, spider)

close_spider(self, spider)

from_crawler(cls, crawler)

示例

示例代码

Guess you like