Scrapy the item and pipline

Item

Item Documents

Creating item

Create item in items.py, the examples are as follows:

import scrapy

class Product(scrapy.Item):
    name = scrapy.Field()
    price = scrapy.Field()

The item in the dictionary python api and similarly, Field as a dictionary pre-defined key
more item usage, please consult the documentation

Pipline

Pipline document

Creating Pipline

• The first step: writing pipline function in piplines.py in:

class Crawler58CityPipeline(object):

    def process_item(self, item, spider):  # 该方法负责处理spider中yield的item
        data = "【{}】{}".format(item['price'], item['title'])
        self.db.sadd(item.table, data)

• Step two: Enable written pipline function in the settings.py:

ITEM_PIPELINES = {
   'crawler_58city.pipelines.Crawler58CityPipeline': 300,
}

• The third step: yield item objects in the spider

item = HouseInfo(title=title, price=price)   # 定义好的item对象
yield item   # 返回后被pipline中的process_item方法处理