In-depth understanding of Python Scrapy - [First introduction to Scrapy]

Why Choose Scrapy

Before I start explaining Scrapy, let me first talk to you about why you chose it Scrapy. In the course of a year of crawler development, I have encountered problems large and small, such as:

• The web page/interface is unstable and causes timeout.
• The developed crawler has low performance and usually needs to start multiple processes to improve performance.
• The structures and styles of crawler programs developed by different engineers are uneven and difficult to manage and maintain.
• Program coupling is too high and difficult to expand
• Additional tools such as random UA, random agents, and logging need to be implemented by yourself, which increases the workload.
• …

In response to the above problems, we choose to use Scrapy.

What is Scrapy

An open source and collaborative framework for extracting the data you need from websites.
In a fast, simple, yet extensible way.

ScrapyPythonIt is a fast, simple and powerful web crawler framework suitable for web crawlers. It is usually used to crawl web sites and extract structured data from pages. It can also be used for monitoring and automated testing . The architecture diagram is as follows:

How Scrapy works

Understanding the working principle is more conducive to subsequent learning (you can also read the quick start first and then come back to read here). The Scrapyoperation flow chart is as follows:

The running process is as follows:

1. After the program is started, one/more will be created Spiders（爬虫）, processed and then handed over .SpidersRequests（请求）SpiderMiddlewares（爬虫中间件）Engine（引擎）
2. EngineMultiple requests will Spidersbe forwarded to Scheduler（调度器）the scheduler to arrange the requests.
3. SchedulerReturn requests that need to be executed immediately Engine.
4. EngineProcess the request DownloaderMiddlewares（下载器中间件）and then send it to Downloader（下载器）.
5. DownloaderUse Requeststhe completed page/interface to download and generate it Responses（响应）, and then transfer the process to Responsesit .DownloaderMiddlewaresEngine
6. EngineHand the process Responsesback SpiderMiddlewaresto the crawler for processing Responses.
7. SpidersResponsesThe result produced after processing is returned to Engine. (Spiders processing Responses)
8. 步骤7SpidersThe object returned by the processing result Requestswill be returned 步骤2; the returned Items（数据结构化对象）or dict（字典对象）will be handed over to ItemPipelines（数据管道）processing.
9. ItemPipelinesControl how data is persisted and processed through customization .

Get started with Scrapy

1. Install Scrapy

Install via the following command Scrapy:

pip install scrapy

ScrapyAfter the installation is completed, a will be provided scrapy工具. If the command scrapy --helpdisplays the following, the installation is successful:

> scrapy --help
Scrapy 2.6.2 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  commands
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

2. Create scrapy project

scrapy startproject xxxCreate a project with the command Scrapy:

scrapy startproject MySpider

After the command is executed, a directory will be generated in the current directory MySpider. The directory structure is as follows:

MySpider/
├─scrapy.cfg
└─MySpider/
  ├─items.py
  ├─middlewares.py
  ├─pipelines.py
  ├─settings.py
  ├─__init__.py
  └─spiders/
    └─__init__.py

• items.pyFile storage customizedItems
• middlewares.pyfile storage SpiderMiddlewaresandDownloaderMiddlewares
• pipelines.pyFile storage customizedItemPipelines
• settings.pyFile stores global configuration information
• spiders/Directory stores allSpiders

Then use the first MySpider/directory as the project root directory

3. Create Scrapy crawler

Create a Scrapy crawler command. scrapy genspider [spidername] [allow_domain]Take the Manmanbuy historical price interface as an example to create a Manmanbuy crawler:

scrapy genspider manmanbuy manmanbuy.com

Slowly buy historical price crawling process:

1. Visit the http://tool.manmanbuy.com/HistoryLowest.aspx page to obtain the value of the hidden <input id="ticket" …> tag.
2. Process the value obtained in step 1 to generate the Authorization parameter of the request header.
3. Generate the value of the request parameter token
4. Call the http://tool.manmanbuy.com/api.ashx interface to obtain the historical price of the product (this interface relies on valid cookies, how to obtain valid cookies is not the focus of this article and will not be explained yet)

At this time, the spiders/generated manmanbuy.pyfile can be found in the directory. The content of the file is as follows:

import scrapy


class ManmanbuySpider(scrapy.Spider):
    name = 'manmanbuy'
    allowed_domains = ['manmanbuy.com']
    start_urls = ['http://manmanbuy.com/']

    def parse(self, response):
        pass

Among them nameis the name of the crawler, allowed_domainsthe domain name allowed to be accessed, and start_urlsthe address to start crawling.

SpiderSupports two ways to start crawling, one is a convenient configuration start_urlsmethod, which will crawl the configuration directly after startup url, and the other is a rewrite start_requestsmethod that returns custom initializationRequest

4.Scrapy crawler development

4.1. Writing Spiders（MySpider/spiders/manmanbuy.py）:

import scrapy
from urllib.parse import quote
import hashlib
import time
import copy


class ManmanbuySpider(scrapy.Spider):
    name = 'manmanbuy'
    allowed_domains = ['manmanbuy.com']

    def start_requests(self):
        # 以京东单个商品查询历史价格为例, 商品ID: 100011493273
        item_urls = ['https://item.jd.com/100011493273.html']
        # 定义请求头
        headers = {
    
    
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
        }
        # 第一步先从h5页面获取ticket参数
        for item_url in item_urls:
            yield scrapy.Request(url='http://tool.manmanbuy.com/HistoryLowest.aspx?url=' + item_url,
                                 headers=headers,
                                 # 透传参数
                                 meta={
    
    'key': item_url})

    def parse(self, response: scrapy.http.Response):
        # 从页面中获取ticket值
        ticket = response.css('#ticket')[0].attrib['value']
        # 获取下一段接口请求参数
        req = parse_req({
    
    'key': response.meta['key'], 'method': 'getHistoryTrend'})
        # 请求头
        headers = {
    
    
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
            # 计算auth
            'Authorization': parse_basic_auth(ticket),
        }
        return scrapy.FormRequest(url='http://tool.manmanbuy.com/api.ashx',
                                  method='POST',
                                  formdata=req,
                                  headers=headers,
                                  cookies=self.get_cookies(),
                                  # 自定义回调地址
                                  callback=self.parse_history_price)

    def get_cookies(self):
        # 省略获取cookie逻辑
        cks = '_ga=GA1.2.604426644.1596510819; ASP.NET_SessionId=bbyuxdftfkcf5mrijdgkmnc5; Hm_lvt_01a310dc95b71311522403c3237671ae=1658906329; Hm_lvt_85f48cee3e51cd48eaba80781b243db3=1658748396,1658906330; _gid=GA1.2.472137414.1658906330; _gat_gtag_UA_145348783_1=1; 60014_mmbuser=VQYJA1IFBTBSVwNdClFVUgdRUQcLUgdXDg1RBgNTAVVUVAZRBQFeAw%3d%3d; Hm_lpvt_85f48cee3e51cd48eaba80781b243db3=1658906625; Hm_lpvt_01a310dc95b71311522403c3237671ae=1658906625'
        cookies = {
    
    }
        for one in cks.split(';'):
            k, v = one.strip().split("=")
            cookies[k] = v
        return cookies

    def parse_history_price(self, response: scrapy.http.Response):
        # 输出相应
        self.logger.info(response.text)


def parse_basic_auth(ticket):
    """
    这是解析ticket的值啊，就是上面说的那逻辑
    """
    return 'BasicAuth ' + ticket[:160][-4:] + ticket[:160 - 4]


def parse_req(d):
    """
    这是解析请求,增加t和token参数
    """
    d['t'] = str(int(time.time() * 1000))
    n = copy.deepcopy(d)
    ks = list(n.keys())
    ks.sort()
    ask = 'c5c3f201a8e8fc634d37a766a0299218'
    mask = ask
    for k in ks:
        mask += f'{
      
      k}{
      
      quote(str(n[k])).replace("/", "%2F")}'
    mask += ask
    mask = mask.upper()
    md5 = hashlib.md5()
    md5.update(mask.encode('utf-8'))
    d['token'] = md5.hexdigest().upper()
    return d

4.2. Modification Settings（MySpider/settings.py）:

# robots.txt 文件检查, 默认为: true, 需要改为Flase
ROBOTSTXT_OBEY = False

Run scrapy crawl manmanbuythe command to start the crawler and observe the log to obtain data normally:

{
    
    "msg":"","code":0,"data":{
    
    "haveTrend":1,"changPriceRemark":"降幅5%","runtime":41,"zouShi_test":2,"changePriceCount":14,"spbh":"1|100011493273","spUrl":"https://item.jd.com/100011493273.html","spPic":"http://img13.360buyimg.com/n7/jfs/t1/201578/31/15673/77560/619479ceEd1bde507/c0dab826b71e0b84.jpg","currentPrice":1049.00,"spName":"荣耀Play5T 22.5W超级快充 5000mAh大电池 6.5英寸护眼屏 全网通8GB+128GB极光蓝","lowerDate":"2022-03-08T00:00:00+08:00","lowerPrice":899.00,"bjid":551120462,"zouShi":2,"siteId":1,"siteName":"京东商城","datePrice":"[1621353600000,1199.00,\"\"],[1621440000000,1199.00,\"\"],[1621526400000,1199.00,\"\"],[1621612800000,1199.00,\"\"],[1621699200000,1199.00,\"\"],[1621785600000,1199.00,\"\"],[1621872000000,1199.00,\"\"],[1621958400000,1199.00,\"\"],[1622044800000,1199.00,\"\"],[1622131200000,1199.00,\"\"],[1622217600000,1199.00,\"\"],[1622304000000,1199.00,\"\"],[1622390400000,1199.00,\"\"],[1622476800000,1199.00,\"\"],[1622563200000,1199.00,\"\"],[1622649600000,1199.00,\"\"],[1622736000000,1199.00,\"\"],[1622822400000,1199.00,\"\"],[1622908800000,1199.00,\"\"],[1622995200000,1199.00,\"\"],[1623081600000,1199.00,\"\"],[1623168000000,1199.00,\"\"],[1623254400000,1199.00,\"\"],[1623340800000,1199.00,\"1199元\"],[1623427200000,1199.00,\"\"],[1623513600000,1199.00,\"\"],[1623600000000,1199.00,\"\"],[1623686400000,1199.00,\"\"],[1623772800000,1199.00,\"\"],[1623859200000,1199.00,\"\"],[1623945600000,1099.00,\"购买1件,当前价:1199.00,满减：每满1180减100\"],[1624032000000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624118400000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624204800000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624291200000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624377600000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624464000000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624550400000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624636800000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624723200000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624809600000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624896000000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1624982400000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1625068800000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1625155200000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1625241600000,1139.00,\"购买1件,当前价:1199.00,可叠加优惠券2：满750减60\"],[1625328000000,1199.00,\"\"],[1625414400000,1199.00,\"\"],[1625500800000,1189.0,\"京东秒杀价:1189\"],[1625587200000,1199.00,\"\"],[1625673600000,1189.0,\"\"],[1625760000000,1199.0,\"\"],[1625846400000,1189.0,\"\"],[1625932800000,1199.0000,\"\"],[1626019200000,1199.0000,\"\"],[1626105600000,1189.0,\"\"],[1626192000000,1199.0,\"\"],[1626278400000,1189.0,\"\"],[1626364800000,1199.0000,\"\"],[1626451200000,1199.0000,\"\"],[1626537600000,1199.0000,\"\"],[1626624000000,1199.0000,\"\"],[1626710400000,1199.0000,\"\"],[1626796800000,1199.0000,\"\"],[1626883200000,1189.00,\"\"],[1626969600000,1199.0000,\"\"],[1627056000000,1199.0000,\"\"],[1627142400000,1199.0000,\"\"],[1627228800000,1199.0000,\"\"],[1627315200000,1189.00,\"\"],[1627401600000,1199.0000,\"\"],[1627488000000,1189.00,\"1189元\"],[1627574400000,1189.00,\"\"],[1627660800000,1199.00,\"\"],[1627747200000,1179.00,\"1179元\"],[1627833600000,1189.0000,\"\"],[1627920000000,1199.00,\"\"],[1628006400000,1199.00,\"\"],[1628092800000,1189.00,\"\"],[1628179200000,1189.0000,\"\"],[1628265600000,1189.0000,\"\"],[1628352000000,1189.0000,\"\"],[1628438400000,1199.00,\"\"],[1628524800000,1189.00,\"\"],[1628611200000,1199.0,\"\"],[1628697600000,1189.0000,\"\"],[1628784000000,1189.0000,\"\"],[1628870400000,1189.0000,\"\"],[1628956800000,1199.00,\"1199元\"],[1629043200000,1189.0000,\"\"],[1629129600000,1199.00,\"\"],[1629216000000,1189.00,\"\"],[1629302400000,1199.0000,\"\"],[1629388800000,1169.0,\"京东秒杀价:1169\"],[1629475200000,1199.00,\"\"],[1629561600000,1199.00,\"\"],[1629648000000,1199.00,\"\"],[1629734400000,1169.00,\"\"],[1629820800000,1199.0,\"\"],[1629907200000,1189.0,\"京东秒杀价:1189\"],[1629993600000,1199.00,\"\"],[1630080000000,1199.00,\"\"],[1630166400000,1199.00,\"\"],[1630252800000,1199.00,\"\"],[1630339200000,1189.00,\"1189元包邮\"],[1630425600000,1189.00,\"\"],[1630512000000,1175.00,\"购买1件,plus价格1175\"],[1630598400000,1189.00,\"\"],[1630684800000,1199.0,\"\"],[1630771200000,1189.0000,\"\"],[1630857600000,1199.0,\"\"],[1630944000000,1189.0000,\"\"],[1631030400000,1199.0,\"\"],[1631116800000,1099.00,\"购买1件,当前价:1199.00,满减：每满1180减100\"],[1631203200000,1189.0000,\"\"],[1631289600000,1189.00,\"\"],[1631376000000,1199.00,\"\"],[1631462400000,1189.0000,\"\"],[1631548800000,1189.0,\"\"],[1631635200000,1199.0,\"\"],[1631721600000,1189.00,\"\"],[1631808000000,1199.00,\"\"],[1631894400000,1189.0,\"\"],[1631980800000,1199.0,\"\"],[1632067200000,1169.00,\"\"],[1632153600000,1169.00,\"\"],[1632240000000,1169.00,\"\"],[1632326400000,1189.0,\"\"],[1632412800000,1169.0,\"\"],[1632499200000,1199.0,\"\"],[1632585600000,1169.00,\"\"],[1632672000000,1199.00,\"\"],[1632758400000,1169.0,\"\"],[1632844800000,1199.0000,\"\"],[1632931200000,1169.0,\"\"],[1633017600000,1169.0,\"\"],[1633104000000,1169.0,\"\"],[1633190400000,1169.00,\"\"],[1633276800000,1169.0000,\"\"],[1633363200000,1199.00,\"\"],[1633449600000,1199.00,\"\"],[1633536000000,1169.0,\"\"],[1633622400000,1169.0,\"\"],[1633708800000,1169.0,\"京东秒杀价:1169\"],[1633795200000,1169.0,\"\"],[1633881600000,1199.00,\"\"],[1633968000000,1169.00,\"\"],[1634054400000,1199.0,\"\"],[1634140800000,1169.00,\"\"],[1634227200000,1199.0,\"\"],[1634313600000,1199.0,\"\"],[1634400000000,1169.0,\"\"],[1634486400000,1199.0,\"\"],[1634572800000,1169.00,\"\"],[1634659200000,1199.0000,\"\"],[1634745600000,1169.00,\"\"],[1634832000000,1199.0,\"\"],[1634918400000,1199.0,\"\"],[1635004800000,1199.0,\"\"],[1635091200000,1199.0,\"\"],[1635177600000,1199.0,\"\"],[1635264000000,1199.0,\"\"],[1635350400000,1199.0,\"\"],[1635436800000,1199.0,\"\"],[1635523200000,1099.00,\"1099元 \"],[1635609600000,1099.0,\"\"],[1635696000000,1099.00,\"\"],[1635782400000,1099.00,\"\"],[1635868800000,1099.00,\"\"],[1635955200000,1099.00,\"购买1件,plus价格1099\"],[1636041600000,949.00,\"购买1件,当前价:1099.00,可叠加优惠券2：满880减150\"],[1636128000000,1099.00,\"\"],[1636214400000,1199.0,\"\"],[1636300800000,1099.0,\"\"],[1636387200000,1099.0,\"\"],[1636473600000,1099.0,\"\"],[1636560000000,979.00,\"购买1件,当前价:1099.00,满减：每满1080减120\"],[1636646400000,1099.0000,\"\"],[1636732800000,1199.00,\"\"],[1636819200000,1199.00,\"\"],[1636905600000,1199.00,\"\"],[1636992000000,1199.00,\"\"],[1637078400000,1099.00,\"\"],[1637164800000,1099.00,\"\"],[1637251200000,1099.00,\"\"],[1637337600000,1099.00,\"\"],[1637424000000,1099.00,\"1099元\"],[1637510400000,1099.00,\"\"],[1637596800000,1099.0,\"\"],[1637683200000,1099.00,\"\"],[1637769600000,1099.00,\"\"],[1637856000000,1099.00,\"\"],[1637942400000,1099.00,\"\"],[1638028800000,1199.0000,\"\"],[1638115200000,1199.00,\"\"],[1638201600000,1199.00,\"\"],[1638288000000,1099.0,\"\"],[1638374400000,1099.00,\"\"],[1638460800000,1099.00,\"\"],[1638547200000,1199.0,\"\"],[1638633600000,1199.00,\"\"],[1638720000000,1099.0,\"\"],[1638806400000,1099.00,\"\"],[1638892800000,1099.00,\"\"],[1638979200000,1099.00,\"1099元\"],[1639065600000,1099.00,\"\"],[1639152000000,1089.0,\"\"],[1639238400000,1089.0,\"\"],[1639324800000,1099.00,\"购买1件,当前价:1199.00,满减：满1150减100\"],[1639411200000,1099.00,\"购买1件,当前价:1199.00,满减：满1150减100\"],[1639497600000,1099.00,\"购买1件,当前价:1199.00,满减：满1150减100\"],[1639584000000,1099.00,\"\"],[1639670400000,1099.0,\"\"],[1639756800000,1099.0,\"\"],[1639843200000,1099.0,\"\"],[1639929600000,1099.00,\"1099元\"],[1640016000000,1099.0,\"\"],[1640102400000,1099.0,\"\"],[1640188800000,1099.0,\"\"],[1640275200000,1099.00,\"1099元\"],[1640361600000,1099.0,\"\"],[1640448000000,1069.0,\"\"],[1640534400000,1099.0,\"\"],[1640620800000,1099.0,\"\"],[1640707200000,1099.0,\"\"],[1640793600000,1099.0,\"\"],[1640880000000,1099.0,\"\"],[1640966400000,1099.00,\"1099元\"],[1641052800000,1099.0,\"\"],[1641139200000,1099.0,\"\"],[1641225600000,1099.0,\"\"],[1641312000000,1099.0,\"\"],[1641398400000,1099.0,\"\"],[1641484800000,1099.0,\"\"],[1641571200000,1099.0,\"\"],[1641657600000,1099.00,\"1099元\"],[1641744000000,1099.0,\"\"],[1641830400000,1099.0,\"\"],[1641916800000,1099.0,\"\"],[1642003200000,1099.0,\"\"],[1642089600000,1099.0,\"\"],[1642176000000,1099.0,\"\"],[1642262400000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1642348800000,1099.00,\"\"],[1642435200000,1099.00,\"\"],[1642521600000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1642608000000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1642694400000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1642780800000,1099.0,\"\"],[1642867200000,1099.0,\"\"],[1642953600000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1643040000000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1643126400000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1643212800000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1643299200000,1099.0,\"\"],[1643385600000,1099.0,\"\"],[1643472000000,1099.0,\"\"],[1643558400000,949.00,\"购买1件,当前价:1099.00,满减：满1000减50,可叠加优惠券2：满880减100\"],[1643644800000,949.00,\"购买1件,当前价:1099.00,满减：满1000减50,可叠加优惠券2：满880减100\"],[1643731200000,1099.0,\"\"],[1643817600000,1099.0,\"\"],[1643904000000,1099.0,\"\"],[1643990400000,1099.0,\"\"],[1644076800000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1644163200000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1644249600000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1644336000000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1644422400000,1099.00,\"\"],[1644508800000,1099.00,\"1099元\"],[1644595200000,1199.00,\"1199元\"],[1644681600000,1199.00,\"1089元\"],[1644768000000,1039.00,\"购买1件,当前价:1089.00,满减：满1000减50\"],[1644854400000,1039.00,\"购买1件,当前价:1089.00,满减：满1000减50\"],[1644940800000,1099.00,\"1049元\"],[1645027200000,1099.00,\"\"],[1645113600000,1099.00,\"\"],[1645200000000,1099.00,\"\"],[1645286400000,1099.00,\"\"],[1645372800000,1099.00,\"\"],[1645459200000,1099.00,\"\"],[1645545600000,1099.00,\"\"],[1645632000000,1099.00,\"\"],[1645718400000,1099.00,\"1099元\"],[1645804800000,1099.00,\"1069元\"],[1645891200000,1049.00,\"购买1件,当前价:1099.00,满减：满1000减50\"],[1645977600000,1099.00,\"\"],[1646064000000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1646150400000,1099.00,\"\"],[1646236800000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1646323200000,1099.00,\"\"],[1646409600000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1646496000000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1646582400000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1646668800000,899.00,\"购买1件,当前价:1099.00,满减：每满1080减200\"],[1646755200000,1099.00,\"\"],[1646841600000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1646928000000,1099.0,\"\"],[1647014400000,1099.0,\"\"],[1647100800000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1647187200000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1647273600000,1099.00,\"\"],[1647360000000,1099.00,\"1099元\"],[1647446400000,1049.0,\"购买1件,当前价:1099.0,满减：满1050减50\"],[1647532800000,1099.0,\"\"],[1647619200000,1099.00,\"\"],[1647705600000,1099.00,\"\"],[1647792000000,1099.00,\"\"],[1647878400000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1647964800000,1099.00,\"\"],[1648051200000,1099.00,\"\"],[1648137600000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1648224000000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1648310400000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1648396800000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1648483200000,1039.00,\"购买1件,当前价:1089.00,满减：满1050减50\"],[1648569600000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1648656000000,1049.00,\"购买1件,当前价:1099.00,满减：满1050减50\"],[1648742400000,1049.00,\"\"],[1648828800000,1049.00,\"\"],[1648915200000,1049.00,\"1049元\"],[1649001600000,1099.00,\"\"],[1649088000000,1049.00,\"\"],[1649174400000,1049.00,\"\"],[1649260800000,1049.00,\"\"],[1649347200000,1049.00,\"\"],[1649433600000,1099.00,\"\"],[1649520000000,1099.00,\"\"],[1649606400000,1099.00,\"\"],[1649692800000,1049.0,\"购买1件,当前价格1049\"],[1649779200000,1099.00,\"\"],[1649865600000,1099.00,\"\"],[1649952000000,1049.00,\"1049元\"],[1650038400000,1099.00,\"\"],[1650124800000,1049.00,\"\"],[1650211200000,1049.00,\"\"],[1650297600000,1049.00,\"\"],[1650384000000,1049.00,\"\"],[1650470400000,1099.00,\"1099元\"],[1650556800000,1099.00,\"\"],[1650643200000,1099.00,\"\"],[1650729600000,1099.00,\"\"],[1650816000000,1099.00,\"\"],[1650902400000,1099.00,\"\"],[1650988800000,1099.00,\"\"],[1651075200000,1099.00,\"1099元\"],[1651161600000,1099.00,\"\"],[1651248000000,1099.00,\"\"],[1651334400000,1049.00,\"\"],[1651420800000,1099.00,\"\"],[1651507200000,1099.0000,\"\"],[1651593600000,1099.0000,\"\"],[1651680000000,1099.00,\"1099元\"],[1651766400000,1089.0,\"购买1件,当前价格1089\"],[1651852800000,1049.00,\"\"],[1651939200000,1089.00,\"\"],[1652025600000,1099.00,\"1099元\"],[1652112000000,1089.00,\"\"],[1652198400000,1049.00,\"\"],[1652284800000,1089.00,\"\"],[1652371200000,1049.00,\"\"],[1652457600000,1099.00,\"\"],[1652544000000,1099.00,\"\"],[1652630400000,1099.00,\"\"],[1652716800000,1099.00,\"1099元\"],[1652803200000,1099.00,\"\"],[1652889600000,1099.00,\"\"],[1652976000000,1049.00,\"\"],[1653062400000,1099.00,\"\"],[1653148800000,1099.00,\"\"],[1653235200000,1099.00,\"1099元\"],[1653321600000,1099.00,\"\"],[1653408000000,1099.00,\"\"],[1653494400000,1099.00,\"\"],[1653580800000,1099.00,\"\"],[1653667200000,1099.00,\"\"],[1653753600000,1099.00,\"\"],[1653840000000,1099.00,\"\"],[1653926400000,1049.0,\"\"],[1654012800000,1049.00,\"\"],[1654099200000,1049.00,\"\"],[1654185600000,1049.00,\"\"],[1654272000000,1049.00,\"\"],[1654358400000,1049.00,\"\"],[1654444800000,1049.00,\"\"],[1654531200000,1049.00,\"\"],[1654617600000,1049.00,\"\"],[1654704000000,1049.00,\"\"],[1654790400000,1049.00,\"\"],[1654876800000,1049.00,\"\"],[1654963200000,1049.00,\"\"],[1655049600000,1049.00,\"\"],[1655136000000,1049.00,\"\"],[1655222400000,1049.00,\"\"],[1655308800000,1049.00,\"1049元\"],[1655395200000,1049.00,\"\"],[1655481600000,1049.00,\"\"],[1655568000000,1049.00,\"\"],[1655654400000,1049.00,\"\"],[1655740800000,1049.00,\"\"],[1655827200000,1049.00,\"1049元\"],[1655913600000,1049.00,\"\"],[1656000000000,1049.00,\"\"],[1656086400000,1049.00,\"\"],[1656172800000,1049.00,\"1049元\"],[1656259200000,1049.00,\"\"],[1656345600000,1049.00,\"\"],[1656432000000,1049.00,\"\"],[1656518400000,1049.00,\"\"],[1656604800000,1049.00,\"1049元\"],[1656691200000,1049.00,\"\"],[1656777600000,1049.00,\"\"],[1656864000000,1049.00,\"1049元\"],[1656950400000,1049.00,\"\"],[1657036800000,1049.00,\"\"],[1657123200000,1049.00,\"\"],[1657209600000,1049.00,\"\"],[1657296000000,1049.00,\"\"],[1657382400000,1049.00,\"\"],[1657468800000,1049.00,\"\"],[1657555200000,1049.00,\"\"],[1657641600000,1049.00,\"\"],[1657728000000,1049.00,\"\"],[1657814400000,1049.00,\"\"],[1657900800000,1049.00,\"1049元\"],[1657987200000,1049.00,\"\"],[1658073600000,1049.00,\"\"],[1658160000000,1049.00,\"\"],[1658246400000,1049.00,\"\"],[1658332800000,1049.00,\"1049元\"],[1658419200000,1049.00,\"\"],[1658505600000,1049.00,\"\"],[1658592000000,1049.00,\"\"],[1658678400000,1049.00,\"\"],[1658764800000,1049.00,\"\"],[1658851200000,1049.00,\"\"]","ZheKouCount":95},"count":0}

5.Scrapy data persistence development

5.1. Writing Items（NySpider/items.py）:

import scrapy

class HistoryPriceItem(scrapy.Item):
    """
    自定义历史价格存储Item
    """
    # 商品URL
    itemUrl = scrapy.Field()
    # 图片URL
    picUrl = scrapy.Field()
    # 历史价格信息
    detailPrice = scrapy.Field()

5.2. Writing ItemPipelines（MySpider/pipelines.py）, taking file storage as an example:

import scrapy.crawler
from itemadapter import ItemAdapter
from scrapy import signals

class FilePipeline:

    def __init__(self, filename='store.txt'):
        self.filename = filename

    def process_item(self, item, spider):
        # 使用适配器包装item, 防止直接对item进行修改/删除影响后续Pipeline
        adapter = ItemAdapter(item)
        # 写入文件
        self.fp.write(adapter.get('itemUrl') + "    " + adapter.get('picUrl') + "    " + adapter.get('detailPrice') + '\n')
        return item

    @classmethod
    def from_crawler(cls, crawler:scrapy.crawler.Crawler):
        s = cls()
        # 通过信号绑定行为
        # 爬虫启动时创建文件fp
        crawler.signals.connect(s.opened, signal=signals.spider_opened)
        # 爬虫停止时关闭文件fp
        crawler.signals.connect(s.closed, signal=signals.spider_closed)
        return s

    def closed(self, spider):
        self.fp.close()

    def opened(self, spider):
        self.fp = open(self.filename, 'w', encoding='utf-8')
        self.fp.write('商品URL    主图URL    历史价格信息\n')

5.3. Modification Spiders（MySpider/spiders/manmanbuy.py）:

import scrapy
import json
from MySpider.items import HistoryPriceItem


class ManmanbuySpider(scrapy.Spider):
     # 省略未修改内容
     
     custom_settings = {
    
    
         # 配置使用的Item管道
        'ITEM_PIPELINES': {
    
    
            'MySpider.pipelines.FilePipeline': 300,
        }
    }
     
    def parse_history_price(self, response: scrapy.http.Response):
        # 解析价格响应
        self.logger.info(response.text)
        data = json.loads(response.text)
        # 返回Item
        return HistoryPriceItem(itemUrl=data['data']['spUrl'], picUrl=data['data']['spPic'], detailPrice=data['data']['datePrice'])

Here is a description of scrapy's 5种adding configuration methods. Commonly used ones are 3种that high-priority configurations will overwrite low-priority Keyconfigurations with the same configuration. Different Keyconfigurations are combined. According to priority, from high to low, they are:

1. Command line configuration
2. Crawler configuration
3. Project global configuration

SpidersThe parameters in custom_settingsare the crawler configuration

5.4. Run scrapy crawl manmanbuythe command to start the crawler, observe the current directory and find that a store.txtfile is generated. The content of the file is as follows:

商品URL    主图URL    历史价格信息
https://item.jd.com/100011493273.html    http://img13.360buyimg.com/n7/jfs/t1/201578/31/15673/77560/619479ceEd1bde507/c0dab826b71e0b84.jpg    [1621353600000,1199.00,""],[1621440000000,1199.00,""],[1621526400000,1199.00,""],[1621612800000,1199.00,""],[1621699200000,1199.00,""],[1621785600000,1199.00,""],[1621872000000,1199.00,""],[1621958400000,1199.00,""],[1622044800000,1199.00,""],[1622131200000,1199.00,""],[1622217600000,1199.00,""],[1622304000000,1199.00,""],[1622390400000,1199.00,""],[1622476800000,1199.00,""],[1622563200000,1199.00,""],[1622649600000,1199.00,""],[1622736000000,1199.00,""],[1622822400000,1199.00,""],[1622908800000,1199.00,""],[1622995200000,1199.00,""],[1623081600000,1199.00,""],[1623168000000,1199.00,""],[1623254400000,1199.00,""],[1623340800000,1199.00,"1199元"],[1623427200000,1199.00,""],[1623513600000,1199.00,""],[1623600000000,1199.00,""],[1623686400000,1199.00,""],[1623772800000,1199.00,""],[1623859200000,1199.00,""],[1623945600000,1099.00,"购买1件,当前价:1199.00,满减：每满1180减100"],[1624032000000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624118400000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624204800000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624291200000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624377600000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624464000000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624550400000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624636800000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624723200000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624809600000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624896000000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1624982400000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1625068800000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1625155200000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1625241600000,1139.00,"购买1件,当前价:1199.00,可叠加优惠券2：满750减60"],[1625328000000,1199.00,""],[1625414400000,1199.00,""],[1625500800000,1189.0,"京东秒杀价:1189"],[1625587200000,1199.00,""],[1625673600000,1189.0,""],[1625760000000,1199.0,""],[1625846400000,1189.0,""],[1625932800000,1199.0000,""],[1626019200000,1199.0000,""],[1626105600000,1189.0,""],[1626192000000,1199.0,""],[1626278400000,1189.0,""],[1626364800000,1199.0000,""],[1626451200000,1199.0000,""],[1626537600000,1199.0000,""],[1626624000000,1199.0000,""],[1626710400000,1199.0000,""],[1626796800000,1199.0000,""],[1626883200000,1189.00,""],[1626969600000,1199.0000,""],[1627056000000,1199.0000,""],[1627142400000,1199.0000,""],[1627228800000,1199.0000,""],[1627315200000,1189.00,""],[1627401600000,1199.0000,""],[1627488000000,1189.00,"1189元"],[1627574400000,1189.00,""],[1627660800000,1199.00,""],[1627747200000,1179.00,"1179元"],[1627833600000,1189.0000,""],[1627920000000,1199.00,""],[1628006400000,1199.00,""],[1628092800000,1189.00,""],[1628179200000,1189.0000,""],[1628265600000,1189.0000,""],[1628352000000,1189.0000,""],[1628438400000,1199.00,""],[1628524800000,1189.00,""],[1628611200000,1199.0,""],[1628697600000,1189.0000,""],[1628784000000,1189.0000,""],[1628870400000,1189.0000,""],[1628956800000,1199.00,"1199元"],[1629043200000,1189.0000,""],[1629129600000,1199.00,""],[1629216000000,1189.00,""],[1629302400000,1199.0000,""],[1629388800000,1169.0,"京东秒杀价:1169"],[1629475200000,1199.00,""],[1629561600000,1199.00,""],[1629648000000,1199.00,""],[1629734400000,1169.00,""],[1629820800000,1199.0,""],[1629907200000,1189.0,"京东秒杀价:1189"],[1629993600000,1199.00,""],[1630080000000,1199.00,""],[1630166400000,1199.00,""],[1630252800000,1199.00,""],[1630339200000,1189.00,"1189元包邮"],[1630425600000,1189.00,""],[1630512000000,1175.00,"购买1件,plus价格1175"],[1630598400000,1189.00,""],[1630684800000,1199.0,""],[1630771200000,1189.0000,""],[1630857600000,1199.0,""],[1630944000000,1189.0000,""],[1631030400000,1199.0,""],[1631116800000,1099.00,"购买1件,当前价:1199.00,满减：每满1180减100"],[1631203200000,1189.0000,""],[1631289600000,1189.00,""],[1631376000000,1199.00,""],[1631462400000,1189.0000,""],[1631548800000,1189.0,""],[1631635200000,1199.0,""],[1631721600000,1189.00,""],[1631808000000,1199.00,""],[1631894400000,1189.0,""],[1631980800000,1199.0,""],[1632067200000,1169.00,""],[1632153600000,1169.00,""],[1632240000000,1169.00,""],[1632326400000,1189.0,""],[1632412800000,1169.0,""],[1632499200000,1199.0,""],[1632585600000,1169.00,""],[1632672000000,1199.00,""],[1632758400000,1169.0,""],[1632844800000,1199.0000,""],[1632931200000,1169.0,""],[1633017600000,1169.0,""],[1633104000000,1169.0,""],[1633190400000,1169.00,""],[1633276800000,1169.0000,""],[1633363200000,1199.00,""],[1633449600000,1199.00,""],[1633536000000,1169.0,""],[1633622400000,1169.0,""],[1633708800000,1169.0,"京东秒杀价:1169"],[1633795200000,1169.0,""],[1633881600000,1199.00,""],[1633968000000,1169.00,""],[1634054400000,1199.0,""],[1634140800000,1169.00,""],[1634227200000,1199.0,""],[1634313600000,1199.0,""],[1634400000000,1169.0,""],[1634486400000,1199.0,""],[1634572800000,1169.00,""],[1634659200000,1199.0000,""],[1634745600000,1169.00,""],[1634832000000,1199.0,""],[1634918400000,1199.0,""],[1635004800000,1199.0,""],[1635091200000,1199.0,""],[1635177600000,1199.0,""],[1635264000000,1199.0,""],[1635350400000,1199.0,""],[1635436800000,1199.0,""],[1635523200000,1099.00,"1099元 "],[1635609600000,1099.0,""],[1635696000000,1099.00,""],[1635782400000,1099.00,""],[1635868800000,1099.00,""],[1635955200000,1099.00,"购买1件,plus价格1099"],[1636041600000,949.00,"购买1件,当前价:1099.00,可叠加优惠券2：满880减150"],[1636128000000,1099.00,""],[1636214400000,1199.0,""],[1636300800000,1099.0,""],[1636387200000,1099.0,""],[1636473600000,1099.0,""],[1636560000000,979.00,"购买1件,当前价:1099.00,满减：每满1080减120"],[1636646400000,1099.0000,""],[1636732800000,1199.00,""],[1636819200000,1199.00,""],[1636905600000,1199.00,""],[1636992000000,1199.00,""],[1637078400000,1099.00,""],[1637164800000,1099.00,""],[1637251200000,1099.00,""],[1637337600000,1099.00,""],[1637424000000,1099.00,"1099元"],[1637510400000,1099.00,""],[1637596800000,1099.0,""],[1637683200000,1099.00,""],[1637769600000,1099.00,""],[1637856000000,1099.00,""],[1637942400000,1099.00,""],[1638028800000,1199.0000,""],[1638115200000,1199.00,""],[1638201600000,1199.00,""],[1638288000000,1099.0,""],[1638374400000,1099.00,""],[1638460800000,1099.00,""],[1638547200000,1199.0,""],[1638633600000,1199.00,""],[1638720000000,1099.0,""],[1638806400000,1099.00,""],[1638892800000,1099.00,""],[1638979200000,1099.00,"1099元"],[1639065600000,1099.00,""],[1639152000000,1089.0,""],[1639238400000,1089.0,""],[1639324800000,1099.00,"购买1件,当前价:1199.00,满减：满1150减100"],[1639411200000,1099.00,"购买1件,当前价:1199.00,满减：满1150减100"],[1639497600000,1099.00,"购买1件,当前价:1199.00,满减：满1150减100"],[1639584000000,1099.00,""],[1639670400000,1099.0,""],[1639756800000,1099.0,""],[1639843200000,1099.0,""],[1639929600000,1099.00,"1099元"],[1640016000000,1099.0,""],[1640102400000,1099.0,""],[1640188800000,1099.0,""],[1640275200000,1099.00,"1099元"],[1640361600000,1099.0,""],[1640448000000,1069.0,""],[1640534400000,1099.0,""],[1640620800000,1099.0,""],[1640707200000,1099.0,""],[1640793600000,1099.0,""],[1640880000000,1099.0,""],[1640966400000,1099.00,"1099元"],[1641052800000,1099.0,""],[1641139200000,1099.0,""],[1641225600000,1099.0,""],[1641312000000,1099.0,""],[1641398400000,1099.0,""],[1641484800000,1099.0,""],[1641571200000,1099.0,""],[1641657600000,1099.00,"1099元"],[1641744000000,1099.0,""],[1641830400000,1099.0,""],[1641916800000,1099.0,""],[1642003200000,1099.0,""],[1642089600000,1099.0,""],[1642176000000,1099.0,""],[1642262400000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1642348800000,1099.00,""],[1642435200000,1099.00,""],[1642521600000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1642608000000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1642694400000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1642780800000,1099.0,""],[1642867200000,1099.0,""],[1642953600000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1643040000000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1643126400000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1643212800000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1643299200000,1099.0,""],[1643385600000,1099.0,""],[1643472000000,1099.0,""],[1643558400000,949.00,"购买1件,当前价:1099.00,满减：满1000减50,可叠加优惠券2：满880减100"],[1643644800000,949.00,"购买1件,当前价:1099.00,满减：满1000减50,可叠加优惠券2：满880减100"],[1643731200000,1099.0,""],[1643817600000,1099.0,""],[1643904000000,1099.0,""],[1643990400000,1099.0,""],[1644076800000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1644163200000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1644249600000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1644336000000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1644422400000,1099.00,""],[1644508800000,1099.00,"1099元"],[1644595200000,1199.00,"1199元"],[1644681600000,1199.00,"1089元"],[1644768000000,1039.00,"购买1件,当前价:1089.00,满减：满1000减50"],[1644854400000,1039.00,"购买1件,当前价:1089.00,满减：满1000减50"],[1644940800000,1099.00,"1049元"],[1645027200000,1099.00,""],[1645113600000,1099.00,""],[1645200000000,1099.00,""],[1645286400000,1099.00,""],[1645372800000,1099.00,""],[1645459200000,1099.00,""],[1645545600000,1099.00,""],[1645632000000,1099.00,""],[1645718400000,1099.00,"1099元"],[1645804800000,1099.00,"1069元"],[1645891200000,1049.00,"购买1件,当前价:1099.00,满减：满1000减50"],[1645977600000,1099.00,""],[1646064000000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1646150400000,1099.00,""],[1646236800000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1646323200000,1099.00,""],[1646409600000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1646496000000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1646582400000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1646668800000,899.00,"购买1件,当前价:1099.00,满减：每满1080减200"],[1646755200000,1099.00,""],[1646841600000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1646928000000,1099.0,""],[1647014400000,1099.0,""],[1647100800000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1647187200000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1647273600000,1099.00,""],[1647360000000,1099.00,"1099元"],[1647446400000,1049.0,"购买1件,当前价:1099.0,满减：满1050减50"],[1647532800000,1099.0,""],[1647619200000,1099.00,""],[1647705600000,1099.00,""],[1647792000000,1099.00,""],[1647878400000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1647964800000,1099.00,""],[1648051200000,1099.00,""],[1648137600000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1648224000000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1648310400000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1648396800000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1648483200000,1039.00,"购买1件,当前价:1089.00,满减：满1050减50"],[1648569600000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1648656000000,1049.00,"购买1件,当前价:1099.00,满减：满1050减50"],[1648742400000,1049.00,""],[1648828800000,1049.00,""],[1648915200000,1049.00,"1049元"],[1649001600000,1099.00,""],[1649088000000,1049.00,""],[1649174400000,1049.00,""],[1649260800000,1049.00,""],[1649347200000,1049.00,""],[1649433600000,1099.00,""],[1649520000000,1099.00,""],[1649606400000,1099.00,""],[1649692800000,1049.0,"购买1件,当前价格1049"],[1649779200000,1099.00,""],[1649865600000,1099.00,""],[1649952000000,1049.00,"1049元"],[1650038400000,1099.00,""],[1650124800000,1049.00,""],[1650211200000,1049.00,""],[1650297600000,1049.00,""],[1650384000000,1049.00,""],[1650470400000,1099.00,"1099元"],[1650556800000,1099.00,""],[1650643200000,1099.00,""],[1650729600000,1099.00,""],[1650816000000,1099.00,""],[1650902400000,1099.00,""],[1650988800000,1099.00,""],[1651075200000,1099.00,"1099元"],[1651161600000,1099.00,""],[1651248000000,1099.00,""],[1651334400000,1049.00,""],[1651420800000,1099.00,""],[1651507200000,1099.0000,""],[1651593600000,1099.0000,""],[1651680000000,1099.00,"1099元"],[1651766400000,1089.0,"购买1件,当前价格1089"],[1651852800000,1049.00,""],[1651939200000,1089.00,""],[1652025600000,1099.00,"1099元"],[1652112000000,1089.00,""],[1652198400000,1049.00,""],[1652284800000,1089.00,""],[1652371200000,1049.00,""],[1652457600000,1099.00,""],[1652544000000,1099.00,""],[1652630400000,1099.00,""],[1652716800000,1099.00,"1099元"],[1652803200000,1099.00,""],[1652889600000,1099.00,""],[1652976000000,1049.00,""],[1653062400000,1099.00,""],[1653148800000,1099.00,""],[1653235200000,1099.00,"1099元"],[1653321600000,1099.00,""],[1653408000000,1099.00,""],[1653494400000,1099.00,""],[1653580800000,1099.00,""],[1653667200000,1099.00,""],[1653753600000,1099.00,""],[1653840000000,1099.00,""],[1653926400000,1049.0,""],[1654012800000,1049.00,""],[1654099200000,1049.00,""],[1654185600000,1049.00,""],[1654272000000,1049.00,""],[1654358400000,1049.00,""],[1654444800000,1049.00,""],[1654531200000,1049.00,""],[1654617600000,1049.00,""],[1654704000000,1049.00,""],[1654790400000,1049.00,""],[1654876800000,1049.00,""],[1654963200000,1049.00,""],[1655049600000,1049.00,""],[1655136000000,1049.00,""],[1655222400000,1049.00,""],[1655308800000,1049.00,"1049元"],[1655395200000,1049.00,""],[1655481600000,1049.00,""],[1655568000000,1049.00,""],[1655654400000,1049.00,""],[1655740800000,1049.00,""],[1655827200000,1049.00,"1049元"],[1655913600000,1049.00,""],[1656000000000,1049.00,""],[1656086400000,1049.00,""],[1656172800000,1049.00,"1049元"],[1656259200000,1049.00,""],[1656345600000,1049.00,""],[1656432000000,1049.00,""],[1656518400000,1049.00,""],[1656604800000,1049.00,"1049元"],[1656691200000,1049.00,""],[1656777600000,1049.00,""],[1656864000000,1049.00,"1049元"],[1656950400000,1049.00,""],[1657036800000,1049.00,""],[1657123200000,1049.00,""],[1657209600000,1049.00,""],[1657296000000,1049.00,""],[1657382400000,1049.00,""],[1657468800000,1049.00,""],[1657555200000,1049.00,""],[1657641600000,1049.00,""],[1657728000000,1049.00,""],[1657814400000,1049.00,""],[1657900800000,1049.00,"1049元"],[1657987200000,1049.00,""],[1658073600000,1049.00,""],[1658160000000,1049.00,""],[1658246400000,1049.00,""],[1658332800000,1049.00,"1049元"],[1658419200000,1049.00,""],[1658505600000,1049.00,""],[1658592000000,1049.00,""],[1658678400000,1049.00,""],[1658764800000,1049.00,""],[1658851200000,1049.00,""]

It means the program is executing normally

PS: If you want to persist data to Mysql/MongoDB/Elasticsearch, you only need to write the corresponding ItemPipelinesimplementation and modify the crawler import ITEM_PIPELINESconfiguration to achieve decoupling of data persistence and crawler logic.

Conclusion

This article briefly explains Scrapythe reasons for using it, and demonstrates how to develop a Scrapy crawler project through a rationale. We will continue to provide you with Scrapymore detailed explanations in the future.