Scrapy frame framework Scrapy

Scrapy framework

An introduction

    Scrapy an open source and collaborative framework, which was originally intended to crawl a page, you can use it for quick, simple, scalable way to extract the required data from the site (more precisely, the network crawl) designed. But Scrapy Extensive use, can be used such as data mining, testing, monitoring and automation field may be applied in the acquisition of data returned by the API (e.g. Amazon Associates Web Services) or general web crawler.

    Scrapy is based on the framework developed from twisted, twisted python web framework is a popular event-driven. Thus Scrapy using one kind of non-blocking code (also known as asynchronous) to implement concurrency. The overall structure is as follows

Components：

1. Engine (EGINE)

   Engine control system is responsible for the data flow between all components, and trigger events when certain actions occur. For more information, see the section above the data stream.

2. A scheduler (SCHEDULER)
   for receiving a request sent over the engine, is pressed into the queue, and returns the engine again when the request may be thought of as a URL of the priority queue, it is determined by the URLs to crawl is a What, while removing duplicate URLs
3. Downloader (DOWLOADER)
   for downloading web content and web content back to EGINE, downloader is built on this twisted efficient asynchronous model
4. Reptiles (SPIDERS)
   SPIDERS a developer-defined class, for parsing Responses, and extracts items, or send a new request
5. Pipeline Project (ITEM PIPLINES)
   in the extracted items after they are responsible, including cleaning, validation, persistence (for example to the database) and other operations
6. 下载器中间件(Downloader Middlewares)
   位于Scrapy引擎和下载器之间，主要用来处理从EGINE传到DOWLOADER的请求request，已经从DOWNLOADER传到EGINE的响应response，你可用该中间件做以下几件事
   1. process a request just before it is sent to the Downloader (i.e. right before Scrapy sends the request to the website);
   2. change received response before passing it to a spider;
   3. send a new Request instead of passing received response to a spider;
   4. pass response to a spider without fetching a web page;
   5. silently drop some requests.
7. 爬虫中间件(Spider Middlewares)
   位于EGINE和SPIDERS之间，主要工作是处理SPIDERS的输入（即responses）和输出（即requests）

官网链接：https://docs.scrapy.org/en/latest/topics/architecture.html

二 安装

#Windows平台
    1、pip3 install wheel #安装后，便支持通过wheel文件安装软件，wheel文件官网：https://www.lfd.uci.edu/~gohlke/pythonlibs
    3、pip3 install lxml
    4、pip3 install pyopenssl
    5、下载并安装pywin32：https://sourceforge.net/projects/pywin32/files/pywin32/
    6、下载twisted的wheel文件：http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
    7、执行pip3 install 下载目录\Twisted-17.9.0-cp36-cp36m-win_amd64.whl
    8、pip3 install scrapy

#Linux平台
    1、pip3 install scrapy

三 命令行工具

#1 查看帮助
    scrapy -h
    scrapy <command> -h

#2 有两种命令：其中Project-only必须切到项目文件夹下才能执行，而Global的命令则不需要
    Global commands:
        startproject #创建项目
        genspider    #创建爬虫程序
        settings     #如果是在项目目录下，则得到的是该项目的配置
        runspider    #运行一个独立的python文件，不必创建项目
        shell        #scrapy shell url地址  在交互式调试，如选择器规则正确与否
        fetch        #独立于程单纯地爬取一个页面，可以拿到请求头
        view         #下载完毕后直接弹出浏览器，以此可以分辨出哪些数据是ajax请求
        version      #scrapy version 查看scrapy的版本，scrapy version -v查看scrapy依赖库的版本
    Project-only commands:
        crawl        #运行爬虫，必须创建项目才行，确保配置文件中ROBOTSTXT_OBEY = False
        check        #检测项目中有无语法错误
        list         #列出项目中所包含的爬虫名
        edit         #编辑器，一般不用
        parse        #scrapy parse url地址 --callback 回调函数  #以此可以验证我们的回调函数是否正确
        bench        #scrapy bentch压力测试

#3 官网链接
    https://docs.scrapy.org/en/latest/topics/commands.html

#1、执行全局命令：请确保不在某个项目的目录下，排除受该项目配置的影响
scrapy startproject MyProject

cd MyProject
scrapy genspider baidu www.baidu.com

scrapy settings --get XXX #如果切换到项目目录下，看到的则是该项目的配置

scrapy runspider baidu.py

scrapy shell https://www.baidu.com
    response
    response.status
    response.body
    view(response)
    
scrapy view https://www.taobao.com #如果页面显示内容不全，不全的内容则是ajax请求实现的，以此快速定位问题

scrapy fetch --nolog --headers https://www.taobao.com

scrapy version #scrapy的版本

scrapy version -v #依赖库的版本

#2、执行项目命令：切到项目目录下
scrapy crawl baidu
scrapy check
scrapy list
scrapy parse http://quotes.toscrape.com/ --callback parse
scrapy bench
    

示范用法

四 项目结构以及爬虫应用简介 

-scrapy
        -安装
            -pip3 install scrapy
                -先装Twisted
                -装pywin32
        -新建项目
            -scrapy startproject 项目名字
        -新建爬虫
            -scrapy genspider 爬虫名 爬取的域名
        -项目目录介绍
            -spiders
                -所有的爬虫程序
            -items.py
                -类似于django的model类
            -middlewares.py
                -中间件
            -pipelines.py
                -持久化相关
            -settings.py
                -配置文件
            -scrapy.cfg
                -部署相关
        -运行爬虫
            -scrapy crawl cnblogs --nolog
            
        -爬取数据

文件说明：

• scrapy.cfg  项目的主配置信息，用来部署scrapy时使用，爬虫相关的配置信息在settings.py文件中。
• items.py    设置数据存储模板，用于结构化数据，如：Django的Model
• pipelines    数据处理行为，如：一般结构化的数据持久化
• settings.py 配置文件，如：递归的层数、并发数，延迟下载等。强调:配置文件的选项必须大写否则视为无效，正确写法USER_AGENT='xxxx'
• spiders      爬虫目录，如：创建文件，编写爬虫规则

注意：一般创建爬虫文件时，以网站域名命名

#在项目目录下新建：entrypoint.py
from scrapy.cmdline import execute
execute(['scrapy', 'crawl', 'xiaohua'])

默认只能在cmd中执行爬虫，如果想在pycharm中执行需要做

import sys,os
sys.stdout=io.TextIOWrapper(sys.stdout.buffer,encoding='gb18030')

关于windows编码

五 Spiders

1、介绍

#1、Spiders是由一系列类（定义了一个网址或一组网址将被爬取）组成，具体包括如何执行爬取任务并且如何从页面中提取结构化的数据。

#2、换句话说，Spiders是你为了一个特定的网址或一组网址自定义爬取和解析页面行为的地方

2、Spiders会循环做如下事情