scrapy-splash抓取动态jd小米10价格

其他 2020-02-16 22:34:12 阅读次数: 0

一、安装splash

#docker安装

#拉取镜像
docker pull scrapinghub/splash

#运行容器
docker run -p 8050:8050 scrapinghub/splash

访问你自己服务器的ip，http://10.0.0.11:8050

二、安装scrapy-splash创建项目

pip install scrapy-splash

创建scrapy项目

scrapy startproject JDspider

配置setting

ROBOTSTXT_OBEY = False

SPIDER_MIDDLEWARES = {
    'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}

DOWNLOADER_MIDDLEWARES = {
    'scrapy_splash.SplashCookiesMiddleware': 723,
    'scrapy_splash.SplashMiddleware': 725,
    'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810
}

SPLASH_URL = 'http://10.0.0.11:8050' #你自己的服务器地址

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

创建spider文件

import scrapy
from scrapy_splash import SplashRequest
import logging

search_script = '''
function main(splash, args)
  splash.images_enabled = false
  splash:set_user_agent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36')
  assert(splash:go(args.url))
  splash:wait(0.5)
  local input = splash:select("#keyword")
  input:send_text('{}')
  splash:wait(0.5)
  local form = splash:select('.input_submit')
  form:click()
  splash:wait(2)
  splash:runjs("document.getElementsByClassName('bottom-search')[0].scrollIntoView(true)")
  splash:wait(6)
  return splash:html()
end

'''

class JsSpider(scrapy.Spider):
    name = "jd"
    allowed_domains = ["www.jd.com"]
    start_urls = [
        "https://search.jd.com/"
    ]

    def start_requests(self):
        splash_args = {
            'wait': 2,
            'lua_source': search_script.format("小米10")
        }
        for url in self.start_urls:
            yield SplashRequest(url, self.parse_result, endpoint='execute',
                                args=splash_args)

    def parse_result(self, response):
        if response.status == 200:
            ul_list = response.xpath('//*[@id="J_goodsList"]/ul/li')
            print(ul_list)
            print(len(ul_list))
            for i in range(1, len(ul_list) + 1):
                logging.info(u'----------使用splash爬取京东网异步加载内容-----------')
                xm10_price = response.xpath(
                    '//*[@id="J_goodsList"]/ul/li[{}]/div/div[3]/strong/i/text()'.format(i)).extract_first()
                logging.info(u"find：%s" % xm10_price)
                logging.info(u'---------------success----------------')

创建项目启动文件

from scrapy.cmdline import execute
execute(['scrapy', 'crawl', 'jd'])

三、运行项目输出结果

2020-02-16 22:15:26 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: JDspider)
2020-02-16 22:15:26 [scrapy.utils.log] INFO: Versions: lxml 4.3.2.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.5.2, w3lib 1.20.0, Twisted 19.2.0, Python 3.6.7 |Anaconda, Inc.| (default, Oct 28 2018, 19:44:12) [MSC v.1915 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0i  14 Aug 2018), cryptography 2.3.1, Platform Windows-10-10.0.17763-SP0
2020-02-16 22:15:26 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'JDspider', 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter', 'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage', 'NEWSPIDER_MODULE': 'JDspider.spiders', 'SPIDER_MODULES': ['JDspider.spiders']}
2020-02-16 22:15:26 [scrapy.extensions.telnet] INFO: Telnet Password: e23c115bddb3c3fa
2020-02-16 22:15:26 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2020-02-16 22:15:26 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_splash.SplashCookiesMiddleware',
 'scrapy_splash.SplashMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-02-16 22:15:26 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_splash.SplashDeduplicateArgsMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-02-16 22:15:26 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-02-16 22:15:26 [scrapy.core.engine] INFO: Spider opened
2020-02-16 22:15:26 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-02-16 22:15:26 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-02-16 22:15:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://search.jd.com/ via http://10.0.0.11:8050/execute> (referer: None)
[<Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item gl-item-presell" d...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item gl-item-presell" d...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="1000053...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6549112...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="1000035...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6545310...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="1000054...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6550526...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6545274...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="5873228...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item gl-item-presell" d...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6554474...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6549278...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6212269...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6111394...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6549862...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6551420...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6112452...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6121304...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6556373...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6551411...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6103996...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6554278...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6079772...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6555427...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6557779...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6363949...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6199133...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6553114...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="4723083...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6363895...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6128068...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6088586...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6364131...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6549037...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6550853...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6549124...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6546202...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6557937...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6557971...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6362546...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6557774...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6557828...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6557978...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6549308...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6558412...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6558070...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6551411...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6558723...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6558417...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6558373...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6561199...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6551616...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6560815...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6559425...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6557857...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6559487...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6557723...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item gl-item-presell" d...'>, <Selector xpath='//*[@id="J_goodsList"]/ul/li' data='<li class="gl-item" data-sku="6557979...'>]
60
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4299.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：5499.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：2799.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4489.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：599.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4489.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：3199.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4399.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4399.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：2699.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4999.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4199.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4199.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：2599.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：2599.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4178.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4799.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：2799.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：3299.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4199.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4499.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：3099.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4899.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：5199.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4199.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4199.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：2679.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：2999.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4999.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：2999.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：3059.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：2859.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：799.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：589.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4199.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4399.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：5499.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：5499.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：4199.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:36 [root] INFO: find：5499.00
2020-02-16 22:15:36 [root] INFO: ---------------success----------------
2020-02-16 22:15:36 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4799.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4199.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：5499.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：5499.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：5499.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：5299.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4699.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：5499.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：5488.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4199.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：5499.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4799.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4099.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4499.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4399.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4199.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：5399.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4799.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：5499.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [root] INFO: ----------使用splash爬取京东网异步加载内容-----------
2020-02-16 22:15:37 [root] INFO: find：4499.00
2020-02-16 22:15:37 [root] INFO: ---------------success----------------
2020-02-16 22:15:37 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 22:15:37 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1123,
 'downloader/request_count': 1,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 458413,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'elapsed_time_seconds': 10.35831,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 14, 15, 37, 94205),
 'log_count/DEBUG': 1,
 'log_count/INFO': 190,
 'response_received_count': 1,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'splash/execute/request_count': 1,
 'splash/execute/response_count/200': 1,
 'start_time': datetime.datetime(2020, 2, 16, 14, 15, 26, 735895)}
2020-02-16 22:15:37 [scrapy.core.engine] INFO: Spider closed (finished)

猜你喜欢

转载自www.cnblogs.com/angelyan/p/12319111.html
今日推荐
美国拟限制 AI 大模型出口中国和俄罗斯
苹果将与 OpenAI 达成协议，将 ChatGPT 应用于 iPhone
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
报告：Django 仍然是 74% 开发者的首选
《2024 年一季度互联网投融资运行情况》研究报告
15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
周排行
curl的POST请求，封装方法
8.1.1. Integer Types
Java基础 Day05(个人复习整理)
Python - Django - 中间件 process_exception
小L的试卷
【Shell编程】 （函数）判断用户是否存在
python(css样式)
spring ant path 匹配原则 - 【笔记】
《JavaScript与JScript从入门到精通》(美)James.Jaworski.中译本.扫描版.pdf
Eclipse运行带参数的java程序
每日归档
更多
2024-05-12(0)
2024-05-11(38)
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022