Python3 Scrapy框架学习一：爬取猫眼Top100榜

以下操作基于Windows平台。

打开CMD命令提示框：

输入 如下命令：

打开项目里的items.py文件，定义如下变量，用于存储。

class MaoyanItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    movie = scrapy.Field()
    actor = scrapy.Field()
    release = scrapy.Field()
    score = scrapy.Field()

然后打开项目里的spiders文件夹内的maoyanTop100.py文件。

# -*- coding: utf-8 -*-
import time
import scrapy
from scrapy.http import Request
from maoyan.items import MaoyanItem


class Maoyantop100Spider(scrapy.Spider):
    name = 'maoyanTop100'
    #allowed_domains = ['maoyan.com/board/4']
    #start_urls = ['http://maoyan.com/board/4']
    
    def start_requests(self):
         for offset in range(0,100,10):
            self.url = 'http://maoyan.com/board/4?offset={}'.format(offset)
            yield Request(self.url,callback = self.parse)
            time.sleep(1)

    def parse(self, response):
        context = response.css('dd')   #分析得知所有的电影item均在该标签内
        for info in context:
            item = MaoyanItem()
            item['movie'] = info.css('p.name a::text').extract_first().strip()
            item['actor'] = info.css('.star::text').extract_first().strip()
            item['release'] = info.css('.releasetime::text').extract_first().strip()
            score = info.css('i.integer::text').extract_first().strip()
            score += info.css('i.fraction::text').extract_first().strip()
            item['score'] = score
            yield item

打开settings.py文件，加入如下两行代码：

USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'
FEED_EXPORT_ENCODING = 'gbk'

然后在命令提示符输入:

scrapy crawl maoyanTop100 -o maoyan.csv

爬取结束后会生成一个csv文件，效果图如下：