Python crawler entry [21]: user station network know almost crawler scrapy

Full stop reptiles sometimes it is actually easier to do, because the rule is relatively easy to set up, only need to do anti-climb on it, and today we know almost crawling. Continue to use the scrapycourse for this little demand, the use of scrapy do with the chopper, but after all, this series at this stage need to continue to use scrapyundue, so, I wrote a while on finished.

The first step you find a crawling seed, counted as reptiles entrance

https://www.zhihu.com/people/zhang-jia-wei/following

We need the following information, all of the block diagram is the information we need.

Get the user watch list

Acquires the Web page returned by the code data, the data will be found by the HTML + JSON spliced ​​together, a lot more cost analytical

class ZhihuSpider(scrapy.Spider):
    name = 'Zhihu'
    allowed_domains = ['www.zhihu.com']
    start_urls = ['https://www.zhihu.com/people/zhang-jia-wei/following']

    def parse(self, response):
        all_data = response.body_as_unicode()
        print(all_data)

First look at the basic configuration of the environment, such as the number of seconds, crawling UA, whether to store cookies, enable random UA middlewareDOWNLOADER_MIDDLEWARES

middlewares.py file

from zhihu.settings import USER_AGENT_LIST # 导入中间件
import random

class RandomUserAgentMiddleware(object):
    def process_request(self, request, spider):
        rand_use  = random.choice(USER_AGENT_LIST)
        if rand_use:
            request.headers.setdefault('User-Agent', rand_use)
Python资源分享qun 784758214 ,内有安装包，PDF，学习视频，这里是Python学习者的聚集地，零基础，进阶，都欢迎

setting.py file

BOT_NAME = 'zhihu'

SPIDER_MODULES = ['zhihu.spiders']
NEWSPIDER_MODULE = 'zhihu.spiders'
USER_AGENT_LIST=[  # 可以写多个，测试用，写了一个
    "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
]
# Obey robots.txt rules
ROBOTSTXT_OBEY = False
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 2
# Disable cookies (enabled by default)
COOKIES_ENABLED = False
# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en',
}
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
    'zhihu.middlewares.RandomUserAgentMiddleware': 400,
}
# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
   'zhihu.pipelines.ZhihuPipeline': 300,
}

The main function of crawling, description

1. start_requests crawling first request for processing, as a program entry
2. The following code process mainly two cases, one in HTML part, one is part JSON
3. JSON re module portion using matching by formatting module json
4. extract_first() The first match of the array get xpath
5. dont_filter=False scrapy URL deduplication

 # 起始位置
    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url.format("zhang-jia-wei"), callback=self.parse)

    def parse(self, response):

        print("正在获取 {} 信息".format(response.url))
        all_data = response.body_as_unicode()

        select = Selector(response)

        # 所有知乎用户都具备的信息
        username = select.xpath("//span[@class='ProfileHeader-name']/text()").extract_first()       # 获取用户昵称
        sex = select.xpath("//div[@class='ProfileHeader-iconWrapper']/svg/@class").extract()
        if len(sex) > 0:
            sex = 1 if str(sex[0]).find("male") else 0
        else:
            sex = -1
        answers = select.xpath("//li[@aria-controls='Profile-answers']/a/span/text()").extract_first()
        asks = select.xpath("//li[@aria-controls='Profile-asks']/a/span/text()").extract_first()
        posts = select.xpath("//li[@aria-controls='Profile-posts']/a/span/text()").extract_first()
        columns = select.xpath("//li[@aria-controls='Profile-columns']/a/span/text()").extract_first()
        pins = select.xpath("//li[@aria-controls='Profile-pins']/a/span/text()").extract_first()
        # 用户有可能设置了隐私，必须登录之后看到，或者记录cookie！
        follwers = select.xpath("//strong[@class='NumberBoard-itemValue']/@title").extract()

        item = ZhihuItem()
        item["username"] = username
        item["sex"] = sex
        item["answers"] = answers
        item["asks"] = asks
        item["posts"] = posts
        item["columns"] = columns
        item["pins"] = pins
        item["follwering"] = follwers[0] if len(follwers) > 0 else 0
        item["follwers"] = follwers[1] if len(follwers) > 0 else 0

        yield item

        # 获取第一页关注者列表
        pattern = re.compile('<script id=\"js-initialData\" type=\"text/json\">(.*?)<\/script>')
        json_data = pattern.search(all_data).group(1)
        if json_data:
            users = json.loads(json_data)["initialState"]["entities"]["users"]
        for user in users:
            yield scrapy.Request(self.start_urls[0].format(user),callback=self.parse, dont_filter=False)
Python资源分享qun 784758214 ,内有安装包，PDF，学习视频，这里是Python学习者的聚集地，零基础，进阶，都欢迎

When acquiring data, I bypassed the part of the data, which may be part of the data to match the regular expression by a positive.

Data storage, is still usedmongodb