Python scrapy fake agent and use of fake_userAgent

The use of scrapy camouflage agent and fake_userAgent

When the camouflage browser proxy is crawling web pages, some servers are not very good at filtering requests. It is also possible to camouflage your browser information directly without using IP to camouflage the request.

The first method:

1. Add the following content to the setting.py file, this is the header information of some browsers

USER_AGENT_LIST = ['zspider/0.9-dev http://feedback.redkolibri.com/',
                    'Xaldon_WebSpider/2.0.b1',
                    'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)',
                    'Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)',
                    'Speedy Spider (Entireweb; Beta/1.3; http://www.entireweb.com/about/search_tech/speedyspider/)',
                    'Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)',
                    'Speedy Spider (Entireweb; Beta/1.1; http://www.entireweb.com/about/search_tech/speedyspider/)',
                    'Speedy Spider (Entireweb; Beta/1.0; http://www.entireweb.com/about/search_tech/speedyspider/)',
                    'Speedy Spider (Beta/1.0; www.entireweb.com)',
                    'Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)',
                    'Speedy Spider (http://www.entireweb.com/about/search_tech/speedyspider/)',
                    'Speedy Spider (http://www.entireweb.com)',
                    'Sosospider+(+http://help.soso.com/webspider.htm)',
                    'sogou spider',
                    'Nusearch Spider (www.nusearch.com)',
                    'nuSearch Spider (compatible; MSIE 4.01; Windows NT)',
                    'lmspider ([email protected])',
                    'lmspider [email protected]',
                    'ldspider (http://code.google.com/p/ldspider/wiki/Robots)',
                    'iaskspider/2.0(+http://iask.com/help/help_index.html)',
                    'iaskspider',
                    'hl_ftien_spider_v1.1',
                    'hl_ftien_spider',
                    'FyberSpider (+http://www.fybersearch.com/fyberspider.php)',
                    'FyberSpider',
                    'everyfeed-spider/2.0 (http://www.everyfeed.com)',
                    'envolk[ITS]spider/1.6 (+http://www.envolk.com/envolkspider.html)',
                    'envolk[ITS]spider/1.6 ( http://www.envolk.com/envolkspider.html)',
                    'Baiduspider+(+http://www.baidu.com/search/spider_jp.html)',
                    'Baiduspider+(+http://www.baidu.com/search/spider.htm)',
                    'BaiDuSpider',
                    'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0) AddSugarSpiderBot www.idealobserver.com',
                ]

2. Create a MidWare file in the same level directory of the spider and write a HeaderMidWare.py file with the content as

# encoding: utf-8
from scrapy.utils.project import get_project_settings
import random

settings = get_project_settings()

class ProcessHeaderMidware():
    """process request add request info"""

    def process_request(self, request, spider):
        """
        随机从列表中获得header， 并传给user_agent进行使用
        """
        ua = random.choice(settings.get('USER_AGENT_LIST'))  
        spider.logger.info(msg='now entring download midware')
        if ua:
            request.headers['User-Agent'] = ua
            # Add desired logging message here.
            spider.logger.info(u'User-Agent is : {} {}'.format(request.headers.get('User-Agent'), request))
        pass

3. Add in the setting.py file

DOWNLOADER_MIDDLEWARES = {
'projectName.MidWare.HeaderMidWare.ProcessHeaderMidware': 543,
}

The second method: the use of fake_userAgent

fake_userAgent is an open source project on github
1. Install fake_userAgent
pip install fake-useragent
2. Create a MidWare file in the same level directory of the spider and write a user_agent_middlewares.py file content as

# -*- coding: utf-8 -*-
from fake_useragent import UserAgent

class RandomUserAgentMiddlware(object):
    #随机跟换user-agent
    def __init__(self,crawler):
        super(RandomUserAgentMiddlware,self).__init__()
        self.ua = UserAgent()
        self.ua_type = crawler.settings.get('RANDOM_UA_TYPE','random')#从setting文件中读取RANDOM_UA_TYPE值

    @classmethod
    def from_crawler(cls,crawler):
        return cls(crawler)

    def process_request(self,request,spider):  ###系统电泳函数
        def get_ua():
            return getattr(self.ua,self.ua_type)
        # user_agent_random=get_ua()
        request.headers.setdefault('User_Agent',get_ua())
        pass

3. Add in setting.py

RANDOM_UA_TYPE = 'random'##random    chrome

DOWNLOADER_MIDDLEWARES = {

'projectName.MidWare.user_agent_middlewares.RandomUserAgentMiddlware': 543, 

'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware':None,

}

The fake_userAgent fake agent is configured. Compared with the first method, there is no need to write a large series of browser headers. Those browser headers will be available at https://fake-useragent.herokuapp.com/browsers/0.1.7.

---

There will be some errors when the fake_userAgent is enabled for the first time. I think it is caused by the need to cache some content when the project requests the network.

github address: https://github.com/sea1234/fake-useragent