Python爬虫之scrapy框架中ip代理池的设置

• 开放代理

class IpProxyDownloadMiddleware(object):
    PROXIES = ['110.52.235.131:9999','110.52.235.249:9999','112.17.38.141:3128']

    def process_request(self,request,spider):
        proxy = random.choice(self.PROXIES)
        request.meta['proxy'] = 'http://'+proxy

注：上面在设置代理时，要添加协议名称，http://或https://,否则会报错。

twisted.web.error.SchemeNotSupported: Unsupported scheme: b''

• 独享代理

  class IpProxyDownloadMiddleware(object):
    def process_request(self,request,spider):
        proxy = "121.199.6.124:16816"
        user_password = "970138074:rcdj35ur"
        request.meta['proxy'] = 'http://'+proxy
        #bytes
        bs64_user_password = base64.b64encode(user_password.encode('utf-8'))
        request.headers['Proxy-Authorization'] = 'Basic' + bs64_user_password.decode('utf-8')