Scrapy添加IP的方法

版权声明：本文为博主原创文章，未经博主允许不得转载。如有侵权，请联系本人进行删除，谢谢! https://blog.csdn.net/dongdong0071/article/details/79438032

1、settings.py中添加下载中间件

DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 127,
    'lianjia.middlewares.MyproxiesSpiderMiddleware': 126,
}

2、middlewares.py中添加代码

from scrapy import signals
import random
from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware


class MyproxiesSpiderMiddleware(object):

    def __init__(self, ip=''):
        self.ip = ip

    def process_request(self, request, spider):
        thisip = random.choice(self.IPPOOL)
        print("this is ip:" + thisip["ipaddr"])
        request.meta["proxy"] = "http://" + thisip["ipaddr"]

    IPPOOL = [
        {"ipaddr": "66.81.206.241:3128"},
        {"ipaddr": "27.147.163.77:80"},
        {"ipaddr": "121.58.17.52:80"},
        {"ipaddr": "183.57.36.87:8888"},
        {"ipaddr": "47.52.222.165:80"},
        {"ipaddr": "47.52.44.31:8080"},
        {"ipaddr": "202.109.237.35:80"}
    ]