Copyright Notice: Copyright: This article is a blogger original article, shall not be reproduced without the bloggers allowed. https://blog.csdn.net/qq_42658739/article/details/91358117
1. Use your own proxy setting up the pool, or others to build a good agent pool, usually in the form of web api's.
[Proposed Github more to stroll, have you unexpected gains, but also to improve their English and participate in projects that improve]
(1) The first step of course is the address of your api.
E.g:
self.PROXY_POOL_URL = 'http://localhost:5000/get'
(2) sends a request to the agent pool api acquired IP returned generally string [back]
def get_proxy(self):#代理池 并未启用
try:
response = requests.get(self.PROXY_POOL_URL)
if response.status_code == 200:
return {
'http':response.text,
'https':response.text
}
#proxies=Movie.get_proxy(self)
except ConnectionError:
return None
(3) after obtaining attention requests accepted proxy format is a dictionary format
2. In addition, you can put online by the IP edge proxy website crawling side use, or crawl after the api database or a Web page easy to use