When I was doing a crawler before, I often encountered the problem that for a webpage, if you use the same IP multiple times, the IP will be banned. We can manually change the proxy IP and continue this problem, but it will be a bit troublesome. I am a lazy person. Generally speaking, manually changing the IP is too troublesome, and it does not conform to the lazy virtue of programmers, so the following story follows.
proxy_pool is an open source proxy pool that aggregates all major free ip proxy pools . When your own crawler is blocked because it crawls too fast, the proxy pool can come in handy
The main function of the crawler agent IP pool project is to regularly collect free agents published on the Internet for verification and storage, and to regularly verify and store agents to ensure the availability of agents, and to provide API and CLI. At the same time, you can also expand the proxy source to increase the quality and quantity of the proxy pool IP.