As an automated program, Python crawlers are very useful for scenarios that require a large amount of data to be crawled. However, because the website is worried about being illegally obtained by crawlers, it often adopts various anti-crawling methods to block or limit the work of crawlers. The following will introduce some common anti-climbing techniques and corresponding countermeasures.
1. IP ban
IP ban is a common anti-crawling technology, which is mainly restricted by monitoring the IP address of visitors. When an IP address has been visited many times or certain requests are too frequent, the server will block the IP address to prevent crawlers from further accessing the website.
solution:
(1) Use proxy IP
Using a proxy IP is one of the main ways to deal with IP bans. The proxy IP is equivalent to a middleman, sending the request to the target website, and the real IP address is replaced by the proxy IP. In this way, even if the server blocks the current IP address, the crawler can still continue to access the website through other proxy IPs.
(2) Use IP pool
Since the proxy IP may be blocked by the website, it is necessary to ensure the stability of the proxy IP. Establish an IP pool, monitor the available proxy IP at any time, and dynamically select the available proxy IP when the crawler requests, so that the crawler can access more stably and efficiently.
2. User-Agent detection
User-Agent refers to the user information carried in the HTTP request header. Since the default value of the User-Agent of each crawler is the same, the server can identify and reject the request of the crawler by detecting the User-Agent.
solution:
(1) Set different User-Agent
When the crawler requests, replacing the User-Agent can avoid detection to a certain extent. Multiple User-Agents can be randomly set in the configuration file and randomly selected for use to avoid being restricted by the server.
(2) Simulate browser behavior
Simulating the browser behavior can make the User-Agent more real, and can realize the normal access of crawlers through complex authentication mechanisms such as verification codes. The specific implementation method includes setting the values of the Referer field, Cookie field, and Accept-Encoding field to be consistent with the simulated browser, and simulating operations such as registration and login before requesting a download page.
3. Verification code
Verification codes are generally used to protect sensitive information and prevent machines from maliciously grabbing or maliciously registering accounts. The types of verification codes include picture verification codes, voice verification codes, sliding verification codes, click verification codes, etc. Many anti-crawler technologies use verification codes to prevent machine crawlers from accessing.
solution:
(1) Manually enter the verification code
You can use the manual method, that is, manually enter the verification code, to realize the normal crawling of the crawler. However, this method is costly and inefficient.
(2) Coding service API
By using the coding service API that recognizes the verification code, send the verification code picture to the API server, and the API server will return the correct answer of the verification code. It should be noted that the accuracy and stability of the coding service will have a great impact on the efficiency of the crawler.
4. Request frequency limit
Since crawlers can request the same page in rapid succession, the server usually detects the frequency of requests to prevent crawlers from obtaining large amounts of data too quickly. The server usually automatically limits the number of requests on the website or blocks requests for a period of time to prevent excessive use of crawlers.
solution:
(1) Use random request intervals
By setting different request time intervals, it can simulate human operation behavior and reduce the probability of being detected. You can simulate human browsing behavior as much as possible by setting random time intervals and the number of requests. At the same time, it should be noted that too long time interval and too few requests may reduce the crawler efficiency.
(2) Use asynchronous requests
By using asynchronous requests, multiple requests can be sent at one time to obtain a large amount of data in a short period of time. This method can improve the crawling efficiency, but you need to pay attention to the load on the server and the anti-crawling mechanism of the website.
5. Dynamic page data acquisition
Dynamic pages usually use JavaScript to dynamically generate the content of the page. After the crawler obtains the source code of the page, it will become difficult to analyze it. This requires us to use some technical means to simulate real page requests.
solution:
(1) Use a headless browser
Using a headless browser can simulate a human browser very well, and it is different from traditional crawlers in terms of loading JS and rendering pages. Using a headless browser to obtain pages does not require writing a lot of JS code and anti-crawler strategies, but is consistent with human operations on web pages, more in line with the natural output mode, and can pass most anti-crawler mechanisms.
(2) Analyze Ajax requests
The data loading method that often appears in dynamic pages is obtained through Ajax requests. We can use developer tools to view XHR requests, copy and modify the discovered APIs, provide a basis for obtaining data loaded by Ajax, and obtain API address information by analyzing these requests.
In short, the continuous strengthening of the anti-crawling mechanism and the security measures of more and more Web sites require us to constantly update and learn to understand the latest anti-crawling mechanism and countermeasures. It is possible to use technical means to carry out stealthy crawling. Due to various considerations such as data volume and crawling efficiency, it is still necessary to make decisions based on specific conditions and choose an appropriate crawling solution.