Encounter the following exception when using requests crawl the web:
requests.exceptions.ConnectionError: HTTPSConnectionPool Max retries exceeded
- the reason
1.http number of connections exceeds the maximum limit, by default connection Keep-alive, so this leads to too many connections the server can no longer maintain the new connection.
2.ip blocked
3. requests too
- solve
1. persistent header is not used in the connection ‘Connection’: ‘close’or requests.adapters.DEFAULT_RETRIES = 5
2. If the requests too, may be provided the time.sleep
3. Use proxy ip
sometimes 4.Requests less than the requested page request, or a request to blank page timeout to re try several times, using a try ... except statement