The following error occurs while scraping data in Python:
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.
Too many redirects have occurred in requests , more than 30.
Two solutions:
- You can add allow_redirects=False in the request request. The default is allow_redirects=True, so this can solve the problem. But doing so also results in the inability to obtain redirected data.
The following methods can be utilized:
1. First enable the response to requests:
s = requests.session()
2. Set the user agent in the request header:
s.headers['User-Agent'] = 'User Agent Encoding' [Encoding can copy the request URL to the browser, press f12 to open the NetWork of the developer tools to find the request information]
3. Request data
data = s.get(url)