Redirect 301 error in Scrapy
1. Problem description
[scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301)
2. The cause of the problem
301 Moved Permanently The requested resource has been permanently moved to a new location, and any future references to this resource should use one of several URIs returned by this response. If possible, the client with the link editing function should automatically modify the requested address to the address returned from the server.
3. Solution
-
Use the browser to log in to the website to be crawled normally, get the latest URL, and change the URL address in the crawler accordingly
-
Add in the settings.py file
- HTTPERROR_ALLOWED_CODES = [301]