scrapy appear [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to how to solve

During this period, it has been used for User-Agent, and open source is not connected directly to jump, use requests also tested and found not to be redirected in the browser.

Search long, long time to no avail, then reluctantly began to pay attention [scrapy.downloadermiddlewares.redirect] DEBUG this thing I do is open up a configuration, checked and no configuration associated with this, but suddenly found that I actually configure a DEFAULT_REQUEST_HEADERS

This is configured before doing Baidu search, which specifies the host, a reptile load will use this header, this result does not match the specified host with source links inside, it took place redirection. A back annotation like ..

Guess you like

Origin www.cnblogs.com/tulintao/p/12596064.html