Scrapy Crawled (200) <GET http://www.baidu.com/> (referer: None) error and solution

As shown in the figure below, this error is established when the scrapy framework is established. As shown in the figure, the left side of the picture is the project structure of the scrapy framework. If there is an error in the title, first click the settings.py file as shown in the figure to find Line 40, as shown in the figure, I have already framed it. These two lines were commented when the file was just opened. Uncomment these two lines and add the "user-agent" field, which can be obtained in the browser. I won’t introduce it in detail, you can Baidu by yourself, and then run the project again, you can find the source code of the network.

 

Guess you like

Origin blog.csdn.net/weixin_55109596/article/details/123736893