This error occurs because using wget or curl resource request is rejected by the server, in order to prevent crawlers consume server resources, the server selectively in accordance with your request header shield, it is necessary to modify wget and curl agent to User-Agent camouflage.
First, modify the wget-Agent of the User
1, UA wget temporary change of
Plus parameters -U, representatives set up in front of wget User Agent
wget www.google.com -U "Mozilla / 5.0 ( Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 72.0.3626.121 Safari / 537.36"
What is the User Agent, how to obtain, can refer to the following this blog, of course, you can also directly above me with this.
What is UserAgent and viewed using a browser UserAgent method:
https://blog.csdn.net/BobYuan888/article/details/88950275
2, permanent change of UA Wget
Modify the configuration file / etc / wgetrc, add the following line:
header = User-Agent: Mozilla / 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 72.0.3626.121 Safari / 537.36
above this configuration after the entry into force of all users, if you need only the current user , you can add the line above ~ / .wgetrc, if not, then this document can be created manually.
Second, the modified curl-Agent of the User
1, UA curl of the temporary change
Use the following parameters:
curl https://www.google.com --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
2, permanent change of UA curl
Modify the configuration file ~ / .curlrc, add the following line:
–user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36
reference:
1、https://www.linpx.com/p/on-an-interesting-play-wget-use.html
2、https://chaifeng.com/_curl_wget_user-agent/
————————————————
