In the crawler request, there must be a header request header, fake_useragent is an extension package applied to UA in python
Run in CMD installation mode
pip install fake_useragent
Instructions
from fake_useragent import UserAgent
ua = UserAgent()print(ua.random) #random
print(ua.ie) #ie
print(ua.chrome) #Google
print(ua.firefox) #火狐print(ua.safari)
#Apple
print(ua.opera) #欧朋
The result is as follows
Mozilla/5.0 (Windows NT 6.4; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2225.0 Safari/537.36
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 7.0; InfoPath.3; .NET CLR 3.1.40767; Trident/6.0; en-IN)
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1468.0 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20130330 Firefox/21.0
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
Opera/12.0(Windows NT 5.1;U;en)Presto/22.9.168 Version/12.00
practice
from fake_useragent import UserAgent
ua = UserAgent()def html(url):
try:
r = req.get(url,timeout=80,headers={'User-Agent': ua.random})
r.raise_for_status()
r.encoding = r.apparent_encoding
return r.text
except Exception as e:
print(e)