爬虫自动获取UserAgent

导入

from fake_useragent import UserAgent,FakeUserAgentError
import random

获取UserAgent对象

try:
    ua = UserAgent()
except FakeUserAgentError:
    pass

用random随机获取一个UserAgent，并加入到headers

base_headers = {
    'User-Agent':  ua.random,
    'Accept-Encoding': 'gzip, deflate, sdch',
    'Accept-Language': 'zh-CN,zh;q=0.8'
}

最后带着Hearder请求

reaponse = requests.get(url, headers=headers)