Article Directory
proxy
step:
1. The parameter is a dictionary {'type':'agent ip: port number'}
proxy_support=urllib.request.ProxyHandler({
})
2. Customize, create an opener
opener=urllib.request.build_opener(proxy_support)
3a. Install opener
urllib.request.install_opener(opener)
3b. Call opener
opener_open(url)
How to add headers
Proxy ip can also be obtained Use-Agent
, add the following code below the second step
opener.addheaders=[('User-Agent','Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.25 Safari/537.36 Core/1.70.3866.400 QQBrowser/10.8.4379.400')]
Can
Implementation code
As for where to find the proxy ip, you can find it here, domestic proxy server
Separate proxy ip:
import urllib.request
url='https://www.baidu.com/'
proxy_support=urllib.request.ProxyHandler({
'http':'165.227.35.172:8768'})
opener=urllib.request.build_opener(proxy_support)
urllib.request.install_opener(opener)
response=urllib.request.urlopen(url)
html=response.read().decode('utf-8')
print(html)
The proxy ip forms a list:
mport urllib.request
import random
url='https://www.baidu.com/'
iplist=['165.227.35.172:8768','165.227.35.172:8768','165.227.35.172:8768']
proxy_support=urllib.request.ProxyHandler({
'http':random.choice(iplist)})
opener=urllib.request.build_opener(proxy_support)
urllib.request.install_opener(opener)
response=urllib.request.urlopen(url)
html=response.read().decode('utf-8')
print(html)