How to set http proxy by crawler

1. Find the proxy IP

Looking for http proxy platform

Shenlong http (you can get 1000ip for free within 1 day after registration)

Dragonfly Agent

For the rest, you can refer to this blog

Free IP proxy network

Two, set up nginx

Note: nginx currently only applies to http proxy. If you want https proxy, you need to look for other proxy servers. Currently, charges are generally required

1. Download nginx

nginx download address

2. After downloading, unzip

3. Open the nginx conf/nginx.conf file (you can use Notepad), and add the following code in http{}:

server {
    
    
	resolver 60.18.23.81:35161;
	#resolver是DNS服务器,后面输入代理IP
	listen 8888;
	#代表服务器的端口号是8888
	location / {
    
    
	#location /指任何任何URL都通过这个代理(因为指定路径为/)
		proxy_pass http://$http_host$request_uri;
		#proxy——pass后面的地址表示代理服务器根据客户端的请求向资源服务器发送的URL。$http_host$request_uri是nginx的内部变量,分别表示客户端发过来的IP(域名)以及请求路径
	}
}

4. Open nginx

Three, Chrome set up a proxy server

Open the settings and find the proxy.
Insert picture description here

Open the "Proxy server" option, enter the proxy IP in the address, enter 8888 in the port, and then save it.
At this point, you can open a web page to see if the setting is successful. If the web page is displayed successfully, the proxy setting is successful

Fourth, the requests library uses a proxy

import requests
proxies = {
    
    'http':'114.99.17.51:63488'}
r = requests.get('http://www.china.com.cn/',proxies=proxies)
print(r.text)

The result is displayed successfully
Insert picture description here

Guess you like

Origin blog.csdn.net/sgsdsdd/article/details/115002584