Selenium adds socks5 proxy with account secret
We all know that when using selenium to develop crawlers, it is inevitable to use socks5 high-anonymity proxy. In general, we use the method as follows (the development language is python):
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("proxy-server=socks5://127.0.0.1:1080")
driver = webdriver.Chrome( "./chromedriver", chrome_options=chrome_options)
Our proxy is: socks5://127.0.0.1:1080, of course this is a local socks5 proxy that can be used without an account password, but if we set up a server and set up a socks5 proxy service on it, if you don’t set an account password, wouldn’t it be It will become an open socks5 proxy service that everyone can use, you ask me why? Because there is always some opportunistic person who scans the whole network and finds open proxies on your server.
But you have to ask again, chrome does not support socks5 proxy plus authentication, I can't blame me for this.
It is true that chrome does not natively support socks5 proxy authentication, so Brother Xiaoma introduces a method here, and there are other suitable ways to communicate.
Let's get straight to the point. It's actually very simple. If the server's socks5 proxy needs to be verified, we don't directly use the server's socks5 proxy. We add a layer of forwarding in the middle, so that the forwarding middleware supports socks5 verification. Ability to control this middleware. So now our process is like this:
the idea is clear, then the next question is how do we implement the forwarding middleware of the socks5 proxy. Of course, we don't need to re-implement the socks5 protocol in person, there are already corresponding implementations on the market, we only need to have the eyes to discover.
1. Install the Pproxy tripartite library
install pproxy
pip install pproxy
2. How to use
1. Enter on the cmd command line:
Find the installation path of python, enter cmd to start the command line:
Change the ip and port to socks5 proxy, fill in the account password, this method is to monitor port 8080, you can also set other ports, as long as there is no conflict
pproxy -l socks5://127.0.0.1:8080 -r socks5://ip:prot#账号:密码 -vv
The above command is very simple - l stands for listening listen, -r stands for forwarding relay, then the meaning is very clear, we listen to the local port 8080 to provide unauthenticated socks5 service, and forward the received data packets to the needs indicated by ip:port Authenticated socks5 service.
If successful, you will see the image below
2. Add an agent to selenium
option.add_argument('--proxy-server=socks5://127.0.0.1:8080')
Implementation ideas
We can use python to start a child process, start pproxy as a forwarding middleware before selenium starts chrome, and then set the started chrome proxy as the local socks5 forwarding middleware