Selenium basics 3-modify the initial options of webdriver such as header and cookies

Although ip can use a proxy, in the crawling process, the browser request is still easily recognized by tools such as waf, and it is very easy to be walled, so when using selenium, I was thinking, can I send the request like Use the request to modify the header dynamically.

Let me start with the conclusion. After consulting the information, it proved that selenium does not support dynamic modification of the header request. Some bloggers also gave some opinions.

github issue

• WebDriver lacks HTTP response header and status code methods #141
• How to add a header to a request? (using python bindings + webdriver) #2047

stackoverflow question

• How to get status code by using selenium.py (python code)

It's probably impossible to achieve, and no one has seen this. But there is also flexibility. We can configure the header before selenium starts webdriver, which can solve this problem to a certain extent. In addition, I also posted some bloggers' solutions.

https://blog.csdn.net/u013948858/article/details/92614346

The main solutions are as follows:

1. Use another driver/library instead of selenium (this depends on the library or package given by the field of programming: python selenium-wire)
2. Write browser-specific plugins (or find existing plugins) that allow you to add headers to the request. (It seems that there are plug-ins: chrome and modify-headers firefox, but there are requirements for the browser version)
3. Use browsermob-proxy or some other proxy. (Agent is a good idea)
4. It can be done by using a proxy such as fiddler. Fiddler also provides an API-only version of the FiddlerCore component, as well as programmatic access to all proxy settings and data, allowing you to modify the headers of the http response. (Much like the third agent, basically the same)

Related articles about modifying the initial request webdriver.option() are as follows:

https://www.zhihu.com/question/35547395?sort=created

https://www.cnblogs.com/TTyb/p/6128323.html

https://www.cnblogs.com/yangjintao/p/10599868.html

This article is more detailed:
https://blog.csdn.net/xc_zhou/article/details/82415870

Some code block examples are as follows:

options = webdriver.ChromeOptions()

#设置浏览器header
options.add_argument('User-Agent=Mozilla/5.0 (Linux; U; Android 8.1.0; zh-cn; BLA-AL00 Build/HUAWEIBLA-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.132 MQQBrowser/8.9 Mobile Safari/537.36')

#导入chromedriver路径
browser = webdriver.Chrome(executable_path = r'C:\xx\chromedriver.exe',chrome_options = options)

webdriver.ChromeOptions() can also customize the initial information such as cookies, whether to load pictures, and whether to run visually. Not only applicable to chrome, here is just an example of chrome.