Recently, due to work needs, I studied mitmproxy as a proxy crawler, and encountered a typical problem, that is, during the crawling process, it is necessary to regularly update the secondary proxy ip (everyone knows =. =)
What can be found on the Internet is the result of 18 years, and it has long been unusable. Through the official github of mitmproxy, I found the latest solution and shared it with students who need it.
def request(flow: http.HTTPFlow) -> None:
address = proxy_address(flow)
is_proxy_change = address != flow.server_conn.via.address
server_connection_already_open = flow.server_conn.timestamp_start is not None
if is_proxy_change and server_connection_already_open:
# server_conn already refers to an existing connection (which cannot be modified),
# so we need to replace it with a new server connection object.
flow.server_conn = Server(flow.server_conn.address)
flow.server_conn.via = ServerSpec("http", address)
PS: To enable the secondary agent, you need to add option when restarting the service
# Usage: mitmdump
# -s change_upstream_proxy.py
# --mode upstream:http://default-upstream-proxy:8080/
# --set connection_strategy=lazy
# --set upstream_cert=false
Original address: https://github.com/mitmproxy/mitmproxy/discussions/5173
In addition, there is another problem. The above solution only modifies the secondary proxy settings of the current request every time, and does not modify the configuration of the current mitproxy service synchronously (that is, the mode parameter we gave when we started), so all requests will always be The old proxy before the request will appear, resulting in long request time, 502 and other problems.
After studying the source code, the part of modifying the service configuration is added to solve this problem
is_proxy_change = proxy_address != flow.server_conn.via.address
server_connection_already_open = flow.server_conn.timestamp_start is not None
if is_proxy_change and server_connection_already_open:
# server_conn already refers to an existing connection (which cannot be modified),
# so we need to replace it with a new server connection object.
flow.server_conn = Server(flow.server_conn.address)
if is_proxy_change:
print("原代理" + str(flow.server_conn.via.address) + '|新代理' + str(proxy_address))
flow.server_conn.via = ServerSpec('http', proxy_address)
mode_option = {'mode': str('upstream:' + proxyinfo)}
server = getServer()
# 更新运行环境中的代理设置
print("当前运行环境代理配置：" + ctx.master.options.__getattr__('mode'))
ctx.master.options.update(**mode_option)
print("当前运行环境配置更新后：" + ctx.master.options.__getattr__('mode'))