python 代理浏览器 爬虫资源

zb.py....................................................

import os

index = 0
def response(flow):
    global index
    path=""
    print('===========开始执行===========')
    if flow.request.url.find("http://")>=0:
        path=flow.request.url.split("http://")
        print("http请求:")
        down(path,flow,flow.request.url)
    elif flow.request.url.find("https://")>=0:
        path=flow.request.url.split("https://")
        print("https请求:")
        down(path,flow,flow.request.url)
def down(path,flow,url):
    if flow.request.url.split('.')[-1] == 'jpg' or flow.request.url.split('.')[-1] == 'png' or flow.request.url.split('.')[-1] == 'json'or flow.request.url.split('.')[-1] == 'mp3':
        duan=path[1].split("/")
        yuming=duan[0]#域名 cdn-bzsc.jikewan.com
        duancounnt=len(duan)
        realname=duan[duancounnt-1]#文件真实名字 80.json
        realpath=path[1].replace(yuming,"")
        realpath="ziyuan"+realpath.replace(realname,"")
        #print("真实路径："+realpath)
        #print("真实名字："+realname)
        if not os.path.exists(realpath):
            os.makedirs(realpath)
        file_name = realpath + realname
        with open(file_name, 'wb') as f:
            f.write(flow.response.get_content())
            f.close
         #记录log
        with open("ziyuan/addresslog.txt", 'a') as f:
            f.write(url+"\n")
            f.close()

python需要大于3.7

安装pip

执行或者去官网下载mitmproxy  安装版本程序

pip install -i https://pypi.tuna.tsinghua.edu.cn/simple  mitmproxy

安装证书

C:\Users\Administrator\.mitmproxy 下面找找

执行pytho文件

mitmdump -p 8585 -s zb.py

360浏览器示例设置代理

 到此完事！可下载网站的图片,和json文件 

【tips:可用来获取页游资源】

打开网站 下面网站做测试

http://pic.5tu.cn/photo/fj/photo_fj.htm