python学习之第二十一天(抓包工具)

知识点目录
1.什么叫抓包?
2.常用的抓包软件有哪些
3.如何抓取手机端app?
4.如何抓取微信某技术号的文章信息?
5.如何抓取教师端的相关请求并用代码实现对应功能?
6…熟悉mysql安装及navicate破解
7.爬虫过程中cookie失效怎么解决?
8.如何使用正则采集网页源码的json数据

1.什么叫抓包?

抓包（packet capture）就是将网络传输发送与接收的数据包进行截获、重发、编辑、转存等操作，也用来检查网络安全。抓包也经常被用来进行数据截取等。

2.常用的抓包软件有哪些

Fiddler是在windows上运行的程序，专门用来捕获HTTP，HTTPS的。

wireshark能获取HTTP，也能获取HTTPS，但是不能解密HTTPS，所以wireshark看不懂HTTPS中的内容。

Charles其实是一款代理服务器，通过过将自己设置成系统（电脑或者浏览器）的网络访问代理服务器，然后截取请求和请求结果达到分析抓包的目的。该软件是用Java写的，能够在Windows，Mac，Linux上使用。安装Charles的时候要先装好Java环境。
Charles的主要功能：
（1）截取Http 和 Https 网络封包。
（2）支持重发网络请求，方便后端调试。
（3）支持修改网络请求参数。
（4）支持网络请求的截获并动态修改。
（5）支持模拟慢速网络。

3.如何抓取手机端app?

抓取手机app配置

# 1. 安装抓包工具 charles  fiddler
# 2. 手机和电脑连在同一个网络中
#    电脑开热点让手机连
# 3. 手机设置连接热点的端口和IP地址
# 4. 打开手机对应的app，进行数据抓取即可

charlse抓取手机app的相关配置
https://blog.csdn.net/h176nhx7/article/details/79236495

charles抓取https请求(默认只能抓http,不配置这个抓不了https请求)
https://www.cnblogs.com/fighter007/p/9162617.html

fiddler抓取https
https://www.cnblogs.com/joshua317/p/8670923.html

4.如何抓取微信某技术号的文章信息?

import re
import requests
# 这是在抓包工具抓到的数据
url = "https://mp.weixin.qq.com/mp/profile_ext?action=home&__biz=MzIxODM4MjA5MA==&scene=124&uin=Mjk0OTE4MjM4Mg%3D%3D&key=b3176bc64ae00a443cf14744a77cdb6159d0fa4b4d0a3035c47bfe42179336fc33415d97d68c88dfec86758a402e817fc4d1a6eee22d84d91d46dde7f692bb69baa126d7493316067f1bd4e0d2a9bd5c&devicetype=Windows+10&version=62060833&lang=zh_CN&a8scene=7&pass_ticket=3ItgczzTtwCsxt6Cl1f0SwP%2B2rIpyevYK3FtiIZOPFtq7qqsH%2BHdiDCKc8j9PXkD&winzoom=1"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36 QBCore/3.53.1159.400 QQBrowser/9.0.2524.400 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 MicroMessenger/6.5.2.501 NetType/WIFI WindowsWechat",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate",
    "Accept-Language": "zh-CN,zh;q=0.8,en-US;q=0.6,en;q=0.5;q=0.4",
    "Cookie": "wxuin=2949182382; devicetype=Windows10; version=62060833; lang=zh_CN; pass_ticket=3ItgczzTtwCsxt6Cl1f0SwP+2rIpyevYK3FtiIZOPFtq7qqsH+HdiDCKc8j9PXkD; wap_sid2=CK7no/4KElw4Z19OX1lRam1XdEUwTTF2Q0NOQ29kWVhJdXJNbGt0eFYteGZBdElUazhrUk5VRVlfQTJoQXhLQUFYUUpvMWwzUmQ0dzhBalUwVjdUMTRGcThvQzVVdjhEQUFBfjCTkfnqBTgNQJVO",
}

response = requests.get(url, headers=headers)
print(response.text)

# 把response.text数据粘到wx.html中
f = open("wx.html", encoding="utf-8")
content = f.read()
f.close()

# 用正则提取需要的数据
pattern = re.compile("var msgList = '(.*?)'")
pattern = re.compile("var msgList = (.*);")
result = pattern.findall(content)
print(result)

5.如何抓取教师端的相关请求并用代码实现对应功能?

# 利用抓包工具抓取用户登录的post请求的数据,然后实现用代码模拟登录
# 
# 1.先用抓包工具抓取教师登录后提交的post请求数据
# 
# 2.表单
params = """{"mobile":"18790868582","password":"7194be6e78a0ebdd3ca9d6626c2ad13b"}"""

# 3.请求头的数据
headers = {
    "Accept": "*/*",
    "Origin": "https://teacher.zhiyou888.com",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36",
    "Content-Type": "application/json",
    "Referer": "https://teacher.zhiyou888.com/login.html",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "zh-CN,zh;q=0.9",
}
url = "https://teacher.zhiyou888.com/api/common/user/login"

# 4.用代码模拟浏览器实现登录
response = requests.post(url, headers=headers, data=params)
print(response.text)

#
url = "https://teacher.zhiyou888.com/api/course-focuses"
headers = {
    "Accept": "application/json, text/plain, */*",
    "Origin": "https://teacher.zhiyou888.com",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36",
    "Content-Type": "application/json;charset=UTF-8",
    "Referer": "https://teacher.zhiyou888.com/",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "zh-CN,zh;q=0.9",
    "Cookie": "ZHIYOU_TEACHER=ZHIYOU:USER:TOKEN:add63aeb8628487eb7d01f3af4bdb7e0",
}

params = """{"date":"2019-08-22","classId":654,"comment":["hello world","hello"]}"""
# 添加课程重点（只能添加一次）
response = requests.post(url, data=params, headers=headers)
# 更新课程
response = requests.put(url, data=params, headers=headers)
# 删除课程
response = requests.delete(url+"?classId=654&date=2019-08-22", headers=headers)
# 查询课程
response = requests.get("https://teacher.zhiyou888.com/api/common/course-focuses/654?pageNo=1", headers=headers)
print(response.text)

6.如何激活navicate专业版?

见2019-8-22学习笔记

7.爬虫过程中cookie失效怎么解决?

1.简单的但是不太可取,cookie失效后,刷新页面,获取新的cookie,然后更新代码的cookie

2.动态获取cookie,这种做法难度较大

8.如何使用正则采集网页源码的json数据

和正则匹配其它数据一样,写好正则表达式就能匹配到所需的数据