Data movement crawling

- crawling movement end data
 - capture tool:
     - fiddler
     - blue and white
     - miteproxy 

    - environment structures
         1 . Fiddler configured to: tools- "options-" connections-> Allow remote cooxxx-> set a port
         2 test. port is available: access localhost in the browser pc: 8886 
        3 .pc open a hotspot, allowing the phone to connect (to ensure that the cell phone and pc fiddler is located in the same network segment)
         4 access localhost on your phone: 8886 were trust certificates installed and
         5 . settings in the phone proxy
             - wifi connected phone set proxy ip and port (both based fiddler)

Example of use

import requests
# headers = {
# #     'User-Agent':'qu tou tiao/3.6.1 (iPhone; iOS 12.3.1; Scale/3.00)/qukan_ios'
# # }
# # url = 'https://api.1sapp.com/content/getListV2?qdata=NTRGQzhENkZFMUE1M0I2NkM0QzU2MjZGOEQ3NjI1RkQuY0dGeVlXMGZPRVEyTTBaRFJVTXRRalE0TlMwME1qVkdMVGswUmtNdFEwVkVOemt4TlVRM09ETTJIblpsY25OcGIyNGZOaDV3YkdGMFptOXliUjlwYjNNZVpXTWZNUT09LtagD4vABRxcsa6fa7qpeSB5kCo897LVZGyk2MyKfjIoRx0c9SbUZYexOhselSV%2BhoaUc/BpFA764n5CPUQajjBkMtkiUyhnw9JMPLBL7yfzwHlqwoBf2b%2BqG%2BfJgeug73zQATh9Gm5HaQZcuMV6M2mehrfdVi3hFwSyCVtun8zAqlDpP8hSBrjfiJf8v5%2BqDJzJo9zW8k2dX%2B%2B/o30iwq/Cy2FXeqz8izPZXQO5%2BLDLXmYCKplQzRYT51GIQzIybsgg/sb/as5aziAwV95SpihCfwh1I0c86pOAs5egm93wR3%2Bt/p/PIaubhxsIrA9SFpxkBULVx48Rj/C997VsBbYD7gsNsLtnAqb14V1ZEa1P8OA3yXxW9Hn7C87IVXfm24NBZoZyA6kDh2OP0jP1bItQciJuCAd9mODlSmwxVthrf6iY/PZcl9G4P13%2B1PfCz32kSDItFxLIrtYk%2BOfgjaC3ct9EzjMTEY7SjM0jKQ8lFSEj6cbmZGZIIJcNiSLnokFHAaVDTygfcvmjqrifhYc7pQzzEfqrkPNycl3SC36XU2v16uA5brE4vOUdx2AS2mD%2BH5BGQJOgPgvK6FuFjOaHW1nt9XQCAhwCF4QPSO11z6QyQo0mEbsTeWhvVmnN1BCgOlOI985qquH1nRL / 3XGyIK% 2B36E2icJZcJxMwi1QM% 2BReUsbBLkzX3mYq / hGWeG7nvJUwEI5jhmM1tvI3QRiFK% 2B1Zn1DIQVnYOEc8PJwR0rleY8mNDxDZSjJL8kiXlbUUH% 2Bg% 3D% 3D' 
# # 
# # #ERROR: ssl 
# # #verify = False can ignore certificate errors 
# # json_data = Requests. GET (url = url, headers = headers, the Verify = False) .json () 
# # Print (json_data) 

# crawling vibrato short video: a slide to a video, then enter the author's home video, all video data to crawl home page 
# 1 . in fillder the need to capture data packets to the beginning of api.amemv, (then extract url all requests header information and then carries fiddler performed for the request url) back in response to this packet is set json data to search for the keyword in v9 json data, as long as you are carrying url v9 to crawl the corresponding video url

View Code

fiddler Profile
Phone APP packet capture settings
- fiddler settings
- Install Certificate Download
- Security certificate installation
- LAN Settings
- fiddler phone capture test

1 What is Fiddler?
Fiddler are located in the client and server-side HTTP proxy, is also one of the most commonly used http capture tool. It can record all HTTP requests between clients and servers, can target specific HTTP requests, request data analysis, set breakpoints, debugging web applications, modify data requests and even modify the data returned by the server, very powerful, It is a web debugging tool.

Since it is a proxy, that is to say: all client requests have to go through Fiddler, and then forwarded to the appropriate server, on the contrary, all the server's response, it will first and then sent to the client through Fiddler, for this reason, Fiddler supports all http proxy can be set to 127.0.0.1:8888 browsers and applications. After using Fiddler, requesting web client and the server is as follows:

You can set this feature using the proxy, we can capture the mobile phone APP. How to set? And not be anxious, let me just get Fiddler installed on it!

Fiddler Download: https://www.telerik.com/fiddler

Fool installation, a key in the end. Fiddler software interface as shown:

2 Phone APP packet capture settings

a. Fiddler settings

Open Fiddler software, open the Settings tool. (Fiddler software menu bar: Tools-> Options)

In HTTPS as follows:

In Connections set as follows, using the default port 8888 here, of course, you can change yourself, but be careful not to conflict with the port already in use:

Allow remote computers to connect: to allow other requests to machine up fiddler

. b Download security certificate
enter the address in the computer browser: http: // localhost: 8888 / , click FiddlerRoot certificate, download a security certificate:

c. the security certificate installation
certificate is required for installation on the phone, so that when computer software Fiddler capture, the phone card to use the Internet computers will not error.

Android phone installation: the certificates in the phone's internal or external memory card, and then through the phone's "System Security -" install the certificate from the storage device installation "menu.

然后找到拷贝的FiddlerRoot.cer进行安装即可。安装好之后，可以在信任的凭证中找到我们已经安装好的安全证书。

苹果手机安装：

- 保证手机网络和fiddler所在机器网络是同一个网段下的

- 在safari中访问http://fiddle机器ip：fiddler端口，进行证书下载。然后进行安装证书操作。

- 在手机中的设置-》通用-》关于本机-》证书信任设置-》开启fiddler证书信任

d. 局域网设置
想要使用Fiddler进行手机抓包，首先要确保手机和电脑的网络在一个内网中，可以使用让电脑和手机都连接同一个路由器。当然，也可以让电脑开放WIFI热点，手机连入。这里，我使用的方法是，让手机和电脑同时连入一个路由器中。最后，让手机使用电脑的代理IP进行上网。
在手机上，点击连接的WIFI进行网络修改，添加代理。进行手动设置，ip和端口号都是fiddler机器的ip和fiddler上设置的端口号。

e. Fiddler手机抓包测试

上述步骤都设置完成之后，用手机浏览器打开百度首页，我们就可以顺利抓包了

Mobile data terminal crawlers crawl

Data movement crawling

2 Phone APP packet capture settings

a. Fiddler settings

Guess you like