Abstract In addition to Web pages, reptiles can grab App data. App pages to load it, you first need to get the data, and these data are generally through the interface request to the server to get the. Since this App is not a browser tool can be more intuitive to see backstage requests, so the main to capture some of the data packet capture technology.
Packet capture tools presented in this book have Charles, mitmproxy and mitmdump. Some simple interface by Charles or mitmproxy analysis, find out the law, then the direct use of simulation program to grab the. But if you encounter more complex interface, you need to perform real-time processing and preservation crawl to the requests and responses use mitmdump docking Python. In addition, since the do-scale collection, you need to automate operations rather than artificial App to collect, so here need a tool called Appium, it can be the same for App automated control like Selenium, such as automated simulation App click, drop-down, etc. operating.
In this section, we will introduce Charles, mitmproxy, mitmdump, Appium installation method.
[Quick Entry]: [Python3 web crawler developed combat] 1.7.1-Charles installation
[Quick Entry]: [Python3 web crawler developed combat] 1.7.2-mitmproxy installation
Source: Huawei cloud community Author: Cui Shu Jing Qing only seek