This article is for learning not to be used for commercial reference
The crawling using fiddler + simulator (download vibrato APP) + pycharm
1. Download the latest version of the fiddler (self Baidu download), and configuration
1.1. Click the menu bar -Tools-Options-HTTPS, as shown in the red box 1 tick, HTTPS beside Connections configuration, as shown in the red box tick
1.2. Click again HTTPS bar, click Actions select the second option ... to this, send the certificate to the desktop Desktop (easy to find, you need to install the emulator)
2. Download and install the simulator, I use the simulator Yagami
Official website: https://www.yeshen.com/
And other pages or modify their own memory. . . .
2.1 Back to the desktop, find the certificate that you just saved, drag it to the simulator, look at the path diagram red box.
2.2 return simulator desktop, open the Settings pull-down into the find - security - Choose from SD card to install - install as shown below, find the certificate, just enter prompt.
2.3 Returning to simulator to find WiFi, go, button press, to modify the network, as modified, native query ip, cmd input ipconfig, IPv4 is behind the local IP
3. simulator download application market vibrato App, this configuration is complete, restart the simulator, and then test whether the fiddler to catch the request packet.
4. Test can catch request. Then analyzes the request.
5, is herein crawling 'vibrato nickname', 'vibrato ID', 'Profile' thus returned view capture data in FIG. 2
6. The data obtained can not save a copy to a local, so you can write scripts using fiddler save data on their own, away from their own to add one shown in Figure 3, the script added the corresponding data is automatically saved to a local file json
7. The following is a fiddler script itself changes as needed
if (oSession.uriContains("https://aweme-eagle-hl.snssdk.com/aweme/v1/user/")){
//上边的网址是你需要保存数据的网址开头的部分，网站修改的话需要及时修改url
var strBody=oSession.GetResponseBodyAsString();
var sps = oSession.PathAndQuery.slice(-64,-54);
//FiddlerObject.alert(sps)
var filename = "E:/此处是本地路径" + "/" + sps + ".json";
var curDate = new Date();
var sw : System.IO.StreamWriter;
if (System.IO.File.Exists(filename)){
sw = System.IO.File.AppendText(filename);
sw.Write(strBody);
}
else{
sw = System.IO.File.CreateText(filename);
sw.Write(strBody);
}
sw.Close();
sw.Dispose();
}
8. vibrato in the simulator, whether to turn down, turn right, the user to view the page, scripting fiddler automatically save the data you need. Figure
9. So the question is when you want to take a lot of time crawling data, it is impossible manually turn the page, so it comes time to use the simulator operations assistant.
9.1, some admit yourself, you want the simulator continuous cycle of operation, the number of repetitions, turn down, remember to save.
9.2. Click on the play button, the emulator will automatically record your operation cycle, has been turning down, fiddler, will continue to have new corresponding caught, more and more local json data.
=======================================================
Basic operations do over, then get the json file how to resolve it, this time knocking the code go.
=======================================================
10. Talk about ideas, not to show the specific code.
10.1, from local to go to json file into a list, circular list of json
10.2 json.load () to read json file content
10.3 take information you need in json.
10.4 store, I saved excel file, there are some problems, I talk about here.
# Error when storing special characters, here with codecs storage, storage csv be encoded chaos problems, solutions, right use Notepad to open the saved csv file, click Save As, choose the encoding format-8 UTF.
With codecs .Open ( "./ douyin / douyin.csv", "A +", encoding = 'UTF-. 8') AS F:
Writer = csv.writer (F, dialect = "Excel")
writer.writerow (L)
10.5 climbed to the data, there will be from repetitive, de-duplication is very simple, I can not say here.
Show you what I crawled data
