As the saying goes, "It is better to teach fish than to teach fish," so the editor today will share with you the method of climbing the historical data of the epidemic situation.
Basic idea: analyze the "anti-pneumonia" section of Tencent News, use the "reverse push method" to find the epidemic data interface, and then use python to simulate the request, and then save the historical data of the epidemic.
1. Analysis data interface
1. Go to the homepage of overseas epidemic situation and analyze the historical data of epidemic situation in overseas countries.
https://news.qq.com/zt2020/page/feiyan.htm#/global
Open the traffic analysis tool, taking "United States" as an example, click Details
Select the first json file that appears and find that the response data is the epidemic data of the United States from January 28 to today, including confirmed_add, confirmed, cured and dead data.
The specific information for this request is:
In the form of a post request, the URL is:
https://api.inews.qq.com/newsqa/v1/automation/foreign/daily/list?country=国家名称2. Enter the homepage of domestic epidemic situation and analyze the historical data interface of epidemic situation in each province in China:
https://news.qq.com/zt2020/page/feiyan.htm#/globalOpen the traffic analysis tool, take "Heilongjiang" as an example, click on the details
Select the first json file that appears, and the response data is the epidemic data from January 28 to today in Heilongjiang, including confirmed_add, confirmed, cured, and dead data
The specific information for this request is:
In the form of a post request, the URL is:
https://api.inews.qq.com/newsqa/v1/query/pubished/daily/list?province=各省名称3. Analyze the historical data interface of each city in the province in the same way:
In the form of a post request, the URL is:
https://api.inews.qq.com/newsqa/v1/query/pubished/daily/list?province=省名称&city=市名称4. Analyze and find the names of overseas countries, domestic provinces, and cities in the province
Open the traffic analysis tool and enter in the URL:
https://news.qq.com/zt2020/page/feiyan.htm#/global① Open the request URL:
https://view.inews.qq.com/g2/getOnsInfo
name=disease_foreign&callback=jQuery34105534161076800497_1587272333891&_=1587272333892Response window:
The returned data contains the names of all countries.
② Open the request URL:
https://view.inews.qq.com/g2/getOnsInfo?name=disease_h5&callback=jQuery34105534161076800497_1587272333887&_=1587272333888
的响应窗口：
The returned data contains the names of provinces and cities across the country.
Step 2: Start writing the program
1. Simulate crawling of historical epidemic data of overseas countries
According to the analysis results of the first step, first crawl the names of each country, and then construct the URL to crawl the historical data codes of each country as follows:
2. Simulate crawling of epidemic situation data of domestic provinces According to the first step analysis results, first crawl the names of each province, and then construct url to crawl the historical data of each province.
3. Simulate crawling of epidemic data of cities in the province
According to the analysis results of the first step, first crawl the names of the cities in the province, and then construct the URL to crawl the historical data of each city. The code is as follows:
The third step: effect display
1. Epidemic data of overseas countries:
2. Domestic epidemic situation data:
3. Epidemic data of Heilongjiang cities: