Preface
The operation of the official account has begun. The most
rewarding part is the collection of materials. It allows me to add interesting activities in my life. Anything I am interested in can be put in the background of the official account. It can be said that the official account can do much more than I expected. At the same time, collecting materials will continue to exercise my ability to collect data and use crawlers. In
terms of small programs, the original plan was to imitate a weather app and a calculator. These two are semi-finished products, and it is not cost-effective to force it to continue. Instead of spending energy on small programs that have nothing to do with the official account, it is better to make two genuine and practical small programs.
progress
Completed more than three articles.
Completed the collection of the first batch of fans on the official account. Completed the establishment of the official account’s
menu bar.
Completed the goal and content of the mini program. The preliminary plan has been formed
. A preliminary understanding of the article publishing process has been completed. And use multimedia materials.
Completed the second small program Web association (I hope it is useful, but it seems really useless)
Incompletely turn on the appreciation function
Incompletely set the message function
Plan to develop a small program for stills collectors (using the simplest and rude means).
Plan to develop a small program for movie recommendation (link to Douban api, with search function).
Plan to use python technology to collect current popular resources in a timely manner-WeChat 8.0 status material, below Is the specific implementation content of this plan
Collect material (short video of WeChat status at station b)
Process analysis
Goal: Collect a large number of WeChat status materials.
Source: A collection of all up masters of station b.
Means: you-get, python crawler
process
Collect with you-get
You-get has to use the following list of functions to download videos.
You-get is really convenient, downloading is fast and accurate, and you can do it in one step.
Collect with python crawler
It’s really cool to use you-get, but it’s not conducive to the practice of python crawlers. The
difficulty of crawling the main video of station b with code lies in understanding the interface of station b, and splicing the required download url through this interface
has not yet been resolved.
Collect with scrapy crawler
First of all, the application of scrapy has to take advantage of its advantages. At present, there are two found: scrapy downloads files fast.
You can follow the link to download the site-wide files.
Then, under the requirement of crawling the b-site video, scrapy can crawl a collection of multiple up masters at one time, and the speed is fast. However, there are many difficulties, and it is estimated that the interface problem is still inevitable.
After all, videos are not as easy to download as pictures. Once you have mastered the scrapy method of downloading pictures and files, and solved the difficulties in the interface of station b, you can use scrapy to download videos of multiple up hosts.
To be continued