Reprinted: http://blog.csdn.net/Chay_Chan/article/details/75319452
original dream
I have always wanted to try to develop a certain type of APP that I have not been exposed to yet. When I first started, I had a dream: to develop social, map, news, payment, e-commerce, live broadcast, and game APPs. When I was a sophomore in the social class, I had already developed an instant messaging APP, using the SDK of NetEase Yunxin; I have also been in touch with maps, payments, and e-commerce; in terms of games, I have done a Plants vs. Zombies demo, which is considered Cocos2D Before the news and live broadcast, the company has not received the development of these projects, so I have never had the opportunity to contact, so I have always wanted to realize the original dream and complete the news and live broadcast APP.
Today, I bid farewell to my alma mater in July, officially graduated, and also resigned. In my sophomore year, I started a company and took over the project with my classmates. It has been a year now. The construction started in more than June. At that time, I made time to write code after get off work. I used the current popular framework, RxJava + Retrofit + MVP. I was close to completing the functions I set in early July. to some problems.
problems encountered
Question 1: Toutiao closed the old interface. Originally, it was based on the Toutiao made by the great god Wey Ye. He obtained the news list according to the interface he knew from the previous packet capture. The background interface could not get the data, so we had to find a way to solve it. Although I have used fidder to capture packets before, but I am not very familiar with it, I am helpless when encountering some https encrypted data packets. Later, I learned that fidder can decode https data packets, and today's headline news data acquisition problem is easily solved;
Question 2: The rules of news data, there are many kinds of item layouts in the news list. In the absence of interface documents, it is necessary to analyze the returned json data, find the rules, and know which field to judge which layout belongs to. This is also Quite time consuming.
Question 3: Video parsing, the video parsing method used in the original reference project is also inexplicably invalid at this time, so I have to find a new method for parsing video myself, and find a lot of methods, some of which are to read the news details page html page, through regular matching to the address of the corresponding video, there are other methods, but I still feel unsatisfactory, I know that I have found a website that parses the address of today's headline video, and the corresponding URL can be obtained by passing in the URL of the details page. The address of the video, so start capturing packets to get that interface, and the problem of video parsing is easily solved.
Bad luck, bad luck
But problems always come one after another. I thought I could happily write a blog today, share this open source project I wrote, and communicate with each other. Who knew that when the video was played in the morning, the video address was parsed wrong again. At this time, My heart is broken. Only after debugging did I know the original interface for parsing video. The author verified the access to the interface. It is estimated that it is to prevent others from directly calling his interface. It must be accessed through his website. After analyzing his js code, I found him Two new parameters have been added to the interface. The parameter names are meaningless. They are "r" and "s", "r" is a random number, and "s" is based on the video address + "@" + r after splicing, Perform a series of processing, AND operation, left shift, right shift, and then generate the corresponding one-end number. The background should also be verified according to this algorithm to verify whether r and s are valid.
After knowing that the author has added verification, I also know the encrypted js code. I wanted to convert the encrypted js function into the method of writing java code, but I found that it was too complicated, so I gave up this idea and found that java can read it. Take the js script, but it can't be used in android. There is no corresponding ScriptEngine class. We have to load the local js script through WebView for parsing
, realize the intermodulation between java code and js, and then obtain the encrypted two parameters generated by js"r " and "s" values.
completed
At present, the functions of the previous top and bottom have been completed, such as the multi-item layout display of news (news list), video list, news details, video details, click on the tab to pull down to refresh, view the comment list, etc. I am very happy to be able to work on projects that I like to do, not for work, but purely to get in touch with some new things. I try to make an app that I have not developed before. I am also looking for a new job recently, although I have already interviewed for a few jobs, But in the end, I refused because the salary did not meet my requirements and the company's development prospects. I felt that I shouldn't be in a hurry to find a job.
introduce
Fine imitation of Toutiao, the data is to capture the data of Toutiao App. Open source project developed with RxJava + Retrofit + MVP for learning purposes only.
Project github address: https://github.com/chaychan/TouTiao
Thanks to the great god wey ye, the project referred to his imitation today's headline project, his project address is: https://github.com/yewei02538/TodayNews
Apk download address
[Click to download experience ]
Demo screenshot
front page
video
Micro headlines
mine
gif
News list, video list
Non-Video News Detail Page
Video playback, video details page
Third-party libraries used
Existing functionality
1. Get news lists of various channels, including video and non-video news;
2. View news details, including details of video and non-video news;
3. View the list of news comments;
4. News data is stored locally, the news data that has been obtained is saved in the local database, and historical news can be viewed when pulling up and loading more;
5. Click on the bottom tab to pull down to refresh;
6. Stop playing when the video in the video list playback is not visible.
Technical points
1. Crawling and analysis of news data, use fidder to crawl, please refer to the following URL for specific use:
http://www.tuicool.com/articles/BJjQZf
2. The display of various Item layouts in the news list uses the BaseRecyclerViewAdapterHelper, the ReclerView universal adapter, and the use of multiple layouts can refer to the following URL:
https://github.com/CymChad/BaseRecyclerViewAdapterHelper/issues/968
3. The parsing of the video source address, the original parsing with reference to the great god wey ye has been invalidated. Currently, it is parsed through the URL of today's headline video parsing , and the json data corresponding to the video download address is obtained by passing in the address of the news details page.