The Gecco open source crawler framework was released on December 31, 2015. Since its release, it has been affirmed by everyone in terms of ease of use and scalability. At present, there are 230+ stars and 100+ forks on github. So, how does Gecco's new crawler framework perform in practical applications? In order to let users use the Gecco framework with confidence, the Gecco team released an APP developed using the Gecco crawler framework - Hangout. This application is more used to verify the ease of use, stability and scalability of the Gecco framework. Any framework that is out of practical application is hooligan.
The Hangout app captures more than 10 mainstream e-commerce and shopping guide platforms, including JD.com, Suning.com, Tmall, and what is worth buying. After data cleaning and aggregation, the following functions are realized:
- [Historical Low] Real-time access to e-commerce price dynamics, whoever buys the new historical low earns
- [Worth buying] All the information on what is worth buying on the whole network is included
- [9.9 free shipping] Tmall 9.9 free shipping real-time update
- [Coupon] Collect coupon information on the whole network
The crawler part of the hangout application mainly uses the core of Gecco and the extension framework of Gecco-Spring. Due to the small scale, the distributed crawler of Gecco-Redis is not used, and the extension framework of Gecco-HtmlUnit is not used due to the efficiency problem. The next step will be to use the Gecco-Redis extension framework for distributed crawling after introducing more e-commerce websites to verify the reliability of Gecco-Redis.
The crawler part of the Hangout application has been tested for 7*24 hours of stability. In the future, Gecco's upgrades will be tested on the Hangout application before the version is released.
The APP currently only has an Android client. It can be downloaded by clicking here , or by scanning the QR code below. Interested friends can install and use it.