First, the recommendation system introduced
Micro recommendation system is a huge project, algorithms and integrated business systems, which are divided into three major subsystems:
1) the line recommended subsystem;
2) online recommendation subsystem;
3) effect evaluation subsystem;
Later we will focus on the design and implementation of the above three subsystems.
Second, the line recommendation subsystem
Recommended subsystem and the line is divided into mining module offline , data management tools into two parts.
Mining module offline
Mining module offline, all types of core algorithm implemented mining the line , which reads various data sources, using various data mining algorithms offline embodiments, the initial output of the mining results, and the results saved for mining a format . Typically, the implementation of these strategies are some of the mining run on hadoop platform job, parallel implementation strategies, and mining results saved on hadoop.
Data Management Tools
Data management tools, i.e. DataMgrTools, it is a tool (or service), it is possible to accept some management commands, read certain data format of the line, or memory Redis these data in real time or periodically the hit line , the online service for reading.
Data management tools are not related to a common business tool, it needs to support multiple upload specific format of the data, because the data mining module outputs the line may be stored in files on HDFS, database, or even a specific binary data.
Point of the tool is achieved: good line definition of the data format, the data format of the line, and converting the data do migrate downstream through the API.
Third, the online recommendation subsystem
Recommended subsystem is divided into online display service, streaming service, recommend core strategy module service several parts and so on.
Display Service
Display service, or an access service, which is part of the overall inlet line recommendation system, i.e., the access layer over the entire recommended system, which provides an interface to the upstream side for the upstream traffic call.
Display Service is a stateless service (line subsystem respective services are stateless service), it can be extended to any level, to achieve the service points are: the definition of a good common interface format.
Shunt Service
Diversion services, it is the recommendation system in a very unique is also a very important service, it is the role of upstream over the request, in accordance with different strategies in different proportions, diverted to a different recommendation algorithm experimental platform (that is, recommended kernel downstream) to go.
How to request a diversion services over the judge assigned to the upstream recommendation algorithm experimental platform? The answer is through the policies and configurations. Architecture can be seen from the figure, nearly all services need to read data (data) and configuration (conf), these data may be dynamically changing data online (eg: data read from the redis), may also be It is relatively static data (for example: a list of cities), conf better understanding that some configuration (for example: 80% of all requests must go a flow algorithm experimental platform). Through these policies and configuration, brought over with the request parameters, calculate the shunt flow distribution service to which experimental platform.
Implement the service points are: to achieve universal support and diversion rules or relationships can be configured with downstream applications platform defines common interface to achieve good traffic demand to play a different experimental platform.
Recommended kernel
Recommended core, is all kinds of core online recommendation algorithm implementation , it is only a common experimental platform containers, each recommended internal services may run a different type of recommendation algorithms.
Although the recommended service ran a different recommendation algorithm, but each algorithm implementation steps are the same, we need to go through:
(1) Pretreatment;
(2) Pre-analysis;
(3) de-emphasis filter;
(4) sorting;
(5) Recommended explanation;
Five steps, each step there may be a variety of different algorithms, different models, each step an algorithm combine to complete a complete process, constitute a "recommendation algorithm experimental platform."
For each of said different steps in the different models may need to access different external service module, for example:
Recommended explain step, there may be two models, the first model may need to access in the recommended interpretation stage, "explained -module1- service", the second model may need to access in the recommended interpretation stage, "explained -module2- service", these different model of access to different business needs, at the architectural level need to support.
Highlights of the service is achieved: to run a variety of strategies in the framework of a referral service to support multiple parallel algorithm development engineer / test multiple recommendation algorithm within a framework, with split service implementation recommendation algorithm experimental platform.
Policy Service
Policy Service, also known as policy service module, which implements a recommendation a recommendation kernel module downstream. In recommending the kernel to perform various recommended steps, each step may have different algorithms / strategies, these algorithms / strategies and tactics may need to call a number and more closely bind the service module, they are not universal service, but relative proprietary service.
For example: ordering service module, need to have a convenient, efficient, and scalable service ordered.
Implement the service points are: to achieve a common services framework, so that the algorithm can quickly generate service module, and their needs implemented in the module, and the module can be called convenient services in the algorithm experimental platform.
Fourth, the effect evaluation subsystem
Impact assessment sub-divided into recommendation calling terminal services , reported by the browser end , analysis of the effect ends .
Recommended end call service
Recommended system call interface 58 city lines of business, such as recruitment business line.
Reported by the browser end
Browser js, when you call recruitment services, to show the recommended system recommended that the results page, and be able to know which recommendation results are clicked on, and these will be presented with the information to be reported clicked.
Real-time analysis of end results
The browser will be shown js recommendation result, with the click of recommended results reporting, there is a platform for real-time effects observed, the first time that the effect on the line after recommendation algorithm / recommended strategy.
Fifth, the overall architecture diagram
Integrated the previous sections, the overall architecture of the system 58 of FIG city recommended above.
Recommendation system is an engineering, algorithms, and integrated business systems, on-line recommendation system, from 58 city officially entered the era of intelligent data recommended.
About -58 city Recommended Business
58 city is a user and the merchant were to depend on the platform, recommended information is critical to the city 58. To 58 city recruitment business line, for example: client recruitment, recommended for users more well-related jobs, can enhance the user experience, but also increased the 58 city of PV; at the end of the recruitment business, for businesses recommended more better correlation resume, businesses can enhance the human experience, downloads resume, thereby increasing the income of 58 city.
Recommended business is so important, at the technical level, how to design the system architecture recommendation, the content of this article focuses on.
