foreword
Jingmai News is the core component of the Jingmai Merchant Open Platform. From the news source to the message center to reaching users, and finally calling up the operation page according to the message protocol, Jingmai News is a complete and healthy ecological closed loop. Below I will introduce in detail how Jingmai news has been continuously improved in the evolution.
Jingmai message frame
I will describe the growth of Jingmai News in 2017 from five aspects: message access, MC system construction, message configuration, message reach, and message monitoring.
1. Message access
The disadvantages of the original message access mainly include the following two points:
1. Diversified message access methods. Jingmai news includes business system news, service information news and other types of news, and the sources of news are various. At that time, in order to quickly access various news sources, servlet access, client access, JMQ access, etc. were provided. The access methods were diversified, and there was no perfect monitoring system, which led to a very embarrassing problem. We ourselves don't know how many types of messages are connected to our messaging system.
2. The message processing center is strongly dependent on the message source. Anycall is the main entrance of system messages. From Anycall to the original message processing background is realized through servlet calls, and the coupling between systems is too strong.
Improvements made to message access:
1. All system messages are uniformly accessed by Anycall to clarify message type boundaries.
2. The access method of Jingmai news is unified. All Jingmai messages are uniformly accessed asynchronously through JMQ, and are isolated through different topics according to different services, avoiding the blocking of other services by services with large data volumes (such as order messages).
3. The completion of projects such as the creation of Maiquan and the access of Dongdong offline messages has enriched the ecology of Jingmai News and greatly increased user stickiness.
2. Construction of MC (Jingmai Message Center) system
The main pain point of the original Jingmai news:
1. The access method is not uniform.
2. Unstable, big promotion was downgraded.
3. The message processing logic is complex, and it is difficult to access new message sources.
4. There is no perfect news tracking and news statistics.
Based on the above reasons, a stable and dedicated message processing center, the MC system, has been rebuilt.
1. Unified JMQ access has been introduced in the previous part.
2. The MC system is not coupled with other systems, and there is no problem that the large amount of messages will affect other businesses of Jingmai, and it can provide stable services during the big promotion.
3. The MC system uses the broker distribution mode. The modular and pluggable processing method makes the access of new message sources extremely simple, which greatly shortens the development cycle. It is precisely the existence of this broker distribution model that projects such as dongdong offline messages and ISV message subscriptions have achieved rapid access and provided services.
4. During the construction of the MC system, the full-link message tracking and message statistics have also been implemented (the message monitoring will be explained in detail in Section 5).
3. Message Configuration
Message filtering, message assembly, message storage, and message push are the four cores of Jingmai Message Center. Message assembly is carried out according to the different configurations of different messages, and these configurations are configured in the config configuration center on the development side. Therefore, the workload of products or operations that want to access a new system message from Anycall is extremely large. of.
For this reason, we unified all configuration links into one page. To obtain configuration information, add a three-tier cache (Guava Cache+redis+DB) to deal with massive calls. The existence of the unified configuration page makes the access of business system messages simple and fast.
Another big optimization is the configuration of the call-up protocol. The call protocol of the previous message was written in the message body, which was extremely inflexible. Even many system messages could not be connected to the call protocol and directly exposed the link in the message body. The user experience was very bad. To this end, the call protocol is connected to the unified protocol management center (which will be described in detail in the following article), and all call protocols will be obtained from the unified protocol management center according to the protocolID carried in the message. The centralization and configuration of the call protocol makes it unnecessary to pay attention to the specific call protocol in the process of message flow in the system, which simplifies the processing logic of the message in the system. Moreover, after the agreement is centralized, the content of the agreement can be directly presented to products and operations, and the whole process of message calling becomes clearer.
4. Message Reach
Jingmai message reach is divided into online notification and offline notification.
Online notification is achieved through the TCP long connection between the server and the client.
Offline notifications were only pushed by apns of IOS at the beginning, and the inability of the Android system to push offline notifications well has always been a major pain point.
In this case, we developed an open source package for Android push, connected with the three major manufacturers of Huawei, Xiaomi, and Meizu, and realized the push of Android offline notifications.
5. Message monitoring
The full-link message tracking system integrates the message flow status of each node of the entire link from the message source to the final message push, and stores it asynchronously. As can be seen from the above figure, the processing method in the system is to subscribe to the same topic of JMQ and store the message logs in ES and HBase respectively. Storing in ES ensures that I can track all messages clearly and transparently in the message management background. Queries and HBase are stored for long-term preservation of data and further analysis.
News statistics are realized by relying on the big data platform of JD.com. Import the data in HBase into the Jingdong data mart, so as to perform statistical analysis on the message data in various dimensions.
Summarize
After a year of growth, Jingmai News has made great progress in terms of stability, monitoring, and rich content. The next plan is to complete the message failure retry mechanism, improve the message delivery rate, and commercialize the message push.
Jingmai is a young and energetic team. The Jingmai messaging system has been continuously improved and optimized along with the growth of Jingmai. To provide users with better service is our eternal pursuit.
Here I recommend an architecture learning exchange group to everyone. Communication and learning group number: 744642380, which will share some videos recorded by senior architects: Spring, MyBatis, Netty source code analysis, high concurrency, high performance, distributed, microservice architecture principles, JVM performance optimization, distributed architecture Wait for these to become the necessary knowledge system for architects. You can also receive free learning resources, which are currently benefiting