C++ Distributed Real-time Application Framework 2.0
Technical exchange and cooperation QQ group: 436466587 Welcome to discuss and exchange
Previous: (6): Containerized Transformation of Large Projects
Copyright statement: The copyright of this article and the technology used belong to the smartguys team. We reserve the right to pursue legal action for plagiarism and unauthorized reprinting!
After the C++ Distributed Real-Time Application Framework (CDRAF) version 1.0 was released, we have made a lot of improvements to the entire framework. At the architectural level, it supports microservice architecture and microservice orchestration. The coupling between CDRAF and business code is further clarified, and all distributed functions no longer need to be concerned by the business side, and are implemented internally by CDRFA. The configuration file on the business side is extremely simplified. Perhaps the next step is to create an automatic code generation function to generate the corresponding business code framework according to the configuration. Re-implemented such as node delay statistics (CDRFA automatically implements delay statistics for each packet, and services do not need to be concerned), single-number log tracking (changed to the scheme of message packet coloring, CDRFA automatically prints logs when it encounters colored packets), and application command functions (give instructions to the application program in the node through the RESTful interface of the system management module) and so on. Provides an interface to comprehensively display the real-time topology diagram of cluster nodes, and supports real-time orchestration of cluster node microservices. A new log framework function has been added to provide high-performance, multi-scenario log output capabilities. Corresponding unit tests have been added in response to these improvements. As a result, the 2.0 version of the C++ distributed real-time application framework also came into being!
1. Cluster real-time topology
The real-time topology diagram shows each node (container instance) of the cluster, the connection line represents the communication direction, and the isolated node represents the node that is not connected to the grid. The interface displays TPS (traffic) and RTT (delay) data in real time. Click on the node to enter the management interface of the container node and do more management of the container node.
2. Microservice Orchestration
According to the plan of the fifth chapter " The Evolution of Microservice Architecture ", we have added a graphical interface to support the orchestration of the microservice architecture. Each large circle represents a type of node, and the small circle represents a microservice port. During orchestration, connect from one node to another node, and select the microservices to be connected to the two nodes to complete the orchestration. It can be choreographed in real time while the cluster is running. After the choreography is completed, the cluster status will also be displayed in the above topology diagram in real time. Of course, new types of nodes can be added according to the needs of the business itself, or new microservice ports can be added to the nodes. It can be seen that each node is independent and uncoupled, and the interaction between nodes is completely carried out through microservice ports, and whether these ports are valid or not is controlled by microservice orchestration.
3. Delay statistics function
The delay statistics function is one of the core data of the distributed framework, which is used to detect the performance of nodes in real time and take corresponding solutions accordingly. It turns out that this function is implemented on the business side. In CDRAF2.0, we put this function into the framework, which can calculate the delay data of all nodes, and the business does not need to care (the business can obtain this function without doing anything).
In CDRAF2.0, we unified the communication model of nodes, and all nodes are composed of Dis (data distribution) and business processing processes. After a service packet arrives at Dis, the framework will enter the current time in the header of the packet. After the service process processes the message and returns to Dis, Dis will calculate the time difference between the two to obtain the delay data. The data of each package is averagely calculated, reported to the status center, and can be displayed in the real-time topology diagram. All this work is done by the framework.
4. Single-number log tracking (packet coloring scheme)
In the original log tracking scheme, after receiving the number tracking command, it will notify all processes of all nodes to pay attention to this number, and turn on log tracking after encountering a packet of this number. This scheme has several major disadvantages:
a) All business processes need to implement a function of number detection and log tracking, and the code will be extremely redundant.
b) When this function is turned on, all nodes in the cluster and all processes in the node will process and monitor whether there is a status of this number. The entire cluster is in a very unstable state.
c) The business may not support tracking multiple numbers at the same time.
To this end, we adjusted the single-number log tracking scheme and adopted the scheme of packet coloring. Detect the number at the location of the message entry. Once the conditions are met, the message packet will be dyed with the header. After receiving the packet, the framework of the subsequent processing link will detect the header before the service. If the header is found to be dyed, the log tracking will be turned on. This Close after the package is processed. When each link is downloaded, the framework will automatically re-dye the downloaded packets to ensure that each link in the processing process receives the dyed packets. The advantages of the new scheme are as follows:
a) It is no longer necessary to implement the corresponding code for this function in the business, as long as the number monitoring is performed at the entry position, and if it matches, the framework is called for coloring, and all subsequent operations are completed by the framework.
b) When this function is turned on, the cluster no longer handles the unstable state, and the business level does not even feel the existence of this function, and the framework handles all the problems.
c) When a new business node is added, this function will be automatically obtained without any modification.
d) In addition to the single-number log tracking function, this solution can also be applied to other business scenarios.
5. General instruction transmission scheme
In CDRAF, if the outside world wants to issue an instruction to a process of a node in the cluster, it will pass the RESTful interface of the system management module, and then pass through the state center, and the communication platform will finally pass it to the corresponding process. But in our previous solution, each additional such command needs to add corresponding code to each module (system management, status center, communication platform) to support it.
In the new scheme, we design a general message path to pass instructions.
The communication framework automatically adds the MonitorMsgInterface endpoint to the SmartAgent process without configuration; the communication framework automatically adds the MonitorMsgInterface endpoint to the SmartMonitor process without configuration; SmartAgent sends business control messages from its own MonitorMsgInterface endpoint, and SmartMonitor receives and processes it from its own MonitorMsgInterface endpoint;
Unified command format
message MONITOR_MSG { string dest = 1 ; bytes msg = 2; }
Among them, dest is the destination to which the control message is to be sent, fill in the process name of the specific destination process, such as "OLCProxy", "OCPro", "OCDis", etc.; if the command needs to be sent to all processes, dest is filled with characters String all. The msg field is the specific control command text.
The framework itself provides several public control commands, enumerated as follows:
Adjust the log level
format: log log level
Among them, the log level includes off , critical , err , warn , info , debug , trace
Examples: log debug
stop thread
format: stop
heavy Communication link information
format: reload
In addition to the public control commands provided by the above framework, SmartMonitor can also receive any message and pass it to the specified process, it does not care about the content of the message, and the content of the message is completely parsed and processed by the business process. In principle, when SmartAgent fills in the value of this field, it should read it directly from ZK; there should be the text of the control command on ZK.
As can be seen from the above adjustments, CDRAF 2.0 is committed to the complete decoupling of distributed related functions and services. In our design and implementation, there is a clear dividing line between business and framework. All the functions that can be done on the framework side can be obtained automatically without writing a line of code on the business side. I think this is a big step for CDRAF 2.0 to be a universal framework!