First, the reason for the backlog of messages
The direct cause of the backlog of messages, the system must be a part of the performance problems , no time to process messages sent upstream , will cause the message backlog.
Second, optimize performance to avoid the backlog of messages
In a system using message queues, for optimal performance, mainly in the business logic of the producers and consumers of the two parts . For performance message queue itself, as users do not need to be too concerned. The main reason for the vast majority of businesses using the message queue, the message queue itself is much larger than the processing power handling capability of business systems. So for optimizing the performance of message queues, more concerned at both send and receive messages, and how our business code with message queues, to achieve the best performance.
1, the transmitting end performance optimization
The transmitting side service processing performance of the code, and in fact little to message queue, because the transmitting end are generally performed first own business logic, and finally send the message. If you send a message code performance can not go up, you need to give priority check, before the message is not caused by too much time-consuming business logic.
Business logic for sending messages, need to pay attention only to set the appropriate batch size and complicated , we can achieve good transmission performance.
Producer process of sending a message, a message to Producer Broker, Broker returns an acknowledgment response after receiving the message, this is a full interactivity. Assuming this average delay time interaction is 1ms, 1ms time we put unbundled, it includes the following time-consuming these steps:
- Preparing the data transmitting end, the sequence logic of the message, and other configuration request time, i.e. transmission takes before sending the request the network terminal;
- Returning a response and time-consuming to send messages in the network transmission;
- Broker message processing delay.
If the transmission is a single-threaded, each transmitting only one message, the second transmission only 1000ms / 1ms * 1 Article / ms = 1000 messages, this case does not play all of the strength of the message queue. Whether it is to increase the batch size each time you send a message, or an increase concurrency, can exponentially enhance performance sent.
As in the end is to select batch transmission or concurrent increase, depending on the nature of the business program of the sender. In short, as long as you can meet the performance requirements, how to realize how easy it is to achieve.
- For example, your end is a micro-messaging service, a major recipient RPC request processing online business . Naturally, the micro-service when dealing with each request, it sends a message directly on the current thread on it , because all RPC frameworks are multi-threaded support multi-concurrent, parallel to achieve a naturally send the message . And online business care about is to request a response delay, select batch transmission delay is bound to affect RPC services. This case, a more sensible way to improve is through concurrent transmit performance.
- If your system is an off-line analysis system , the demand for off-line system performance is that it does not care about the delay, pay more attention to the overall system throughput. Data are transmitted from the database side, such a case is more suitable for bulk transmission , you can read the data from the database bulk and bulk to send a message, the same can be obtained very high throughput with a small amount of concurrency.
2, the consumer end performance optimization
When using message queues, most of the performance problems have emerged in the consumer end, if the consumer can not keep up the speed of transmission speed production end of the message, the message will cause the backlog. If the consumption rate has been slower than the speed of production, a long time, the whole system will be a problem. Either fill the message queue storage can not provide the service, or message loss.
So when designing the system, we must ensure that the performance of the consumer side of the consumer than sending the performance of the production side , so the system can continue running health.
Performance Optimization addition to optimizing the consumer side of the business logic of consumption, but also through horizontal expansion, increasing the number of concurrent end consumer to enhance the overall performance of the consumer. Note: The number of instances of expansion Consumer same time, the topic of the partition must be synchronized expansion (also called queue) number, to ensure that the number of instances and the number of partitions Consumer are equal. If the number of instances of Consumer exceeds the number of partitions, such expansion is in fact no effect, because for consumers, in fact, can only support single-threaded consumption on each partition.
Many consumer programs, they are so slow to solve the problem of consumption:
It service logic processing the received message may be slower, but also difficult to optimize, in order to avoid message backlog, the method OnMessage received message, does not process any business logic, this message into a memory queue which returns the . Then it can start many businesses threads, these threads there is a real business logic processing business messages , message processing threads taken from the memory queue, so that it can not solve the problem of a single Consumer parallel consumption.
But this is a very common error method ! Why wrong? Because it will lose the message . If the node has received the message down, it will not be lost in the memory queue and to process these messages .