The message backlog is normal, and the more and more backlogs need to be dealt with.
The main reason is that for most businesses that use message queues**, the processing capacity of the message queue itself is much greater than that of the business system**. For a single node of the mainstream message queue, the performance of message sending and receiving can reach the level of processing tens of thousands to hundreds of thousands of messages per second, and the processing capacity can be doubled by horizontally expanding the number of instances of Broker.
The business logic that a general business system needs to process is far more complex than that of a message queue. A single node can process hundreds to thousands of requests per second, which can be regarded as very good performance. Therefore, for the performance optimization of the message queue, we are more concerned about how our business code works with the message queue at both ends of the message sending and receiving to achieve the best performance .
1. Performance optimization of the sender
The processing performance of the service code of the sender is actually not related to the message queue, because the sender generally executes its own business logic first, and finally sends the message. If you say that the performance of your code sending messages does not improve, you need to check first, is it because the business logic before sending the message takes too much time.
2. Consumer performance optimization
When using message queues, most of the performance problems occur on the consumer side. If the consumption speed cannot keep up with the message production speed of the sender, it will cause a backlog of messages. If the problem of this performance upside-down is only temporary, then the problem is not big. As long as the performance of the consumer side recovers and exceeds the performance of the sender side, the backlog of messages can be gradually digested.
If the consumption rate has been slower than the production rate for a long time, the entire system will have problems. Either the storage of the message queue is filled and cannot provide services, or the message is lost. This is a serious failure for the entire system.
Therefore, when we design the system, we must ensure that the consumption performance of the consumer end is higher than the transmission performance of the production end, so that the system can continue to run healthily.
In addition to optimizing the consumer business logic, the performance optimization of the consumer end can also improve the overall consumer performance by horizontal expansion and increasing the number of concurrency on the consumer end. One point that needs special attention is that while expanding the number of Consumer instances, the number of partitions (also called queues) in the expansion topic must be synchronized to ensure that the number of Consumer instances and the number of partitions are equal. If the number of Consumer instances exceeds the number of partitions, such expansion is actually ineffective. We talked about the reason before, because for consumers, each partition can actually only support single-threaded consumption.
Sudden news backlog
Either production is fast or consumption is slow.
For a large amount of business growth in a short period of time
, 1. Horizontal expansion, increase the number of consumer terminals
2. Business degradation, reduce unnecessary business
There is also a situation where the efficiency of production and consumption has not changed, but there is news that causes repeated consumption and occupies threads. At this time, you need to check the stack log and consumption log to locate the problem message.
Some message middleware provides a " dead letter queue" function , which automatically throws this kind of repeated consumption failed messages to a special dead letter queue In, avoid the situation of a message card main queue.