Article directory
1. Background
In the message processing process, if the consumption speed of the client cannot keep up with the sending speed of the server, there will be more and more unprocessed messages, and these messages are called accumulated messages. The accumulation of messages will cause delays in message consumption. The following scenarios need to focus on the problem of message accumulation and delay:
- The continuous accumulation caused by the mismatch between the upstream and downstream capabilities of the business system cannot be recovered by itself.
- The business system has high requirements on the real-time consumption of messages, and even the message delay caused by short-term accumulation is unacceptable.
Two, MQ message accumulation
MQ message accumulation means that the messages sent by the producer accumulate in a large number on the Broker side in a short period of time, and cannot be consumed by consumers in a timely manner, resulting in the failure of normal use of business functions.
3. Message accumulation is common in the following situations:
- The newly launched consumer function has a bug, and the message cannot be consumed
- The consumer instance is down or temporarily unable to establish a connection with the Broker due to network problems.
- The producer pushes a large number of messages to the Broker in a short period of time, and the consumer's consumption capacity is insufficient.
- Producers are not aware of Broker consumption accumulation, and continue to push messages to Broker.
Fourth, to solve the above problems, it is necessary to
- To solve the first problem, do a good job of gray scale release. Before each new function goes online, select a certain proportion of consumption instances to do gray scale (grey scale test is to select a specific group of people to use a product or application before it is released, and gradually expand its use). If there is a problem, it will be rolled back in time; if the consumer consumption is normal, it will run smoothly for a period of time before upgrading other instances.
- To solve the second problem, it is necessary to be more active. In extreme cases, when all consumption instances in one IDC (Internet Data Center?) are down, it is necessary to allow consumption instances in other IDCs to consume messages normally. At the same time, if all Brokers in an IDC are down, it is necessary to support producers to send messages to Brokers in other IDCs.
- To solve the third problem, to enhance the consumption capacity, it is mainly to increase the number of consumer threads or increase the number of consumer instances. To increase the number of consumer threads, we must pay attention to the consumption capacity of consumers and their downstream services. Before going online, the parameters of the thread pool should be adjusted. to the optimal state. To increase the number of consumer instances, pay attention to the number of queues. The number of consumer instances must be the same as the number of queues. If the number of consumer instances exceeds the number of queues, the extra consumer instances will not be assigned to the queue. It is useless to increase the number of consumer instances only. , if the number of consumer instances is less than the number of queues, the traffic carried by each consumer instance is different.
- To solve the fourth problem, it is necessary to fuse and isolate. When there is a backlog of messages in a Broker's queue, it needs to be fused, isolated, and new messages are sent to other queues. After a certain period of time, the isolation is released.
5. How to solve the problem of message accumulation and delay
To quickly avoid the impact of message accumulation and delay on the business, you can use the monitoring and alarm function provided by the message queue RocketMQ version, set alarm rules to warn of message accumulation problems in advance, or trigger alarm events through business burying points, and monitor them in time Messages pile up and deal with them.
