Table of contents
1.1. Message accumulation problem
1.2. Solution to message accumulation problem
1. Lazy queue
1.1. Message accumulation problem
When the speed of the producer sending messages exceeds the speed of the consumer processing the messages, messages will accumulate in the queue until the queue reaches the upper limit of message storage.
It's like having a reservoir that is filled with water and drained out at the same time, but the filling rate is faster than the draining rate, so the pool will eventually fill up.
Then, the earliest received message may become a dead letter. By default, the dead letter will be discarded. After discarding, there will be space in the queue, and new messages can be entered into the queue.
1.2. Solution to message accumulation problem
From a consumer perspective:
1. Add more consumers and increase the consumption speed.
If there is only a single consumer and the processing speed is 100ms per message, then if there are three consumers, the theoretical processing speed is 300ms per message.
2. Open a thread pool in the consumer. Whenever a message comes, it is handed over to a thread for processing. Then the current processing speed of the message within the consumer will be accelerated.
However, this method has a limitation, that is, if there are a lot of messages, you will need to allocate many threads. The more threads, it is also a waste for the CPU, because the CPU needs to do context between multiple threads. Switching, so this solution is more suitable for situations where the message processing business is time-consuming. Multiple threads can be opened to allow the CPU to process in parallel.
From a queue perspective:
3. Expand the queue capacity and increase the upper limit of accumulation.
It's like a reservoir. I made this reservoir as big as the Yellow River. If you put more water into it, when will it be filled~
We are already familiar with the first two methods, but in the end, how to expand the queue capacity?
Lazy queue can be used to solve this problem~
1.3. Introducing lazy queue
1.3.1. What is a lazy queue?
For traditional queues, if message persistence is not enabled, all received messages are placed in memory. The purpose is to speed up message delivery. This is also a great advantage of RabbitMQ - fast response speed.
But it also brings a problem. RabbitMQ sets a memory warning value (the upper limit of memory storage, the default is 40%). If messages accumulate, it is easy to reach this warning value. At this time, RabbitMQ will Being in a paused state will prevent the producer from delivering messages, and then clean up some of the messages in the memory and flush them out to the disk. This action is also called "page out". This will cause the concurrency of mq to fluctuate . The performance is unstable (each page out will be time-consuming and pause for a period of time).
The lazy queue is specially used to deal with the problem of message accumulation~
He has the following three characteristics:
1. Received messages will be written directly to disk instead of memory, so it is difficult to trigger mq's memory warning, and page out will almost never occur.
2. Consumers consuming messages from the lazy queue also need to read from the disk and load them into the memory. In fact, this will also make the delay slightly higher. After all, there is still a big gap between the performance of the disk and the memory, but it is also Within the receivable range.
3. Supports millions of message storage, also because it is disk storage (very large space).
1.3.2. Use of lazy queue
There are three ways to set a queue as a lazy queue:
1. In RabbitMQ Managerment, we only need to specify the x-queue-mode attribute as lazy when declaring the queue.
2. Use SpringAMQP to declare a lazy queue, using @Bean as follows:
@Configuration
public class LazyConfig {
@Bean
public Queue lazyQueue() {
return QueueBuilder
.durable("lazy.queue")
.lazy() //开启 x-queue-mode 属性为 lazy
.build();
}
}
3. Use SpringAMQP to declare a lazy queue with the following annotation:
@RabbitListener(queuesToDeclare = @Queue(
name = "lazy.queue",
durable = "true",
arguments = @Argument(name = "x-queue-mode", value = "lazy")
))
public void listenLazyQueue(String msg) {
log.info("消费者接收到了惰性队列的消息! msg=" + msg);
}
1.3.3. Effect demonstration
a) Use @Bean to declare a normal queue and a lazy queue to compare the effects.
@Configuration
public class LazyConfig {
@Bean
public Queue lazyQueue() {
return QueueBuilder
.durable("lazy.queue")
.lazy() //开启 x-queue-mode 属性为 lazy
.build();
}
@Bean
public Queue normalQueue() {
return QueueBuilder
.durable("normal.queue")
.build();
}
}
b) Use two producers to send 100,000 messages to two queues respectively.
@Test
public void testLazyMessage() {
for(int i = 0; i < 1000000; i++) {
//1.准备消息
Message message = MessageBuilder.withBody("hello lazy message".getBytes())
.setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT)
.build();
//2.发送消息
rabbitTemplate.convertAndSend("lazy.queue", message);
}
}
@Test
public void testNormalMessage() {
for(int i = 0; i < 1000000; i++) {
//1.准备消息
Message message = MessageBuilder.withBody("hello normal message".getBytes())
.setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT)
.build();
//2.发送消息
rabbitTemplate.convertAndSend("normal.queue", message);
}
}
The general queue situation is as follows:
Each time a normal queue writes a certain amount of data to memory, it pauses for a period of time and writes data to the disk (page out).
The lazy queue situation is as follows:
Lazy always writes data to the disk, so total is equal to paged out.
