RabbitMq--- lazy queue

Preface

Message accumulation is a common problem when Mq is consumed. Here we will expand on the reasons for message accumulation and how to solve this problem in RabbitMq.

1. Message accumulation problem

When the speed of the producer sending messages exceeds the speed of the consumer processing the messages, messages will accumulate in the queue until the queue reaches the upper limit of storing messages. The earliest received message may become a dead letter and be discarded. This is the message accumulation problem.
Insert image description here
There are three ways to solve message accumulation:

Add more consumers and increase consumption speed
Open the thread pool in the consumer to speed up message processing
Expand the queue capacity and increase the accumulation limit

The idea of solving message accumulation in RabbitMq is to expand the queue capacity, but it separates message storage from the memory and uses disk.

2. Lazy queue

Beginning with version 3.6.0 of RabbitMq, the concept of Lazy Queues , also known as lazy queues, has been added. The characteristics of lazy queues are as follows:

After receiving the message, it is stored directly in the disk instead of in the memory.
When the consumer consumes the message, it reads it from the disk and loads it into the memory.
Supports millions of message storage

2.1 Command line settings

To set a queue as a lazy queue, just specify the x-queue-mode attribute as lazy when declaring the queue . A running queue can be modified into a lazy queue through the command

rabbitmqctl set_policy Lazy "^lazy-queue$" '{"queue-mode":"lazy"}' --apply-to queues

rabbitmqctl: command line tool for RabbitMq
set_policy: add a policy
Lazy: Policy name, can be customized
^lazy-queue$: Use regular expressions to match queue names
{"queue-mode":"lazy"}: Set the queue mode bit lazy mode
–apply-to queues: The target of the policy is all queues

There are two ways to declare lazy queues in SpringAMQP, one is the @Bean way, and the other is the annotation-based way.

2.2 @Bean-based approach

Use .lazy() to turn on x-queue-mode to lazy

import org.springframework.amqp.core.Queue;
import org.springframework.amqp.core.QueueBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/** 惰性队列配置 */
@Configuration
public class LazyConfig {
    
    

    /** 惰性队列 - 增加了.lazy()属性 */
    @Bean
    public Queue lazyQueue() {
    
    
        return QueueBuilder.durable("lazy.queue")
                .lazy() // 开启x-queue-mode为lazy
                .build();
    }
	
	/** 正常队列 */
	@Bean
	public Queue normalQueue()
	{
    
    
		return QueueBuilder.durable("normal.queue")
			.build();
	}
}

2.3 Based on annotation:

Add a new method in the message listening on the consumer side.
Use @Argument to mark the key of the parameter and the value of the corresponding parameter.

@RabbitListener(queuesToDeclare = @Queue(
        name = "lazy.queue",
        durable = "true",
        arguments = @Argument(name = "x-queue-mode", value = "lazy")
))
public void listenLazyQueue(String msg) {
    
    
    log.info("接收到 lazy.queue 的消息：{}", msg);
}

3. Case test

Here we test the normal queue and lazy queue with 100,000 pieces of data, start the consumer service, and add the following two methods to the SpringAmqpTest class of the publisher service:

/** 测试惰性队列 */
@Test
public void testLazyQueue() throws InterruptedException {
    
    
    // 模拟发送十万条数据
    long b = System.nanoTime();
    for (int i = 0; i < 1000000; i++) {
    
    
        // 1.准备消息
        Message message = MessageBuilder
                .withBody("hello, LazyQueue".getBytes(StandardCharsets.UTF_8))
                .setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT) // 改成非持久化，可以看一下LazyQueue的效果
                .build();
        // 2.发送消息
        rabbitTemplate.convertAndSend("lazy.queue", message);
    }
    long e = System.nanoTime();
    System.out.println(e - b);
}

/** 测试正常队列 */
@Test
public void testNormalQueue() throws InterruptedException {
    
    
    // 模拟发送十万条数据
    long b = System.nanoTime();
    for (int i = 0; i < 1000000; i++) {
    
    
        // 1.准备消息
        Message message = MessageBuilder
                .withBody("hello, Spring".getBytes(StandardCharsets.UTF_8))
                .setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT) // 改成非持久化，可以看一下正常队列的效果
                .build();
        // 2.发送消息
        rabbitTemplate.convertAndSend("normal.queue", message);
    }
    long e = System.nanoTime();
    System.out.println(e - b);
}

Data changes in the RabbitMQ console queue:

initialization:
After running the two producer methods, take a look at the two queue overviews
Lazy queue data changes ( all will be stored on the disk as soon as they come in, and only those that need to be consumed are in the memory )
Normal queue data changes:
Normal queues are all in memory (this is the case where persistence is not turned on)