RabbitMQ Learning (3) "Advanced Use of Message Queue"

The previous article introduced the basic use of RabbitMQ, this article summarizes the advanced use of RabbitMQ

1.TTL (Time o To Live) message expiration time

There are two settings

1. Set message expiration time through queue properties Set message expiration time
   through queue properties All messages in the queue will expire when they are not consumed for more than time.

@Bean("ttlQueue")
public Queue queue() {
    
    
    Map<String, Object> map = new HashMap<String, Object>();
    map.put("x-message-ttl", 11000); // 队列中的消息未被消费 11 秒后过期
    return new Queue("TTL_QUEUE", true, false, false, map);
}

2. Set the expiration time of a single
   message. Specify the message attributes when sending the message.

MessageProperties messageProperties = new MessageProperties();
messageProperties.setExpiration("4000"); // 消息的过期属性，单位 ms
Message message = new Message("这条消息 4 秒后过期".getBytes(), messageProperties);
rabbitTemplate.send("TTL_EXCHANGE", "bread.ttl", message);

If both Message TTL and Queue TTL are specified at the same time, the shorter time will take effect.

2. Dead letter queue

In some cases the message will become a dead letter (Dead Letter) .
When the queue is created, you can specify a Dead Letter Exchange (DLX) .
The queue bound to the dead letter exchange is called DLQ (Dead Letter Queue) . DLX is actually
an ordinary exchange, and DLQ is also an ordinary queue (for example, substitute players are also ordinary players).

Under what circumstances will a message become a dead letter?

1. The message is rejected by the consumer and the re-queue is not set: (NACK || Reject) && requeue== false
2. Message expired
3. When the queue reaches the maximum length and exceeds Max length (number of messages) or Max length bytes (number of bytes), the first message to the queue will be sent to DLX.

How to use the dead letter queue?

1. Declare that the original switch (GP_ORI_USE_EXCHANGE) and the original queue (GP_ORI_USE_QUEUE) are bound to each other. The message in the queue expires in 10 seconds, because there is no consumer, it will become a dead letter. Dead letter queue specified original exchange
   unit (GP_DEAD_LETTER_EXCHANGE).

@Bean("oriUseExchange")
public DirectExchange exchange() {
    
    
	return new DirectExchange("ORI_USE_EXCHANGE", true, false, new HashMap<>());
}
​
@Bean("oriUseQueue")
public Queue queue() {
    
    
	Map<String, Object> map = new HashMap<String, Object>();
	map.put("x-message-ttl", 10000); // 10 秒钟后成为死信
	map.put("x-dead-letter-exchange", "DEAD_LETTER_EXCHANGE"); // 队列中的消息变成死信后,进入死信交换机
	return new Queue("ORI_USE_QUEUE", true, false, false, map);
}
​
@Bean
	public Binding binding(@Qualifier("oriUseQueue") Queue queue,@Qualifier("oriUseExchange") DirectExchange
	exchange) {
    
    
	return BindingBuilder.bind(queue).to(exchange).with("ori.use");
}

2. Declare dead letter exchange (DEAD_LETTER_EXCHANGE), dead letter queue (DEAD_LETTER_QUEUE), bind each other

@Bean("deatLetterExchange")
	public TopicExchange deadLetterExchange() {
    
    
	return new TopicExchange("DEAD_LETTER_EXCHANGE", true, false, new HashMap<>());
}
​
@Bean("deatLetterQueue")
	public Queue deadLetterQueue() {
    
    
	return new Queue("DEAD_LETTER_QUEUE", true, false, false, new HashMap<>());
}

@Bean
public Binding bindingDead(@Qualifier("deatLetterQueue") Queue queue,@Qualifier("deatLetterExchange")
TopicExchange exchange) {
    
    
	return BindingBuilder.bind(queue).to(exchange).with("#"); // 无条件路由
}

3. The final consumer monitors the dead letter queue.
4. The producer sends the message.
   Schematic diagram
   

3. Delay queue

In our actual business, we have some scenarios that need to delay sending messages, such as:

1. There is a smart water heater at home, which needs to be activated in 30 minutes
2. Unpaid orders will close in 15 minutes

RabbitMQ itself does not support delay queues. In general, there are three implementation schemes:

1. Store to the database first, scan with a scheduled task
2. Use RabbitMQ's Dead Letter Queue (Dead Letter Queue) to achieve
3. Use rabbitmq-delayed-message-exchange plugin

1. TTL+DLX realizes delay queue

The above code and diagram of the process have been explained

Message flow process:
producer-original exchange-original queue (after exceeding TTL)-dead letter exchange-dead letter queue
-final consumer

The disadvantages of using dead letter queues to implement delayed messages:

1. If the queue is used to set the TTL of the message, when the gradient is very large, such as 1 minute, 2 minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes...it is necessary to create many switches and queues to route messages.
2. If the TTL of the message is set separately, it may cause the message in the queue to be blocked-the previous message is not dequeued (not consumed), and the following messages cannot be delivered (for example, the first message expires TTL is 30min, the second message The TTL is 10min. After 10 minutes, even though the second message should be delivered, the first message cannot be delivered because it has not been out of the queue.)
3. There may be a certain time error.

2. Implementation based on delay queue plug-in (Linux)

RabbitMQ 3.5.7 and later versions provide a plug-in (rabbitmq-delayed-message-exchange) to implement the delayed queue function. At the same time, the plugin depends on Erlang/OPT 18.0 and above.

Plug-in source code address:
https://github.com/rabbitmq/rabbitmq-delayed-message-exchange
Plug-in download address:
https://bintray.com/rabbitmq/community-plugins/rabbitmq_delayed_message_exchange

1. Enter the plugin directory

whereis rabbitmq
cd /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.12/plugins

2. Download plugin

wget
https://bintray.com/rabbitmq/community-plugins/download_file?file_path=rabbitmq_delayed_message_exchange-0.0.1.e
z

If the downloaded file name has a question mark, you need to change the name

mv download_file?file_path=rabbitmq_delayed_message_exchange-0.0.1.ez
rabbitmq_delayed_message_exchange-0.0.1.ez

3. Enable plugin

rabbitmq-plugins enable rabbitmq_delayed_message_exchange

4. Disable plugin

rabbitmq-plugins disable rabbitmq_delayed_message_exchange

5. Plug
   -in use The delayed-messaging feature is used by declaring an x-delayed-message type of Exchange. x-delayed-message is the type provided by the plug-in, not rabbitmq itself (different from direct, topic, fanout, headers)

@Bean("delayExchange")
public TopicExchange exchange() {
    
    
	Map<String, Object> argss = new HashMap<String, Object>();
	argss.put("x-delayed-type", "direct");
	return new TopicExchange("DELAY_EXCHANGE", true, false, argss);
}

Producer:
Specify the x-delay parameter in the message attribute.

MessageProperties messageProperties = new MessageProperties();
// 延迟的间隔时间，目标时刻减去当前时刻
messageProperties.setHeader("x-delay", delayTime.getTime() - now.getTime());
Message message = new Message(msg.getBytes(), messageProperties);
​
// 不能在本地测试，必须发送消息到安装了插件的 Linux 服务端
rabbitTemplate.send("DELAY_EXCHANGE", "#", message);

4. Server Flow Control (Flow Control)

When RabbitMQ produces MQ messages at a much faster rate than consumes messages, a large amount of message accumulation will occur, occupying system resources, and causing the performance of the machine to decrease. We want to control the number of messages received by the server, what should we do? The queue has two properties that control the length:

• x-max-length : The maximum number of messages stored in the queue. If this number is exceeded, the message at the head of the queue will be discarded.
• x-max-length-bytes : The maximum message capacity (in bytes) stored in the queue. If this capacity is exceeded, the message at the head of the queue will be discarded.

It should be noted that setting the queue length is only meaningful when messages are piled up, and messages that are first entered into the queue will be deleted , which cannot truly achieve server-side current limit.

Is there any other way to achieve server-side current limiting?

4.1 Memory control

RabbitMQ will detect the physical memory value of the machine at startup. By default, when MQ occupies more than 40% of the memory, MQ will actively throw a memory warning and block all connections (Connections). The memory threshold can be adjusted by modifying the rabbitmq.config file. The default value is 0.4, as shown below:

[{rabbit, [{vm_memory_high_watermark, 0.4}]}].

It can also be set dynamically with commands. If set to 0, all messages cannot be published.

rabbitmqctl set_vm_memory_high_watermark 0.3

4.2 Disk Control

Another way is to control the distribution of messages through disk. When the disk space is lower than the specified value (default 50MB), trigger flow control measures.
For example: Specify 30% or 2GB of the disk:

disk_free_limit.relative = 3.0
disk_free_limit.absolute = 2GB

5. Consumer current limit

By default, if not configured, RabbitMQ will send the messages in the queue to consumers as quickly as possible. Because consumers will cache messages locally, if the number of messages is too large, it may cause OOM or affect the normal operation of other processes.

In the case that consumers have limited ability to process messages, such as the number of consumers is too small, or the processing time of a single message is too long, if we want to stop pushing messages before a certain number of messages are consumed, we must use Traffic restriction measures on the consumer side.

The value of prefetch count can be set based on Consumer or channel , which means the maximum number of unacked messages on the Consumer side. When a message exceeding this value is not confirmed, RabbitMQ will stop delivering new messages to the consumer.

channel.basicQos(2); // 如果超过 2 条消息没有发送 ACK，当前消费者不再接受队列消息
channel.basicConsume(QUEUE_NAME, false, consumer);

Spring Boot configuration:

spring.rabbitmq.listener.simple.prefetch=2

Example:
Set the prefetch count of the channel to 5. When the consumer has not sent an ACK to the Broker for 5 messages, RabbitMQ will no longer deliver messages to the consumer.