RabbitMQ message reliability issues, dead letter switches, delayed messages, lazy queues

message reliability

If it is log collection, then if the message is lost in the MQ process (messager processing failure, network failure, server failure, etc.), these messages are insignificant compared with the overall view, not so important, and the loss is lost. If the reliability of the message is considered, the performance will be degraded, and the loss outweighs the gain.
If it is an order service, send a message to MQ, because it is asynchronous, and the order has been notified that the order has been successfully placed, then it must be ensured that the message will not be lost. This must ensure message reliability. Although it is an asynchronous call, it is necessary to ensure that the asynchronous calls to each service must be processed successfully.
Issues to be considered for message reliability:

• The message must be successfully delivered to the consumer
  • The producer is responsible for the part and there will be no problems
  • Consumers are responsible for the part will not go wrong
  • message persistence problem
• The consumer must successfully process the message

The producer ensures that messages are successfully enqueued

The reliability of the message for the producer is responsible for whether the message is successfully sent to the corresponding queue. This process is divided into two parts. The first part is whether the process of sending the message to the switch is successful and whether the switch is routed to the queue successfully. corresponding to 消息确认and消费回执

message confirmation

The message acknowledgment section is divided into three possible cases:

1. The producer failed to send the message to the exchange
2. If the switch is faulty, such as: there is no corresponding switch, it will returnnack
3. Successfully send the message to the specified exchange, returnack

By default, RabbitMQ does not enable message confirmation. The configuration steps are as follows:
4. Enable the message confirmation mechanism in the configuration file

spring:
  rabbitmq:
    publisher-confirm-type: correlated

5. Configuration message, specified as the corresponding failure callback method

  CorrelationData correlationData = new CorrelationData(UUID.randomUUID().toString());// 消息的唯一性
        correlationData.getFuture().addCallback(
            result->{
    
    
                if(result.isAck()){
    
    
                // 成功回调
                    System.out.println("消息成功投递到交换机");
                }
                else{
    
    
                // 失败回调 这里可以进行重试
                    System.out.println("消息未成功投递到交换机");
                }
            },
            ex->{
    
    
                System.out.println("发送消息失败");
            }
        );

When sending a message, pass correlationData the object as a parameter.

rabbitTemplate.convertAndSend("e1","",user,correlationData);

message receipt

The message receipt is returned when an error occurs when the switch routes to the queue.
The configuration steps are as follows:

1. in the configuration file

spring:
  rabbitmq:
    publisher-returns: true # 开启消息回执
    template:
      mandatory: true # true 为自定义消息回执回调  false为直接丢弃掉消息

2. Configure the callback of the message receipt.
   A rabbitMQ object can only be configured with one message receipt. Therefore, in the class, the message receipt will only be executed when the switch fails to route to the message queue.

@Configuration
public class CommonConfig implements ApplicationContextAware {
    
    
    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
    
    
        // 获取RabbitTemplate 
        RabbitTemplate rabbitTemplate = applicationContext.getBean(RabbitTemplate.class);
        // 设置ReturnCallback 
        rabbitTemplate.setReturnCallback((message, replyCode, replyText, exchange, routingKey) -> {
            // 投递失败，记录日志
            log.info("消息发送失败，应答码{}，原因{}，交换机{}，路由键{},消息{}",
                     replyCode, replyText, exchange, routingKey, message.toString());
            // 如果有业务需要，可以重发消息
        });
    }
}

Test :
The test switch fails : send a message to a non-existent switch, because the message is not sent to the switch, so the nack method will be called in the message confirmation to

test that the switch is not routed to the queue , and the switch is bound to a queue, Then delete this queue through the page console

The consumer ensures that messages are successfully dequeued and consumed successfully

When the producer has successfully sent the message to the queue, the producer's mission is complete. The remaining part is pushed to the consumer by the queue, and then the consumer processes the message
, but there are two parts that may go wrong:

1. The consumer failed while getting a message from the queue
2. The messager successfully retrieved the message from the queue, but failed to process it

Consumption Confirmation Mechanism

Either way, the consumer did not successfully process the message. The idea of ​​judgment is: if the consumer successfully processes the message, it will return a confirmation. As long as there is an accident, the queue will not be confirmed, and it will be considered that the consumer has not successfully processed it in the end. message, it will not be deleted from the queue.
Return confirmation is divided into three types:

• manual: return manually, you need to call the api in the code
• auto: Automatically return, through the Aop feature of spring, the code is enhanced, and the proxy class will send a confirmation for it after the code is executed
• none: Get the message and confirm it immediately. As long as the consumer takes out the message from the queue, it will be successfully consumed by default, and the message in the queue will not exist这种方式不能保证消息的可靠性

modification method

• Modify in the configuration file

spring:
  rabbitmq:
    listener:
      simple:
        acknowledge-mode: 模式

Thinking : How many confirmation mechanisms are sent in the process from the consumer taking out the message from the queue to the consumer's successful consumption? Does the consumer send a confirmation when it is taken out of the queue, and then send another confirmation after the consumption is successful, or only send one confirmation after the final successful consumption?
When the queue sends a message to the queue, the status of the message will be set to unacked, and if the consumer sends an acknowledgment, the message will be deleted.

Test: First, let’s take a look at the default situation when a consumer takes out a message from the queue but fails to consume it.
After testing, it was found that consumers failed to consume and would continue to do so 重试. When the stop fails to retry, the message will be re-enqueued. (This is the default)

console information, note that the console here is version 3.9, if the version is too low, it may be different

The biggest problem with this method is: once the consumption fails, the messages in the queue will be continuously pushed to the consumers like kicking a ball, constantly switching from the Unacked state to the ready state, which consumes a lot of MQ resources. As shown in the above figure, the MQ message sending bar The number has soared to 1700 per second
. If the mode is set to nono, as long as the message is taken out by the consumer, even the consumer 消费失败message will be discarded by the queue.
Message retry can also be configured . The default is to keep retrying as long as it fails. constantly 由队列向消费者发送. It can be configured to retry locally according to specified rules.

Consumption failure retry mechanism

The consumer failure retry mechanism is another reliable mechanism for consumption. It does not use the mechanism of returning ack after consumption confirmation, and then the queue receives the confirmation deletion message. Instead, the queue thinks that the consumer will succeed as long as it sends a message to the consumer, and then The message is deleted from the queue (equivalent to none mode), but the consumer fails to consume? It will be retried locally. Note that the retry is performed locally, that is, the queue is only sent to the consumer once, which will not affect the performance of MQ. After successful processing, the consumer will automatically return an ack to the queue. If it fails, it will try locally. After the number of times is exhausted, take corresponding measures according to the corresponding failure strategy

Configuration step:
Change consumer message confirmation to

spring:
  rabbitmq:
    listener:
      simple:
        retry:
          enabled: true # 开启消费者失败重试
          initial-interval: 1000 # 初识的失败等待时长为1秒
          multiplier: 1 # 失败的等待时长倍数，下次等待时长 = multiplier * last-interval
          max-attempts: 3 # 最大重试次数
          stateless: true # true无状态；false有状态。如果业务中包含事务，这里改为false

At this time, test :
When the consumer takes out the message, MQ marks the message as a message Unacked. When the message processing fails, it will retry locally. When the number of retries exceeds the configured upper limit, MQ will still discard the message.
You can interrupt the point to judge how many times you have retried in the end. When the consumption fails, if the upper limit of failure is not exceeded, the callback method will be automatically called again.

When the number of retries exceeds the upper limit, one will be automatically sent reject, and the queue end删除掉消息

Then this brings a problem: if the local multiple attempts still fail, and the reliability of the message is still not guaranteed,
then the failure strategy needs to be modified. The default is to return when the number of failures reaches the upper limit reject, so that the message is deleted in the queue

failure strategy

There are several types of failure strategies

• RejectAndDontRequeueRecoverer: Direct after retries are exhausted reject，丢弃消息. When the local retry is selected, the failure reaches the upper limit, which is the default, as demonstrated above

• ImmediateRequeueMessageRecoverer: After the retries are exhausted, return nack，消息重新入队and kick the ball again

• RepublishMessageRecoverer: After the retries are exhausted, deliver the failure message to the specified exchange

Here is the last processing method: , when the consumer fails to process the message and reaches the upper limit, it will act as a producer and send messages to other switches

Use the third method: the consumer specifies the switch that is forwarded after failure

This method is like I am Party B, but I can't handle it, so I, as Party A, find another Party B to handle
the test : when the producer sends a message to the switch, the switch routes it to the queue, and the consumer takes the message from the queue, Start local processing, fail and re-process. When the processing failure reaches the upper limit, it will be forwarded to the configured failure processing switch.

1. The producer sends a message to the exchange
   
2. The consumer took out the message from the queue, failed to process multiple times, but did not reach the upper limit, and continued to retry
   
   
3. When the number of consumer processing failures reaches the upper limit,
   
   let's check the message forwarded by the consumer in the error queue. We can see that not only the original message content, but also the consumer has added error information to the message.
   

Use a service to listen to this error queue, so that the messages of other service failures can be processed again uniformly

Use the first method: specify a dead letter switch in the queue

Since the use of the dead letter switch is not limited to this scenario, for the sake of clear structure, only a general introduction is given here. The follow-up will introduce in detail
that the dead letter switch is configured in the message queue. When the consumer fails to process messages locally and reaches the upper limit, the first failure strategy is adopted to return reject. The original message queue is the same as ack after receiving this confirmation: directly send The message is discarded, but after the dead letter switch is configured, it will be forwarded to the dead letter switch, and the dead letter switch will be routed to the dead letter queue, and then there is also a consumer dedicated to processing failed messages.

This method is like I am Party B, but I can't handle it, I will tell Party A directly, and Party A will find someone else to be Party B

message persistence problem

Through the producer consumption confirmation mechanism above, the consumer confirmation mechanism has been realized, and the message forwarded by the producer must be successfully processed by the consumer. But if in the middle of the message process, MQ crashes, the message has arrived in the queue, and has been confirmed to the producer, but the consumer does not know that there is a message for it. At this time, the downtime is not persistent, and the entire message is lost. ?
In order to ensure that after the entire link is down, it can still be restored to the state before the downtime, three persistence issues must be considered

• switch persistence
• queue persistence
• Note that the value of message persistence
  is: use Spring AMQP to declare or create 交换机, 队列and send 消息the default 都是持久化.
  Test :
  When I was testing the wrong switch, the declared switch, queue, and messages in the queue were still there. At this time, I used docker to restart RabbitMQ to judge whether it was still there.
  

Of course, when using springAMQP, you can also explicitly declare persistence (also a method of setting non-persistence)

switch persistence

@Bean
public DirectExchange simpleExchange(){
    
    
    // 三个参数：交换机名称、是否持久化、当没有queue与其绑定时是否自动删除
    return new DirectExchange("simple.direct", true, false);
}

queue persistence

@Bean
   public Queue errorQueue(){
    
    
       return new Queue("error.queue",true);
   }

message persistence

So what operations will we have when we only put in a String?

Summary :

As mentioned above, local retries are used after the consumer message processing fails. When the number of retries exceeds the specified number, the consumer can forward the unprocessed message to a certain switch, or send a reject to the queue to let the message Becomes a dead letter and is handled by the dead letter exchange specified by the queue. Here's how to use the dead letter switch for the queue

dead letter exchange

Dead letter exchange, as the name implies, is to send the dead letters in each queue to this 交换机, Dead Letter Exchange is short DLK
for So what is a dead letter? There are three types of dead letters

1. 未被成功消费的消息: The consumer adopts the local retry mechanism, and the failure strategy is the final sending reject. At this time, the message in the queue is not successfully consumed, and it becomes a dead letter
2. 超时的消息: It may be that the queue has set ttl, and the queue timeout may also be that the message has set ttl, and the message has timed out. When the queue times out, all messages in the queue become dead letters, and when the message times out, only this message becomes a dead letter
3. 队列溢出的消息: Because the queue space is limited, when messages are placed in the queue for a long time until the queue has no space to temporarily store these messages, some of the earliest messages will be released and become dead letters, and then new messages will be stored

How to make an exchange a dead letter exchange?
A dead-letter switch is an ordinary switch. For a dead-letter switch 生产者是队列, the queue sent to it specifies its dead-letter switch, and then the dead letters in the queue will be automatically sent to the dead-letter switch.

---

Failed to configure a dead-letter switch

planning

type	name
dead letter exchange	dl.e2
dead letter queue	dl.q2
common switch	e2
normal queue	q2

Configuration: When the consumer processing of the q2 queue fails, it is handed over to the consumer of the dead letter queue dlq2

1. Configuring a dead letter queue is a common set of switches, queues, and binding relationships. And specify the consumer for the queue

  @Bean
    public DirectExchange dle2(){
    
    
        return new DirectExchange("dle2");
    }
    @Bean
    public Queue dlq2(){
    
    
        return new Queue("dlq2");
    }
    @Bean
    public Binding dlbinding(){
    
    
        return BindingBuilder.bind(dlq2()).to(dle2()).with("a1");
    }

 @RabbitListener(queues = "dlq2")

 public void listenDirectQueue2(String msg) {
    
    

  System.out.println("死信队列中"+msg);
 }

2. Configure normal switches and queues and bind them. The queue should specify the dead letter exchange, and then specify its consumers

@Bean
    public Queue q2(){
    
    
        return QueueBuilder.durable("q2")
                .deadLetterExchange("dle2")
                .build();
    }

    @Bean
    public DirectExchange e2(){
    
    
        return new DirectExchange("e2");
    }
    @Bean
    public Binding binding(){
    
    
        return BindingBuilder.bind(q2()).to(e2()).with("a1");
    }

Specify the consumer, the consumer requires to configure the local retry mechanism, and the consumer will handle the failure

3. The producer sends a message to the exchange, which is unaware of the producer
   

Test result:
The q2 queue failed twice, and then the dlq2 queue got the message

---

Configuring a TTL Dead Letter Switch

Set the validity period of the message in the queue, but there is no consumer for the message in this queue, that is to say, the message sent to this queue, because it cannot be consumed, will become a dead letter if it exceeds the validity period of the queue, and then set a dead letter switch for the queue. Then bind a queue for the dead letter switch, and a consumer monitors the queue, so that messages can be sent to the consumer after timing.
Therefore, the TTL dead letter queue can be used to send messages at regular intervals (the queue with a validity period has no consumers, and the message must become a dead letter), and timeout message processing (delivery of unprocessed messages to other consumers).
Of course, not only the validity period can be set for the queue, but also the validity period can be set for the message. The validity period of the queue is equivalent to setting the validity period for all messages in the queue, and the validity period of the message refers to a single message. If it times out, it will be used as a dead letter of this queue

Set all messages in the queue to delay the message Steps: If the consumer function is specified for the validity period queue, it becomes: the timeout message is processed by other switches
Planning

type	name
dead letter exchange	dl.e1
dead letter queue	dl.q1
common switch	e1
normal queue	q1

1. First create a dead letter switch, then bind the queue, and there are consumers listening at the other end of the queue (the dead letter switch is the same as the normal one)

2. Create a switch and a timing queue, and specify its dead letter switch as the value of the previously configured switch.
   
   Note that: the delay queue cannot have consumers, because if there are consumers, the message will not be blocked in the queue and does not exceed the validity period. It will be consumed when it becomes a dead letter

3. The producer sends a message to the switch, and the switch will
   
   set a single message delay for the validity period queue . Here, instead of specifying TTL for the queue, TTL is specified for the message on the producer side. As for the dead letter queue, it is the same as above

For example: put the message into the Message object, specify in the objectTTL

Message message = MessageBuilder
        .withBody("hello, ttl message".getBytes(StandardCharsets.UTF_8))
        .setExpiration("5000")
        .build();

Thinking: 共有几种方式设置消息有效期？
There are two types, one is to specify the validity period for all messages in the queue, and the other is to specify the validity period for a single message. Put the
如何实现订单15分钟有效？
order information into a delay queue, which sets the validity period to 15 minutes, but does not set the consumer, order The information is blocked in this queue for 15 minutes and then handed over to the dead-letter switch, and the consumers of the dead-letter queue corresponding to the dead-letter switch cancel the order according to the information after obtaining the message.

---

Of course, if you want to achieve delayed sending of messages, you can not only use the method of setting the validity period of the queue and then consume it through the dead letter queue, but RabbitMQ also provides a dedicated 延迟消息plug-in. As for the fact that the message queue is full, when a new message is inserted, the oldest message in the queue becomes a dead letter, which will not be demonstrated here. Next, we will introduce the implementation of delayed messages by using plug-ins

delayed message

Application scenarios of delayed messages : delayed messages can be used to set validity period, timing, appointment, etc.
The official provides a DelayExchangeplug-in to implement delayed messages, which is implemented based on switches instead of queues. Although exchanges cannot store messages, plugins are used

Installation of the DelayExchange plugin

1. First, download the plug-in first. Note that the version of the plug-in must be consistent with the version of MQ. The download address is https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/releases
   
2. Then put the plug-in into the MQ plug-in directory.
   Here RabbitMQ is started using Dokcer, and the plug-in directory has been mounted outside at startup.
   Commands when RabbitMQ starts

docker run \
 -e RABBITMQ_DEFAULT_USER=yan\
 -e RABBITMQ_DEFAULT_PASS=1234 \
 -v mq-plugins:/plugins \
 --name mq \
 --hostname mq \
 -p 15672:15672 \
 -p 5672:5672 \
 -d \
 3.9.27-management-alpine

In the command, the MQ plug-in directory is mounted to the named mq-pluginslogical volume. Use the command docker volume inspect mq-pluginsto view the host directory corresponding to the logical volume.

Put .ezthe ending plug-in into this directory

. Why does our plug-in seem out of place, because it needs to be installed

3. Install the plug-in
   Installing the plug-in needs to enter the container for installation, use the command docker exec -it mq bash, and then use the command rabbitmq-plugins enable rabbitmq_delayed_message_exchangeto complete the installation

After the installation is complete
, how is the delay switch used? The generalization is to specify the switch to delayed类型set the delay time of the message.
The delay switch is declared as a type on the basis of the ordinary switch. delayedThen, when sending a message, if the message is sent to this switch, it must be added at the head of the message. Attribute x-delay, the value is the time you want to delay. When sending a message, the delay switch will x-delayknow the delay time according to the value of the message header attribute, and persist the message to the disk, and repost it to the queue bound to it when the time comes.
That is to say, although the switch is declared as a switch 延迟交换机, it is an enhancement on the basis of a normal switch and still has the functions of a normal switch.
Let's use the code to demonstrate this process:
Declare the switch as a delay switch , using the same method of @Bean and annotation

• Use annotations
  
• The way to use @Bean
  

When the producer sends a message, it needs to add an attribute to the request header of the message x-delay. The value is the number of milliseconds of delay.

During the test, it was found that when the message was sent to the delay switch, because the switch did not route the message to the queue in time, the message reliability would be triggered. The message receipt method can be judged according to the returned information, and this situation is ignored

---

Using the delayed message plug-in can replace the use of the dead letter queue. There is another point on the dead letter problem that the oldest message in the queue becomes a dead letter due to insufficient queue space. You can use the switch to receive the dead letter for processing, but how to avoid it? What about this situation?

1. Increase consumer processing power
2. Improving the capacity of the queue
   Improving the processing capacity of consumers is not within the scope of MQ. Here we discuss the second method of improving the message queue.

Lazy Queues Lazy Quenes

What is a lazy queue? A simple understanding is to store the message directly to the disk, and then read it from the disk when it is used.
Why is it called a lazy queue? So disk is slower than memory
so why use lazy queue? The lazy queue exists in the disk. Although the disk is slow, it has a large space, which effectively solves the problem of message accumulation! And the speed is stable. When the inert queue is not used, the messages in the memory will be transferred to the disk after reaching a certain amount. However, the concurrency of MQ in this IO process will decrease, showing a phenomenon of fluctuating. Instead, an inert queue is used, and the messages are directly put into the disk, without intermittency page-out, and the speed is relatively stable.
What are the disadvantages of lazy queues? The disadvantage of the lazy queue is the disadvantage of storing data on the disk: both storage and retrieval must go through the disk, and the timeliness is poor! Disk IO will be worse than memory performance!

The use of lazy queues
Lazy queues are a type of queues, which were MQ3.6released after the version.
The use of lazy queues is also used when declaring queues, and there are also two ways of @Bean and annotations. Note that delayed messages are for configuring switches, and lazy queues are for configuring queues.

• @Bean way
  
• The way of annotation
  
  Sending messages is unchanged, just modify the properties of the queue so that the messages are directly stored in the disk, and then retrieved from the disk when needed. In addition, you can also use commands to modify the existing
  queues to lazy queues
  . For example: I want to modify a common queue named q4 into a lazy queue
  
  inside the MQ container using the commandrabbitmqctl set_policy Lazy "^q4$" '{"queue-mode":"lazy"}' --apply-to queues