RabbitMQ message reliability problem and solution

Description: During the RabbitMQ messaging process, there are the following problems:

• message not sent to switch

• message not sent to queue

• MQ is down, messages are lost in the queue

• After receiving the message, the messager fails to consume normally (the program reports an error), and the message has been removed from the queue at this time

In view of the above problems, the following solutions are provided:

• Message confirmation: confirm whether the message is sent to the switch or queue;

• Message persistence: persistent messages to prevent message loss caused by MQ downtime;

• Consumer message confirmation: Confirm that the consumer has correctly consumed the message before deleting the message from the queue;

message confirmation

The publisher confirm mechanism provided by Rabbit MQ can be used to avoid the loss of messages sent to MQ. The specific implementation is, publisher-confirm (sender confirmation), publisher-return (sender receipt), the former judges the message to the switch, and the latter judges the switch to the queue

---

publisher-confirm (sender confirmed)

• The message is successfully delivered to the switch and returns ack;

• The message is not delivered to the exchange, return nack;

publisher-return (sender receipt)

• The message is delivered to the exchange, but not in the queue, and ack is returned, which is the reason for the failure;

---

Add configuration on the producer side

spring:
  rabbitmq:
    # rabbitMQ相关配置
    host: 118.178.228.175
    port: 5672
    username: root
    password: 123456
    virtual-host: /

    # 开启生产者确认，correlated为异步，simple为同步
    publisher-confirm-type: correlated

    # 开启publish-return功能，基于callback机制
    publisher-returns: true

    # 开启消息路由失败的策略，true是调用returnCallback方法，false是丢弃消息
    template:
      mandatory: true

publisher-return (sender receipt) code

import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.context.annotation.Configuration;

/**
 * 发送者回执实现
 */
@Slf4j
@Configuration
public class CommonConfig implements ApplicationContextAware {
    
    
    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
    
    
        // 获取RabbitTemplate对象
        RabbitTemplate rabbitTemplate = applicationContext.getBean(RabbitTemplate.class);

        // 设置ReturnCallback
        rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {
    
    

            /**
             * 回执信息
             * @param message 信息对象
             * @param replyCode 回执码
             * @param replyText 回执内容
             * @param exchange 交换机
             * @param routingKey 路由键值
             */
            @Override
            public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
    
    
                log.info("消息发送队列失败=====replyCode{}，replyText{}，exchange{}，routingKey{}，message{}",replyCode,replyText,exchange,routingKey,message);
            }
        });
    }
}

publisher-confirm code

    @Test
    public void sendExceptionMessage() {
    
    
        // 路由键值
        String routingKey = "exception";

        // 消息
        String message = "This is a exception message";

        // 给消息设置一个唯一ID
        CorrelationData correlationData = new CorrelationData(UUID.randomUUID().toString());

        // 编写confirmCallBack回调函数
        correlationData.getFuture().addCallback(new SuccessCallback<CorrelationData.Confirm>() {
    
    
            @Override
            public void onSuccess(CorrelationData.Confirm confirm) {
    
    
                if (confirm.isAck()) {
    
    
                    // 消息发送交换机成功
                    log.debug("消息送达至交换机成功");
                } else {
    
    
                    // 消息发送交换机失败，打印消息
                    log.error("消息未能送达至交换机，ID{}，原因{}", correlationData.getId(), confirm.getReason());
                }
            }
        }, new FailureCallback() {
    
    
            // 消息发送交换机异常
            @Override
            public void onFailure(Throwable ex) {
    
    
                log.error("消息发送交换机异常，ID:{}，原因{}", correlationData.getId(), ex.getMessage());
            }
        });

        rabbitTemplate.convertAndSend("amq.direct", routingKey, message, correlationData);
    }

Test, set a routingKey that does not exist, which is captured by the sender confirmation (publisher-confirm);

// 路由键值
String routingKey = "null";

Set a non-existent route, which is caught by the sender's receipt (publisher-return);

rabbitTemplate.convertAndSend("null", routingKey, message, correlationData);

message persistence

Message persistence refers to saving messages to disk. When RabbitMQ is down or shut down, the messages can still be saved after restarting. Messages depend on switches and queues, so persistent messages also need to persist switches and queues.

Create a persistent exchange, queue

import org.springframework.amqp.core.*;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * 消息持久化
 */
@Configuration
public class DurableConfig {
    
    

    /**
     * 交换机持久化
     * @return
     */
    @Bean
    public DirectExchange directExchange(){
    
    
        // 三个参数分别是：交换机名、是否持久化、没有队列与之绑定时是否自动删除
        return new DirectExchange("durable.direct",true,false);
    }

    /**
     * 队列持久化
     * @return
     */
    @Bean
    public Queue durableQueue(){
    
    
        return QueueBuilder.durable("durable.queue").build();
    }

    /**
     * 交换机与队列绑定
     * @return
     */
    @Bean
    public Binding binding(){
    
    
        return BindingBuilder.bind(durableQueue()).to(directExchange()).with("durable");
    }

}

send a persistent message

    /**
     * 发送持久化消息
     */
    @Test
    public void sendDurableMessage() {
    
    
        String routingKey = "durable";

        CorrelationData correlationData = new CorrelationData(UUID.randomUUID().toString());

        Message message = MessageBuilder.withBody("This is a durable message".getBytes(StandardCharsets.UTF_8))
                // 设置该消息未持久化消息
                .setDeliveryMode(MessageDeliveryMode.PERSISTENT).build();

        rabbitTemplate.convertAndSend("durable.direct", routingKey, message, correlationData);
    }

Open the RabbitMQ management platform, you can see "delivery_mode: 2", indicating that the message is a persistent message

(Source code: MessageDeliveryMode class)

In fact, switches and queues are persistent by default (durable: true), so there is no need to set them specially;

Consumer Message Confirmation

introduce

Consumer message confirmation is to ensure that the consumer has consumed the message before MQ deletes the message;

It can be achieved by adding the following line of configuration in the consumer's configuration file. There are three options:

• none: Turn off ack, which means no processing, and the message will be deleted immediately after it is sent to the consumer;

• auto: automatic ack, which means that Spring detects whether the code is abnormal. If there is an exception, the message will be kept, and if there is no exception, the message will be deleted;

• manual: manual ack, you can manually write code according to the business, and return ack;

spring:
  rabbitmq:
    listener:
      simple:
      	# 设置消息确认模式
        acknowledge-mode: none

test: none

You can write code tests, the following is the producer code, send messages

    /**
     * 发送普通消息
     */
    @Test
    public void sendNoneMessage() {
    
    
        String directName = "none.direct";

        String routingKey = "none";

        String message = "This is a test message";

        rabbitTemplate.convertAndSend(directName, routingKey, message);
    }

There is a problem with the consumer code, and the message cannot be consumed normally

    @RabbitListener(bindings = @QueueBinding(
            value = @Queue(name = "none.queue"),
            exchange = @Exchange(name = "none.direct",type = ExchangeTypes.DIRECT),
            key = {
    
    "none"}
    ))
    public void getNoneMessage(String normalMessage){
    
    
        System.out.println(1/0);
        System.out.println("normalMessage = " + normalMessage);
    }

The test results, the program reported an error, and the message failed to be retained

test: auto

Change setting to: auto, try again

But the message is not deleted

This situation is not allowed in actual development, and can be solved by changing the retry mechanism of consumption failure.

Consumption failure retry mechanism

Method 1: Set retry

Because the message fails to be consumed, the message will continue to be retried in an infinite loop, causing the message processing of mq to soar and bring unnecessary pressure. This situation can be solved by adding the following configuration on the consumer side to limit the conditions for failed retries:

spring:
  rabbitmq:
    listener:
      simple:
        retry:
          # 开启消费者失败重试
          enabled: true
          # 初次失败等待时长为1秒
          initial-interval: 1000
          # 失败的等待时长倍数，即后一次等待的时间是前一次等待时间的多少倍
          multiplier: 1
          # 最多重试次数
          max-attempts: 3
          # true 无状态 false 有状态 如果业务中包含事务 改为false
          stateless: true

After it is turned on, the console can find that the information does not return to print continuously, but stops after printing a few pieces, and there is a prompt "Retry Policy Exhausted" in the log information (the retry policy has been exhausted)

This configuration method does not retain the message after several retries, but fails after several retries, then discards the message and loses the message, which is not allowed in actual development.

Method 2: Routing Stored Messages

Therefore, the following method can be used to route the message of consumption failure to another queue for storage through the switch, and then route it back for consumption after the business code is repaired.

code show as below

import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.DirectExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.rabbit.retry.MessageRecoverer;
import org.springframework.amqp.rabbit.retry.RepublishMessageRecoverer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
 * 错误消息队列
 */
@Configuration
public class ErrorMessageQueueConfig {
    
    

    /**
     * 创建一个交换机，用于路由消费失败的消息
     * @return
     */
    @Bean
    public DirectExchange errorExchange(){
    
    
        return new DirectExchange("error.direct");
    }

    /**
     * 创建一个队列，用于存储消费失败的消息
     * @return
     */
    @Bean
    public Queue errorQueue(){
    
    
        return new Queue("error.queue");
    }

    /**
     * 绑定
     * @return
     */
    @Bean
    public Binding errorBinding(){
    
    
        return BindingBuilder.bind(errorQueue()).to(errorExchange()).with("error");
    }

    /**
     * 路由，当消费失败时，把消费失败的消息路由到此队列中，路由key为"error"
     * @param rabbitTemplate
     * @return
     */
    @Bean
    public MessageRecoverer republishMessageRecoverer(RabbitTemplate rabbitTemplate){
    
    
        return new RepublishMessageRecoverer(rabbitTemplate,"error.direct","error");
    }
}

It can be seen that after the message consumption fails, it is not lost, but routed to the error queue and stored. Because the error queue does not have a RabbitListener set, messages can be stored. After the problem with the code is checked out, a monitoring method can be set for the queue to consume this part of the error message.

In addition, it is worth mentioning that the console on the consumer side will report a warning that the routing key is wrong. We can understand that at the bottom of RabbitMQ, messages that fail to consume will be routed to one place in a unified manner, and our method of manually routing messages that fail to consume to a custom queue breaks this "default rule", so a warning like this is reported. This kind of warning is within the controllable range.

Summarize

RabbitMQ sends a message. In order to ensure the reliability of the message, ensure that the message can be received by the switch and the queue, and the message can be consumed normally without being lost due to consumption failure, a series of corresponding methods are provided. Finally, two consumption failure retry methods are provided, which optimizes the consumption process and is very nice.