Rabbitmq reliable message delivery

One, background

It sends a message to the production rabbitmq, due to network failure, etc. may result message. Therefore, there must be a mechanism to ensure RabbitMQ can accurately reach MQ message, if not reach, it must be fed to the production side retransmission.

RabbitMQ message delivery reliability are two main implementations:
1, retry mechanism implemented by consumption achieved by retry @Retryable, can set the number of retries and the retry frequency;
2, to achieve the production of end message delivery reliability.

Both methods the consumer side may receive duplicate message asking the consumer side must implement power consumption and so on.

Second, the message is delivered to the exchange confirmation mode

rabbitmq message delivery process are:
Producer -> rabbitmq Broker Cluster -> Exchange -> Queue -> Consumer

1, sends a message to the production rabbitmq broker cluster, returned from the asynchronous receiving rabbitmq ack acknowledgment information.

2, after the production end receives acknowledgment message ack is returned according to ack is true or false, call processing confirmCallback interface.

 

Open end production confirm the mode application.yml

spring:
  rabbitmq:
    publisher-confirms: true

 

ConfirmCallback confirm implemented method interface, ack true to the success message, ack message transmission failure, false

@Component
@Slf4j
public class RabbitTemplateConfig implements ConfirmCallback{
    @Autowired
    private RabbitTemplate rabbitTemplate;

    @PostConstruct
    public void init() {        
        rabbitTemplate.setConfirmCallback(this);   // 指定 ConfirmCallback
    }
    
    @Override
    public void confirm(CorrelationData correlationData, boolean ack, String cause) {
       if (!ack) {
          //try to resend msg
       } else {
          //delete msg in db
       }
   }
}

Note: confirmCallback callback interface is no message data, even if the message fails, the production side can not directly retransmitted in this callback interface, confirmCallback can only play the role of a notification.

Third, the delivery failure message retransmission mechanism

If rabbitmq returns ack failure, the production side can not confirm whether the message actually sent successfully, can also cause data loss. The best way is to use rabbitmq transaction mechanism, but the transaction mechanism rabbitmq efficiency is very low, only a few hundred messages processed per second, is not suitable for a large amount of concurrency scenarios. 

 

Another realization of ideas:

1, the production side to save messages sent each time, if you delete a message is sent successfully;

2, if the transmission fails to remove the message resending;

3, if the timeout has not received ack mq returned to resend the same message removed.

This avoids the risk of message loss.

 

Redis save the message msg to use an example, the specific implementation plan is:

1, the production side before sending the message, generates a unique confirmation ack ID;

2, to a bond ackId, message value, given redis stored into the cache, setting a timeout period;

3, redis implement a timeout trigger interface, when the key is expired, and resend the message again to step 2;

4, the production side ConfirmCallback implement the interface;

5, when ConfirmCallback interface trigger, if ack is true, then delete the corresponding ackId msg; ack if false, then the corresponding msg taken ackId retransmission;

 

Also online implementations:

By providing redis without triggering the timeout retransmission timeout event, but taken into a ackFailList message, and then opening timing of the task, the scan ackFailList, retransmission failure msg.

The idea of ​​this program on the Internet and on a similar program, but it is the use of additional List to save the message failed because List is stored in memory, do not have the persistence feature, so this is not safe, if the production side abnormal program exit will lead to loss of messages. It can be considered saved to the database.

Fourth, the message is not delivered to the queue of return mode

Production side failed to return by implementing ReturnCallback interface startup messages, message routing will not trigger the callback interface queue.

 

In turn return mode in application.yml

spring:
  rabbitmq:
    publisher-returns: true

 

ReturnCallback implement the interface, the content of the message body can be acquired, for message retransmission

@Component
@Slf4j
public class RabbitTemplateConfig implements ReturnCallback {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    @PostConstruct
    public void init() {        
        rabbitTemplate.setReturnCallback(this);   //指定 ReturnCallback
    }

    @Override
    public void returnedMessage(Message message, int replyCode, String replyText, String exchange, String routingKey) {
        log.info("消息主体 message : {}", message);
        log.info("Message body Message: {}" , replyCode); 
        log.info ( "Description: {}" , replyText); 
        log.info ( "message using an exchanger Exchange: {}" , Exchange); 
        log.info ( " routing keys routing messages used: {} " , routingKey); 
    } 
}