RabbitMQ learning (4) "message queue reliability delivery"

1. Message Reliability Delivery

Preface

In the code, the database must be operated first before sending messages. Avoid data inconsistency caused by database rollback. But if the data is manipulated first, then the message is sent, and there is a problem sending the message, isn't it the same as the business data inconsistency?

In this article, we will analyze the reliability of RabbitMQ delivery, that is, when using RabbitMQ to achieve asynchronous communication, what should I do if the message is lost, and what should I do if the message is repeatedly consumed? RabbitMQ provides many mechanisms to ensure reliable delivery of messages, which is also a feature of RabbitMQ.

Before learning RabbitMQ, we must clarify a problem, because efficiency and reliability cannot be achieved at the same time. If you want to ensure that every link is successful, it will inevitably affect the efficiency of message sending and receiving. Therefore, if the real-time consistency requirements of some services are not particularly high, some reliability can be sacrificed in exchange for efficiency. For example, in the scenario of sending a notification or recording a log, if the user does not receive the notification, it will not cause business impact, and just send it again.

When we use RabbitMQ to send and receive messages, there are several main links:

1. 1 means that the message is sent from the producer to the broker.
   After the producer sends the message to the Broker, how can he know whether his message has been successfully received by the Broker?
2. 2 Represents that the message is routed from Exchange to Queue
   Exchange is a binding list. If the message cannot be routed to the correct queue, what will happen? What should I do?
3. 3 Represents that the message is stored in the Queue. The
   queue is an independent service with its own database (Mnesia), which is really used to store messages. If there is no consumer to consume, the message must always be stored in the queue. If there is a problem with the queue, the message will definitely be lost. How to ensure that messages are stored stably in the queue?
4. 4 On behalf of consumers, subscribe to Queue and consume messages
   . What are the characteristics of queues? FIFO. The messages in the queue are delivered one by one, that is, only after the previous message is received by the consumer, can this message be deleted from the database and continue to deliver the next message. So the question is, how does Broker know that the consumer has received the message?

Solving the MQ reliability delivery is mainly to solve the problems caused by the above four links

There are 9 levels to solve the problem below

1. The message is sent to the RabbitMQ server

The first link is that the producer sends a message to the broker. The message may fail to be sent due to network or Broker problems, and the producer cannot be sure whether the Broker has received it correctly. Two mechanisms are provided in RabbitMQ. The server confirmation mechanism is that when the producer sends a message to the RabbitMQ server, the server will return a response in some way. As long as the producer receives the response, it knows that the message is sent It succeeded.
The first is Transaction mode, and the second is Confirm mode.

• Transaction
  mode How to use the transaction mode?
  We set the channel to transaction mode through a channel.txSelect() method, and then we can publish a
  message to RabbitMQ. If the channel.txCommit() ; method is successfully called, it means that the transaction is successfully submitted and the message must have arrived RabbitMQ.
  If an exception is thrown due to RabbitMQ's abnormal crash or other reasons before the transaction is submitted and executed,
  we can catch it at this time , and then implement the transaction rollback by executing the **channel.txRollback()** method.

AMQP protocol packet capture diagram:

In transaction mode, only after receiving the Commit-OK instruction from the server can the submission be successful. So it can solve the problem confirmed by the producer and the server. But the transaction mode has a shortcoming, it is blocking , a message is not sent, can not send the next message, it will drain the performance of RabbitMQ server. Therefore, it is not recommended to use in a production environment.
Settings in Spring Boot:

rabbitTemplate.setChannelTransacted(true);

• Confirm (acknowledgment) mode
  can be guaranteed message received Broker, but do not consume a lot of properties manner ,
  confirmation mode three

1. Normal confirmation mode
   On the producer's side, the channel is set to Confirm mode by calling the **channel.confirmSelect()** method, and then the message is sent. Once the message is delivered to all matching queues, RabbitMQ will send an acknowledgment (Basic.Ack) to the producer, that is, calling **channel.waitForConfirms()** to return true, so that the producer knows that the message is being served by the server Received.
2. Bulk confirmation method
   Bulk confirmation means to send a batch of messages first after starting the Confirm mode. As long as the channel.waitForConfirmsOrDie(); method does not throw an exception, it means that the message has been received by the server. The batch confirmation method is more efficient than the single confirmation method.
   But there are two problems . The first is the determination of the batch quantity. For different services, how many messages are sent to confirm once? The quantity is too small, and the efficiency cannot be improved. The number of words, will bring another problem, such as we send 1000 messages only confirm once, if the previous 999 messages have been received the service side, if the message was rejected the first 1000, then in front of all the consumer
   interest are Want to resend.
3. Asynchronous confirmation mode
   Asynchronous confirmation mode requires adding a ConfirmListener and using a SortedSet to maintain unconfirmed messages. The Confirm mode is enabled on the Channel, because RabbitTemplate encapsulates the Channel, called ConfirCallback.

rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
    
    
	@Override
	public void confirm(CorrelationData correlationData, boolean ack, String cause) {
    
    
		if (!ack) {
    
    
			System.out.println("发送消息失败：" + cause);
			throw new RuntimeException("发送异常：" + cause);
		}
	}
})

2. The message is routed from the switch to the queue

The second link is the routing of messages from the switch to the queue. Under what circumstances will messages fail to be routed to the correct queue? It may be because the routing key is wrong, or the queue does not exist.

We have two ways to deal with unrouted messages, one is to let the server send back to the producer, and the other is to let the switch route to another backup switch.

• Message postback
  uses mandatory parameters and ReturnListener ( ReturnCallback in Spring AMQP).

rabbitTemplate.setMandatory(true);
rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback(){
    
    
        public void returnedMessage (Message message,
        int replyCode,
        String replyText,
        String exchange,
        String routingKey){
    
    
        System.out.println("回发的消息：");
        System.out.println("replyCode: " + replyCode);
        System.out.println("replyText: " + replyText);
        System.out.println("exchange: " + exchange);
        System.out.println("routingKey: " + routingKey);
    }
});

• Message routing to the backup switch
  When creating the switch, specify the backup switch from the properties.

Map<String,Object> arguments = new HashMap<String,Object>();
arguments.put("alternate-exchange","ALTERNATE_EXCHANGE"); // 指定交换机的备份交换机
channel.exchangeDeclare("TEST_EXCHANGE","topic", false, false, false, arguments);

Note the difference, the queue can be designated a dead letter switch; the switch can be designated a backup switch

3. The message is stored in the queue

The third link is that the message is stored in the queue. If there is no consumer, the queue always exists in the database. If RabbitMQ's service or hardware fails, such as system downtime, restart, shutdown, etc., messages in memory may be lost, so we need to save the message itself and metadata (queue, switch, binding) to disk (Persistent storage).
The solutions are:

1. Queue persistence

@Bean("Queue")
public Queue Queue() {
    
    
	// queueName, durable, exclusive, autoDelete, Properties
	return new Queue("TEST_QUEUE", true, false, false, new HashMap<>());
}

2. Switch persistence

@Bean("Exchange")
public DirectExchange exchange() {
    
    
	// exchangeName, durable, exclusive, autoDelete, Properties
	return new DirectExchange("TEST_EXCHANGE", true, false, new HashMap<>());
}

3. Information endurance

MessageProperties messageProperties = new MessageProperties();
messageProperties.setDeliveryMode(MessageDeliveryMode.PERSISTENT);
Message message = new Message("持久化消息".getBytes(), messageProperties);
rabbitTemplate.send("TEST_EXCHANGE", "test", message);

4. 
   If the cluster has only one RabbitMQ node, even if the switches, queues, and messages are persisted, if the service crashes or the hardware fails, the RabbitMQ service is also unavailable. Therefore, in order to improve the availability of the MQ service and ensure the transmission of messages, we Need to have multiple RabbitMQ nodes

4. The message is delivered to the consumer

If an exception occurs before the consumer has time to deal with the message after receiving the message, or an exception occurs during the processing, it will lead to ④ failure. The server should know the consumer's reception of the message in some way , and decide whether to re-deliver the message to other consumers.

RabbitMQ provides a message acknowledgement mechanism for consumers . Consumers can automatically or manually send ACK to the server. If the ACK message is not received, RabbitMQ will send this message to other consumers after the consumer is disconnected. If there are no other consumers, the consumer will re-consume this message after restarting, and repeat the business logic.

Consumers can specify the autoAck parameter when subscribing to the queue. When autoAck is equal to false, RabbitMQ will wait for the consumer to explicitly reply to the confirmation signal before removing the message from the queue.

How to set manual ACK?
SimpleRabbitListenerContainer or SimpleRabbitListenerContainerFactory

factory.setAcknowledgeMode(AcknowledgeMode.MANUAL);

application.properties

spring.rabbitmq.listener.direct.acknowledge-mode=manual 
spring.rabbitmq.listener.simple.acknowledge-mode=manual

Note the difference between these three values:

• NONE : Automatic ACK
• MANUAL : Manual ACK
• AUTO : If the method does not throw an exception, send an ack.

The producer finally determines whether the consumer has two ways to succeed in consumption:

1. The consumer receives the message and after processing it, calls the producer's API (thinking: Does decoupling break?)
2. The consumer receives the message and after processing it, sends a response message to the producer

Other ways to ensure reliability

1. Consumer callback

1. Call the producer API to modify the data status
2. Send response message to producer

2. Compensation mechanism

What if the producer's API is not called and the consumer's response message is not received? Don't worry, it may be because the consumer processing time is too long or the network timeout. A timeout period should be agreed between the producer and the consumer, such as 5 minutes. For messages that do not get a response beyond this time, a mechanism for regular retransmission can be set, but the sending interval and control times should be sent, such as every 2 minutes. Once, retransmit at most 3 times, otherwise it will cause message accumulation.

Retransmission can be achieved through message drop database + timing task.
Resend, do you send the same message?

3. Message idempotence

If the consumer succeeds in receiving the message from the producer every time, but there is a problem when responding or calling the API, will there be repeated processing of the message? For example: deposit 100 yuan, ATM retransmission 5 times, the core system A total of 6 times were processed, and the balance increased by 600 yuan.
Therefore, in order to avoid repeated processing of the same message, certain measures must be taken. The RabbitMQ server does not have this kind of control (the same batch of messages has an incremental DeliveryTag). It does not know whether you are going to send a message twice, and it can only be controlled on the consumer side.
How to avoid repeated consumption of messages? There may be three reasons for repeated messages:

1. The problem of the producer, the link 1 repeatedly sends the message, for example, when the Confirm mode is turned on but the confirmation is not received, the producer repeats the delivery.
2. There was a problem in link 4. The message was delivered repeatedly because the consumer did not send an ACK or other reasons.
3. Producer code or network issues.

For repeatedly sent messages, a unique business ID can be generated for each message , and repetitive control can be done through the log or message drop.

2. The order of messages

The order of messages means that the order in which consumers consume messages is consistent with the order in which producers produce messages.

For example: 1. Post Weibo; 2. Post a comment; 3. Delete Weibo. The order cannot be reversed.
In RabbitMQ, when there are multiple consumers in a queue, because different consumers consume messages at different rates, the order cannot be guaranteed. Only when there is only one consumer in a queue can sequential consumption be guaranteed (different business messages are sent to different dedicated queues).