How does RabbitMQ ensure the successful delivery of messages? How does RabbitMQ ensure non-repetitive consumption and data loss? In a distributed system, how to ensure data consistency? Can you withstand a series of cannons?

In fact, the principles of these questions are similar, and they can all be answered in a unified way.

First understand a little, ask what our business needs to use MQ, whether it is RabitMQ, kafka, or RocketMQ?

To put it bluntly, MQ asynchronous processing is a decoupling tool in the layered architecture of the Internet, which can help the system handle high concurrency, cut peaks and fill valleys.

Decoupling: Make services relatively independent, reduce strong dependencies, reduce code maintenance costs, and enhance scalability;
Asynchronous: Write messages to the message queue, and run non-essential business logic in an asynchronous manner to speed up the response;
Peak shaving: When the amount of concurrency is large, avoid all requests being directly pressed to the database;

Let's take RabitMQ (similar to others) as an example to show the process of a message from production to consumption:

Understand this process, let's look at the problem together.

Question 1: How to ensure no double consumption?

The key to ensuring that messages are not repeatedly consumed is to ensure the idempotency of the message queue. For this problem, the Redis global ID method is generally used to solve this problem.

When each message is sent, a globally unique ID is assigned as an identification;
Before the consumer starts to consume, first go to Redis to check whether there is a consumption record;
If there is no record, process the data, and save the ID after success;
If there is a record, it will not be consumed, and each record is guaranteed to be consumed only once;

Question 2: How to ensure the successful delivery of the message?

This is the most typical problem in the use of MQ. According to the above figure, we will analyze the data loss scenarios one by one:

Producer loses data
RabbitMQ lost data by itself
Consumer lost data

1. The producer loses the data

When the producer sends data to RabbitMQ, the data may be lost due to network problems during the transmission process. In response to this situation, there are two ways to solve it, first introduce and then conclude.

Transaction capabilities of RabbitMQ

The producer starts the transaction before sending the data, and then sends the message. If the message is not successfully received by RabbitMQ , the producer will receive an abnormal error. At this time, the transaction can be rolled back, and then try to resend; if the message is received, then Things can be submitted.

// 开启事物
channel.txSelect();
try {
    // 发送消息
} catch(Exection e) {
    // 回滚事物
    channel.txRollback()；
    // 重新提交
}

However, this method has a fatal disadvantage: when things are turned on, MQ will become a synchronous blocking operation, and the producer will block waiting for the successful transmission, which goes against the asynchronous original intention of MQ, and also reduces the throughput of the system.

Enable confirm mode

RabbitMQ provides a reliable message delivery mode (confirm).

After the producer has enabled the confirm mode, each message written will be assigned a unique ID, and then how to write it into RabbitMQ, RabbitMQ will send you an ack message back, telling you that the message is sent OK; if RabbitMQ If you fail to process this message, you will be called back to a nack interface, telling you that the message failed and you can try again.

And you can combine this mechanism to know that you maintain the ID of each message in memory. If the callback of the message has not been received after a certain period of time, you can manually resend it. For example, like ensuring the idempotency of the message, the unique ID of the message is stored in Redis, and it will only be deleted after the ack message is successfully received, otherwise it will be resent regularly.

    // 开启confirm
    channel.confirm();
    // 发送成功回调
    public void ack(String messageId){

    }
    // 发送失败回调
    public void nack(String messageId){
        // 重发该消息
    }

In the above 2 methods, the confirm mechanism is usually used to avoid the loss of consumer messages.

2. RabbitMQ lost data by itself

To deal with the loss of data in the message queue, the configuration of persistent disk is generally enabled.

There are usually two steps to set up persistence, and these two must be turned on at the same time:

Set the persistent identifier durable of queue to true, which means it is a persistent queue;
When sending a message, set deliveryMode = 2, so that the message will be set to persistent mode, and RabbitMQ will persist the message to disk at this time;

After this setting, even if RabbitMQ hangs, the data can be restored after restarting.

Moreover, this persistent configuration can be used in conjunction with the confirm mechanism to send an ack signal to the producer after the message is persisted to disk. In this way, if RabbitMQ dies before the message is persisted to disk, the producer will not receive the ack signal, and the producer will automatically resend it.

Moreover, persistence can be combined with the confirm mechanism of production. Only after the message is persisted to the disk, the producer will be notified of ack, so even if rabbitmq hangs before persistence and the data is lost, the producer cannot receive the ack callback. The message will be resent.

To add another case, if the message has not been persisted to the hard disk, maybe the service has died? In this case, mirrored queues can be introduced by introducing mirrored queues, but there is no guarantee that messages will not be lost 100% (the entire cluster will hang up).

3. The consumer loses the data

Enabling manual confirmation mode can solve this problem.

Turn off the autoAck function and enable manual confirmation mode;
Each time after ensuring that the message is processed, manually call ack in the code to ensure that the message must be consumed;

If the autoAck mode is turned on, when the consumer processes the data, the consumer will automatically notify RabbitMQ that the data has been consumed. If it is unfortunate at this time, the consumer has not finished processing the server and the server is down, and both RabbitMQ and the consumer think that the message has been consumed and will not trigger the retransmission mechanism, which will cause the message to be lost.

Under the introduction, there are 3 modes for consumer mobile to choose from:

Automatic acknowledgment mode: the consumer hangs up, and the messages to be acked are returned to the queue. The consumer throws an exception, and the message will continue to be resent until the processing is successful. No message will be lost, even if the service hangs, the unprocessed message will be returned to the queue, but the exception will cause the message to be retried continuously.
Manual acknowledgment mode: If the consumer dies before it can be processed, it will repeatedly send a message to other consumers when there is no response to ack; if the listener handles the exception and does not catch the exception, it will continue to receive the message repeatedly, and then keep on Throws an exception; if the exception is caught, but not acked in finally, the message will be repeatedly sent (retry mechanism).
Unconfirmed mode: acknowledge="none" does not use the confirmation mechanism. As long as the message is sent, it will be removed from the queue immediately. Regardless of whether the client is abnormal or disconnected, it will be removed as long as it is sent and will not be resent.

Question 3: How to ensure the order of consumption?

In the case of 1 producer and multiple consumers. The order of generation is data 1, data 2, and data 3, and the consumed data is distributed, and it cannot be guaranteed that all of it will be sent to a consumer, and it cannot be guaranteed that it must be consumed in the order of 1 - 2 - 3.

In response to this problem, some algorithm can be used. My idea is to put the messages that need to be kept in order into the same message queue. Then use only one consumer to consume the queue.

The messages in the same Queue must be in order, and the consumption of the same consumer from the Queue must also be in order.

Share an article and say in more detail: How does the message queue ensure the ordering of messages? - In order to achieve orderly messages, Zhihu needs to consider both the Producer and the Consumer. First, the Producer must be in order when producing messages. Then, when the Consumer consumes, it must be in order and cannot be chaotic. Producer is ordered by ordinary message systems like RabbitMQ… https://zhuanlan.zhihu.com/p/372469047 Hope it can help you.

How does RabbitMQ ensure the successful delivery of messages? Idempotency? sequential?