How does RabbitMQ ensure that 99.99% of messages are not lost through persistence?

1. Outline of this article

To solve this problem, we need to use the concept of persistence in RabbitMQ . The so-called persistence means that RabbitMQ will solidify the data in memory (Exchange exchange, Queue queue, Message message) to disk to prevent abnormal situations. data lost.

Among them, the persistence of RabblitMQ is divided into three parts:

Persistence of Exchange
Queue persistence
Persistence of messages

2. Persistence of Exchange

In the last blog, we stated that the code of Exchange is like this:

private final static String EXCHANGE_NAME = "normal-confirm-exchange";

// 创建一个Exchange
channel.exchangeDeclare(EXCHANGE_NAME, "direct");

The Exchange declared in this case is non-persistent. When RabbitMQ has an abnormal situation (restart, downtime), the Exchange will be lost, which will affect subsequent messages written to the Exchange . So how to set the Exchange to be persistent? The answer is to set the durable parameter.

durable: Whether the setting is durable. Durable is set to true to indicate persistence, otherwise it is non-persistent.

Persistence can save the switch to disk, and the relevant information will not be lost when the server is restarted.

Set up Exchange persistence:

channel.exchangeDeclare(EXCHANGE_NAME, "direct", true);

The overloaded method called at this time is:

public DeclareOk exchangeDeclare(String exchange, String type, boolean durable) throws IOException {
    return this.exchangeDeclare(exchange, (String)type, durable, false, (Map)null);
}

For better understanding, we create a new production class as follows:

package com.zwwhnly.springbootaction.rabbitmq.durable;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;

import java.io.IOException;
import java.util.concurrent.TimeoutException;

public class DurableProducer {
    private final static String EXCHANGE_NAME = "durable-exchange";
    private final static String QUEUE_NAME = "durable-queue";

    public static void main(String[] args) throws IOException, TimeoutException {
        // 创建连接
        ConnectionFactory factory = new ConnectionFactory();
        // 设置 RabbitMQ 的主机名
        factory.setHost("localhost");
        // 创建一个连接
        Connection connection = factory.newConnection();
        // 创建一个通道
        Channel channel = connection.createChannel();
        // 创建一个Exchange
        channel.exchangeDeclare(EXCHANGE_NAME, "direct");
        channel.queueDeclare(QUEUE_NAME, false, false, false, null);
        channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "");

        // 发送消息
        String message = "durable exchange test";
        channel.basicPublish(EXCHANGE_NAME, "", null, message.getBytes());

        // 关闭频道和连接
        channel.close();
        connection.close();
    }
}

In the sample code, we created a non-persistent Exchange and a non-persistent Queue, and bound them. At this time, the code was run, the Exchange and Queue were newly created, and the message 'durable exchange test' was also correctly delivered to the queue:

At this point, restart the RabbitMQ service and you will find that Exchange is lost:

Modify the following code and set the durable parameter to true:

// 创建一个Exchange
channel.exchangeDeclare(EXCHANGE_NAME, "direct", true);

After running the code at this point, and then restarting the RabbitMQ service, you will find that Exchange is no longer lost:

3. Queue persistence

Careful netizens may find that although the Exchange is not lost after restarting the RabbitMQ service now, the queues and messages are lost, so how to solve the problem that the queues are not lost? The answer is also to set the durable parameter.

durable: Whether the setting is durable. Set to true to make the queue persistent.

Persistent queues will be saved to disk to ensure that relevant information is not lost when the server is restarted.

Simply modify the code that declares the Queue above and set the durable parameter to true:

channel.queueDeclare(QUEUE_NAME, true, false, false, null);

The overloaded method called at this time is as follows:

public com.rabbitmq.client.impl.AMQImpl.Queue.DeclareOk queueDeclare(String queue, boolean durable, boolean exclusive, boolean autoDelete, Map<String, Object> arguments) throws IOException {
    validateQueueNameLength(queue);
    return (com.rabbitmq.client.impl.AMQImpl.Queue.DeclareOk)this.exnWrappingRpc((new com.rabbitmq.client.AMQP.Queue.Declare.Builder()).queue(queue).durable(durable).exclusive(exclusive).autoDelete(autoDelete).arguments(arguments).build()).getMethod();
}

Run the code, then restart the RabbitMQ service, and you will find that the queue is not lost now:

4. Message persistence

Although the Exchange and Queue are not lost after RabbitMQ is restarted now, the messages stored in the Queue are still lost, so how to ensure that the messages are not lost? The answer is to set the delivery mode of the message to 2, which means persistence.

Modify the code to send the message to:

// 发送消息
String message = "durable exchange test";
AMQP.BasicProperties props = new AMQP.BasicProperties().builder().deliveryMode(2).build();
channel.basicPublish(EXCHANGE_NAME, "", props, message.getBytes());

The overloaded method called is:

public void basicPublish(String exchange, String routingKey, BasicProperties props, byte[] body) throws IOException {
    this.basicPublish(exchange, routingKey, false, props, body);
}

Run the code, then restart the RabbitMQ service, and find that Exchange, Queue, and messages are not lost at this time:

So far, we have perfectly solved the problem of message loss after RabbitMQ restarts.

The final code is as follows, you can also download all the source code used in this article through the source code link at the end of the article:

package com.zwwhnly.springbootaction.rabbitmq.durable;

import com.rabbitmq.client.AMQP;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;

import java.io.IOException;
import java.util.concurrent.TimeoutException;

public class DurableProducer {
    private final static String EXCHANGE_NAME = "durable-exchange";
    private final static String QUEUE_NAME = "durable-queue";

    public static void main(String[] args) throws IOException, TimeoutException {
        // 创建连接
        ConnectionFactory factory = new ConnectionFactory();
        // 设置 RabbitMQ 的主机名
        factory.setHost("localhost");
        // 创建一个连接
        Connection connection = factory.newConnection();
        // 创建一个通道
        Channel channel = connection.createChannel();
        // 创建一个Exchange
        channel.exchangeDeclare(EXCHANGE_NAME, "direct", true);
        channel.queueDeclare(QUEUE_NAME, true, false, false, null);
        channel.queueBind(QUEUE_NAME, EXCHANGE_NAME, "");

        // 发送消息
        String message = "durable exchange test";
        AMQP.BasicProperties props = new AMQP.BasicProperties().builder().deliveryMode(2).build();
        channel.basicPublish(EXCHANGE_NAME, "", props, message.getBytes());

        // 关闭频道和连接
        channel.close();
        connection.close();
    }
}

5. Precautions

1) In theory, all messages can be set to be persistent, but this will seriously affect the performance of RabbitMQ. Because writing to disk is more than a little slower than writing to memory. For messages that are not so reliable, persistent processing can be omitted to improve the overall throughput. When choosing whether to persist messages, there is a trade-off between reliability and throughput.

2) After the switches, queues, and messages are set up for persistence, there is still no 100% guarantee that the data will not be lost , because after the persistent messages are correctly stored in RabbitMQ, it will take some time (although it is very short, but can not be ignored) to save. into the disk. If the RabbitMQ service node is down, restarted and other abnormal situations during this period, and the messages have not had time to be placed on the disk, these messages will be lost.

3) Only set the persistence of the queue, and the message will be lost after restart; only set the persistence of the message, the queue will disappear after the restart, and then the message will be lost. It is pointless to set message persistence without setting queue persistence.