As the name implies, a delayed queue is a queue in which messages entering the queue will be delayed for consumption. In general queues, once messages are queued, they will be consumed by consumers immediately.
What can a delay queue do?
Delay queues are mostly used in scenarios that require delayed work. The most common are the following two scenarios:
delayed consumption. for example:
After the user generates an order, it takes a period of time to verify the payment status of the order. If the order has not been paid, the order needs to be closed in time.
After the user is successfully registered, it takes a period of time, such as a week, to check the user's usage. If the user's activity is found to be low, an email or SMS will be sent to remind the user to use it.
Delay retry. For example, a consumer fails to consume a message from the queue, but wants to automatically retry after a delay.
If we don't use a delayed queue, then we can only do it with a polling scanner. This solution is neither elegant nor convenient to make a unified service for developers to use. But with a delay queue, we can do it easily.
How to achieve?
Don't worry, in the following, we will introduce in detail how to use Spring Boot and RabbitMQ to implement delayed queues.
Realize ideas
Before introducing the specific implementation ideas, let's first introduce two features of RabbitMQ, one is Time-To-Live Extensions, and the other is Dead Letter Exchanges.
Time-To-Live Extensions
RabbitMQ allows us to set TTL (time to live) for messages or queues, which is the expiration time. TTL indicates the maximum time, in milliseconds, that a message can survive in the queue. That is to say, when a message is set with TTL or when a message enters the queue with TTL set, the message will "die" after TTL seconds and become a Dead Letter. If both message TTL and queue TTL are configured, the smaller value will be used. For more information, please refer to the official documentation.
Dead Letter Exchange
As mentioned just now, a message with a TTL set will become a Dead Letter after it expires. In fact, in RabbitMQ, there are three "death" forms of messages:
The message was rejected. By calling basic.reject or basic.nack and setting the requeue parameter to false.
The message expired because the TTL was set.
The message entered a queue that has reached its maximum length.
If the queue is set with Dead Letter Exchange (DLX), then these Dead Letters will be republished to Dead Letter Exchange and routed to other queues through Dead Letter Exchange. For more information, please refer to the official documentation.
flow chart
You must have thought of how to combine the TTL and DLX features of RabbitMQ to implement a delay queue.
For the above two scenarios of the delay queue, we have the following two flow charts:
delayed consumption
Delayed consumption is the most common usage pattern for delayed queues. As shown in the figure below, the message generated by the producer will first enter the buffer queue (the red queue in the figure). Through the TTL extension provided by RabbitMQ, these messages will be set to expire, that is, the time to delay consumption. After the messages expire, these messages will be forwarded to the actual consumption queue (blue queue in the figure) through the configured DLX, so as to achieve the effect of delayed consumption.
delayed retry
Delayed retry is essentially a type of delayed consumption, but the structure of this mode is different from the ordinary delayed consumption flow chart, so it is introduced separately.
As shown in the figure below, the consumer finds that the message processing is abnormal, such as an abnormality caused by network fluctuations. Then, if you do not wait for a period of time and try again directly, it is very likely that it will fail to succeed during this period, resulting in a certain waste of resources. Then we can put it in the buffer queue (the red queue in the figure), and wait for the message to enter the actual consumption queue (the blue queue in the figure) again after a period of delay. When the time is up, some abnormal fluctuations have usually recovered, and these messages can be consumed normally.
Code
Next we will introduce how to implement a RabbitMQ-based delay queue in Spring Boot. We assume that the reader already has a basic knowledge of Spring Boot and RabbitMQ. If you want to quickly understand the basics of Spring Boot, you can refer to an article I wrote earlier.
initialization project
First we create a Spring Boot project in Intellij and add the spring-boot-starter-amqp extension.
Configure the queue
From the above flow chart, we can see that the implementation of a delay queue requires a buffer queue and an actual consumption queue. And because in RabbitMQ, we have two configuration methods for message expiration, so in the code, we configure a total of three queues:
1.delay_queue_per_message_ttl: TTL configured on the message buffer queue.
2.delay_queue_per_queue_ttl: The buffer queue with TTL configured on the queue.
3. delay_process_queue: The actual consumption queue.
We configure the above queue as a Bean by means of Java Config. Since we added the spring-boot-starter-amqp extension, Spring Boot automatically creates these queues at startup based on our configuration. In order to facilitate the next test, we configure the DLX of delay_queue_per_message_ttl and delay_queue_per_queue_ttl to be the same, and expired messages will be forwarded to delay_process_queue through DLX.
delay_queue_per_message_ttl
First introduce the configuration code of delay_queue_per_message_ttl:
@BeanQueuedelayQueuePerMessageTTL(){returnQueueBuilder.durable(DELAY_QUEUE_PER_MESSAGE_TTL_NAME) .withArgument("x-dead-letter-exchange", DELAY_EXCHANGE_NAME) // DLX，dead letter发送到的exchange.withArgument("x-dead-letter-routing-key", DELAY_PROCESS_QUEUE_NAME) // dead letter携带的routing key.build();}
Among them, x-dead-letter-exchange declares the DLX name to which the dead letters in the queue are forwarded, and x-dead-letter-routing-key declares the routing-key name that these dead letters carry when forwarding.
delay_queue_per_queue_ttl
Similarly, the configuration code for delay_queue_per_queue_ttl:
@BeanQueuedelayQueuePerQueueTTL(){returnQueueBuilder.durable(DELAY_QUEUE_PER_QUEUE_TTL_NAME) .withArgument("x-dead-letter-exchange", DELAY_EXCHANGE_NAME) // DLX.withArgument("x-dead-letter-routing-key", DELAY_PROCESS_QUEUE_NAME) // dead letter携带的routing key.withArgument("x-message-ttl", QUEUE_EXPIRATION) // 设置队列的过期时间.build();}
The configuration of the delay_queue_per_queue_ttl queue has one more x-message-ttl than the configuration of the delay_queue_per_message_ttl queue, which is used to set the expiration time of the queue.
delay_process_queue
The configuration of delay_process_queue is the simplest:
@BeanQueuedelayProcessQueue(){returnQueueBuilder.durable(DELAY_PROCESS_QUEUE_NAME) .build();}
Configure Exchange
Configure DLX
First, we need to configure DLX, the code is as follows:
@BeanDirectExchangedelayExchange(){returnnew DirectExchange(DELAY_EXCHANGE_NAME);}
Then bind the DLX to the actual consumption queue, namely delay_process_queue. This way all dead letters will be forwarded to delay_process_queue via DLX:
@BeanBindingdlxBinding(Queue delayProcessQueue, DirectExchange delayExchange){returnBindingBuilder.bind(delayProcessQueue) .to(delayExchange) .with(DELAY_PROCESS_QUEUE_NAME);}
Configure Exchange Required for Delayed Retry
From the flowchart of delayed retry, we can see that after message processing fails, we need to forward the message to the buffer queue, so the buffer queue also needs to be bound to an Exchange. In this example, we use delay_process_per_queue_ttl as the buffer queue in delayed retry. How the specific code is configured, I won't go into details here, you can check the code in my Github.
define consumers
We create the simplest consumer ProcessReceiver, this consumer listens to the delay_process_queue queue, for received messages, he will:
1. If the message body in the message is not equal to FAIL_MESSAGE, then he will output the message body.
2. If the message body in the message happens to be FAIL_MESSAGE, then it will simulate throwing an exception, and then redirect the message to the buffer queue (corresponding to the delayed retry scenario).
In addition, we also need to create a new listening container for storing consumers. The code is as follows:
@BeanSimpleMessageListenerContainerprocessContainer(ConnectionFactory connectionFactory, ProcessReceiver processReceiver){ SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();container.setConnectionFactory(connectionFactory);container.setQueueNames(DELAY_PROCESS_QUEUE_NAME); // 监听delay_process_queuecontainer.setMessageListener(new MessageListenerAdapter(processReceiver));returncontainer;}
So far, our pre-configured code has been written, and then we need to write test cases to test our delay queue.
Write test cases
Delayed consumption scenarios
First we write test code that tests the TTL setting on the message.
We use the RabbitTemplate class provided under the spring-rabbit package to send messages. Since we added the spring-boot-starter-amqp extension, Spring Boot will automatically load the RabbitTemplate as a bean into the container during initialization.
The problem of message sending is solved, so how to set TTL for each message? Here we need to use MessagePostProcessor. MessagePostProcessor is usually used to set the Header of the message and the properties of the message. We create a new ExpirationMessagePostProcessor class to be responsible for setting the TTL property of the message:
/*** Set the expiration time of the message*/public class ExpirationMessagePostProcessorimplements MessagePostProcessor{ private final Long ttl; // milliseconds public ExpirationMessagePostProcessor(Long ttl){ this.ttl = ttl;} @Overridepublic Message postProcessMessage(Message message)throws AmqpException {message .getMessageProperties().setExpiration(ttl.toString()); // Set the expiration time of per-message returnmessage;}}
Then when calling the convertAndSend method of RabbitTemplate, pass in ExpirationMessagePostPorcessor. We send 3 messages to the buffer queue with expiration times of 1 second, 2 seconds and 3 seconds. The specific code is as follows:
@Testpublic voidtestDelayQueuePerMessageTTL()throws InterruptedException {ProcessReceiver.latch = new CountDownLatch(3);for(int i = 1; i <= 3; i++) { long expiration = i * 1000;rabbitTemplate.convertAndSend(QueueConfig.DELAY_QUEUE_PER_MESSAGE_TTL_NAME,(Object) ("Message From delay_queue_per_message_ttl with expiration "+ expiration), new ExpirationMessagePostProcessor(expiration));}ProcessReceiver.latch.await();}
Careful friends will surely ask, why add a CountDownLatch to the code? This is because if there is no latch to block the test method, the test case will end directly, the program exits, and we will not see the performance of delayed consumption of messages.
Then similarly, the code to test the TTL setting on the queue is as follows:
@Testpublic voidtestDelayQueuePerQueueTTL()throws InterruptedException {ProcessReceiver.latch = new CountDownLatch(3);for(int i = 1; i <= 3; i++) {rabbitTemplate.convertAndSend(QueueConfig.DELAY_QUEUE_PER_QUEUE_TTL_NAME,"Message From delay_queue_per_queue_ttl with expiration "+ QueueConfig.QUEUE_EXPIRATION);}ProcessReceiver.latch.await();}
We send 3 messages to the buffer queue. In theory, these 3 messages will expire at the same time after 4 seconds.
Delayed retry scenario
We also need to test the delayed retry scenario.
@Testpublic voidtestFailMessage()throws InterruptedException {ProcessReceiver.latch = new CountDownLatch(6);for(int i = 1; i <= 3; i++) {rabbitTemplate.convertAndSend(QueueConfig.DELAY_PROCESS_QUEUE_NAME, ProcessReceiver.FAIL_MESSAGE);}ProcessReceiver.latch.await();}
We send 3 messages to delay_process_queue that will trigger FAIL, and theoretically these 3 messages will be automatically retried after 4 seconds.
View test results
Delayed consumption scenarios
For the scenario test of delayed consumption, we divide the TTL setting on the message and the TTL setting on the queue. First, let's take a look at the test results of the TTL setting on the message:
From the figure above, we can see that ProcessReceiver receives messages after 1 second, 2 seconds, and 3 seconds, respectively. The test results show that messages are not only consumed with a delay, but the delay time of each message can be customized. The delayed consumption scenario test with TTL set on the message was successful.
Then, the test result of the TTL setting on the queue is as follows:
As we can see from the above image, ProcessReceiver received 3 messages at the same time after a delay of 4 seconds. The test results show that the message is not only consumed late, but also that the expiration time of the message is fixed when the TTL is set on the queue. The delayed consumption scenario test with TTL set on the queue was successful.
Delayed retry scenario
Next, let's take a look at the test results of delayed retry:
ProcessReceiver first received 3 messages that would trigger FAIL, and then moved it to the buffer queue. After 4 seconds, it received the 3 messages just now. The delayed retry scenario test succeeded.
If you want to learn and communicate HashMap, nginx, dubbo, Spring MVC, distributed, high-performance, high-availability, redis, jvm, multi-threading, netty, kafka, and are interested in the course, join the group: 629740746 You can also get the video materials shared below for free
