Original: WeChat public account [Ah Q said code], welcome to share, please keep the source for reprinting.
Hello everyone, I am Ah Q!
A few days ago, the leader suddenly announced that the e-commerce project that had been discontinued a few years ago had been restarted again. I carefully read the "childhood" code with a complicated mood, and only I could feel the sadness in my heart.
No, I was called into the "little black room" by the leader yesterday and asked me to refactor the code and upgrade it. Seeing such a "cute" code, ten thousand "xx horses" galloped past in my heart.
What disgusts me the most is that it actually implements the function of "closing overtime orders" with timed tasks . Now that I think about it, I can't laugh or cry. Let's first analyze why everyone is resisting the use of timed tasks to implement this function.
timed task
Closing timeout order is the operation of closing the order without completing the payment within a period of time after the order is created. This function generally requires that the timeout time of each order is consistent .
If we use a scheduled task to perform this operation, it is difficult to grasp the time interval of scheduled task polling:
- The time interval is small enough to achieve the time consistency problem we mentioned within the allowable range of error, but frequent scanning of the database and execution of scheduled tasks will cause consumption of network IO and disk IO, which will have a certain impact on real-time transactions;
- The time interval is relatively large. Since the creation time of each order is inconsistent, it is difficult to meet the above consistency requirements. Examples are as follows:
Assuming that the 30-minute order timeout is automatically closed, the execution interval of the scheduled task is 30 minutes:
- We place an order in 5 minutes;
- When the time comes to the 30th minute, the scheduled task is executed once, but our order does not meet the conditions and is not executed;
- When the time comes to the 35th minute, the order reaches the closing condition, but the scheduled task is not executed, so it is not executed;
- When the time comes to the 60th minute, we start to execute our order closing operation, and at this time, the error reaches 25 minutes.
After all this, we need to abandon this method.
delay queue
In order to meet the needs of the leader, I reached out to the message queue: RabbitMQ. Although it does not provide the function of delay queue itself, we can use its survival time and the characteristics of dead letter switch to realize it indirectly.
First of all, let's briefly introduce what is survival time? What is a dead letter exchange?
survival time
The full spelling of survival time is Time To Live, abbreviation TTL. It supports setting both the message itself (the key to the delay queue) and the queue (all messages in the queue have the same expiration time).
- Set the message itself: Even if the message expires, it will not be erased from the queue immediately, because whether each message expires is determined before it is delivered to the consumer;
- Set the queue: once the message expires, it will be erased from the queue;
If these two methods are used at the same time, the value with the smaller expiration time shall prevail. When the message reaches the expiration time and has not been consumed, then the message is "dead", and we call it a dead letter message.
Conditions for a message to become a dead letter:
- The message is rejected (
basic.reject/basic.nack), andrequeue=false; - The expiration time of the message has expired;
- The queue reaches its maximum length;
Queue Setup Considerations
- The setting of this attribute in the queue is valid only when the queue is declared for the first time. If the queue already exists at the beginning and does not have this attribute, the queue must be deleted and declared again;
- The queue
ttlcan only be set to a fixed value, once set, it cannot be changed, otherwise an exception will be thrown;
dead letter exchange
Dead letter switch Quanpin Dead-Letter-Exchange, short for DLX.
When a message becomes a dead letter in a queue, if the queue where the message is located has x-dead-letter-exchangeparameters set, it will be sent to x-dead-letter-exchangethe switch with the corresponding value. This switch is called a dead letter switch, and the dead letter switch The bound queue is a dead letter queue.
x-dead-letter-exchange: Resend the dead letter to the designated switch after the dead letter occurs;x-dead-letter-routing-key: After the dead letter appears, the dead letter will be sent according to the specified method againrouting-key. If it is not set, the message itself will be used by default.routing-key
The difference between a dead letter queue and a normal queue is that its
RoutingKeysumExchangeneeds to be used as a parameter and bound to a normal queue.
Practical teaching
Let’s take a picture first to feel our overall thinking
- The producer sends
ttla message with the switch and routes it to the delay queue; - Bind the dead letter switch and dead letter forwarding in the delay queue
routing-key; - After the message in the delay queue reaches the delay time, it becomes a dead letter and is forwarded to the dead letter switch and routed to the dead letter queue;
- Finally for consumer consumption.
We implement the code based on the above :
configuration class
@Configuration
public class DelayQueueRabbitConfig {
public static final String DLX_QUEUE = "queue.dlx";//死信队列
public static final String DLX_EXCHANGE = "exchange.dlx";//死信交换机
public static final String DLX_ROUTING_KEY = "routingkey.dlx";//死信队列与死信交换机绑定的routing-key
public static final String ORDER_QUEUE = "queue.order";//订单的延时队列
public static final String ORDER_EXCHANGE = "exchange.order";//订单交换机
public static final String ORDER_ROUTING_KEY = "routingkey.order";//延时队列与订单交换机绑定的routing-key
/**
* 定义死信队列
**/
@Bean
public Queue dlxQueue(){
return new Queue(DLX_QUEUE,true);
}
/**
* 定义死信交换机
**/
@Bean
public DirectExchange dlxExchange(){
return new DirectExchange(DLX_EXCHANGE, true, false);
}
/**
* 死信队列和死信交换机绑定
* 设置路由键：routingkey.dlx
**/
@Bean
Binding bindingDLX(){
return BindingBuilder.bind(dlxQueue()).to(dlxExchange()).with(DLX_ROUTING_KEY);
}
/**
* 订单延时队列
* 设置队列里的死信转发到的DLX名称
* 设置死信在转发时携带的 routing-key 名称
**/
@Bean
public Queue orderQueue() {
Map<String, Object> params = new HashMap<>();
params.put("x-dead-letter-exchange", DLX_EXCHANGE);
params.put("x-dead-letter-routing-key", DLX_ROUTING_KEY);
return new Queue(ORDER_QUEUE, true, false, false, params);
}
/**
* 订单交换机
**/
@Bean
public DirectExchange orderExchange() {
return new DirectExchange(ORDER_EXCHANGE, true, false);
}
/**
* 把订单队列和订单交换机绑定在一起
**/
@Bean
public Binding orderBinding() {
return BindingBuilder.bind(orderQueue()).to(orderExchange()).with(ORDER_ROUTING_KEY);
}
}
Send a message
@RequestMapping("/order")
public class OrderSendMessageController {
@Autowired
private RabbitTemplate rabbitTemplate;
@GetMapping("/sendMessage")
public String sendMessage(){
String delayTime = "10000";
//将消息携带路由键值
rabbitTemplate.convertAndSend(DelayQueueRabbitConfig.ORDER_EXCHANGE, DelayQueueRabbitConfig.ORDER_ROUTING_KEY,
"发送消息！",message->{
message.getMessageProperties().setExpiration(delayTime);
return message;
});
return "ok";
}
}
consumption news
@Component
@RabbitListener(queues = DelayQueueRabbitConfig.DLX_QUEUE)//监听队列名称
public class OrderMQReciever {
@RabbitHandler
public void process(String message){
System.out.println("OrderMQReciever接收到的消息是："+ message);
}
}
test
By calling the interface, it is found that the message will be consumed after 10 seconds
problem escalation
Since the development environment and the test environment use the same switch and queue, the delay time for sending is 30 minutes. However, in order to make it easier for the test students to test in the test environment, the time of the test environment was manually changed to 1 minute.
Problem recurrence
Then the problem came: the message with a delay of 1 minute was not consumed immediately, but was consumed after the 30-minute message was consumed. As for the reason, we will analyze it next, and first use the code to reproduce the problem for everyone.
@GetMapping("/sendManyMessage")
public String sendManyMessage(){
send("延迟消息睡10秒",10000+"");
send("延迟消息睡2秒",2000+"");
send("延迟消息睡5秒",5000+"");
return "ok";
}
private void send(String msg, String delayTime){
rabbitTemplate.convertAndSend(DelayQueueRabbitConfig.ORDER_EXCHANGE,
DelayQueueRabbitConfig.ORDER_ROUTING_KEY,
msg,message->{
message.getMessageProperties().setExpiration(delayTime);
return message;
});
}
The execution results are as follows:
OrderMQReciever接收到的消息是：延迟消息睡10秒
OrderMQReciever接收到的消息是：延迟消息睡2秒
OrderMQReciever接收到的消息是：延迟消息睡5秒
The reason is that the delay queue also satisfies the first-in-first-out feature of the queue. When the 10-second message is not out of the queue, the subsequent messages cannot be successfully dequeued, causing the subsequent messages to be blocked, and the precise delay cannot be achieved.
problem solved
We can use x-delay-messageplugins to solve this problem
The message delay range is Delay > 0, Delay =< ?ERL_MAX_T (the range that can be set in Erlang is (2^32)-1 milliseconds)
- When the producer sends a message to the exchange, it does not enter immediately, but first persists the message to
Mnesia(a distributed database management system); - The plugin will try to confirm whether the message is expired;
- If the message expires, the message will be delivered to the target queue through
x-delayed-typethe type marked switch for consumption by consumers;
practice
Official website download , what I use here is v3.8.0.ezto download the file and put it /usr/local/soft/rabbitmq_server-3.7.14/pluginsin the path of the server, and execute rabbitmq-plugins enable rabbitmq_delayed_message_exchangethe command.
Appears as shown in the figure, which means the installation is successful.
configuration class
@Configuration
public class XDelayedMessageConfig {
public static final String DIRECT_QUEUE = "queue.direct";//队列
public static final String DELAYED_EXCHANGE = "exchange.delayed";//延迟交换机
public static final String ROUTING_KEY = "routingkey.bind";//绑定的routing-key
/**
* 定义队列
**/
@Bean
public Queue directQueue(){
return new Queue(DIRECT_QUEUE,true);
}
/**
* 定义延迟交换机
* args:根据该参数进行灵活路由，设置为“direct”，意味着该插件具有与直连交换机具有相同的路由行为，
* 如果想要不同的路由行为，可以更换现有的交换类型如：“topic”
* 交换机类型为 x-delayed-message
**/
@Bean
public CustomExchange delayedExchange(){
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-delayed-type", "direct");
return new CustomExchange(DELAYED_EXCHANGE, "x-delayed-message", true, false, args);
}
/**
* 队列和延迟交换机绑定
**/
@Bean
public Binding orderBinding() {
return BindingBuilder.bind(directQueue()).to(delayedExchange()).with(ROUTING_KEY).noargs();
}
}
Send a message
@RestController
@RequestMapping("/delayed")
public class DelayedSendMessageController {
@Autowired
private RabbitTemplate rabbitTemplate;
@GetMapping("/sendManyMessage")
public String sendManyMessage(){
send("延迟消息睡10秒",10000);
send("延迟消息睡2秒",2000);
send("延迟消息睡5秒",5000);
return "ok";
}
private void send(String msg, Integer delayTime){
//将消息携带路由键值
rabbitTemplate.convertAndSend(
XDelayedMessageConfig.DELAYED_EXCHANGE,
XDelayedMessageConfig.ROUTING_KEY,
msg,
message->{
message.getMessageProperties().setDelay(delayTime);
return message;
});
}
}
consumption news
@Component
@RabbitListener(queues = XDelayedMessageConfig.DIRECT_QUEUE)//监听队列名称
public class DelayedMQReciever {
@RabbitHandler
public void process(String message){
System.out.println("DelayedMQReciever接收到的消息是："+ message);
}
}
test
DelayedMQReciever接收到的消息是：延迟消息睡2秒
DelayedMQReciever接收到的消息是：延迟消息睡5秒
DelayedMQReciever接收到的消息是：延迟消息睡10秒
In this way, our problem is successfully solved.
limitation
Delayed messages are stored in a Mnesiatable with only one disk copy on the current node, and they will survive node restarts.
While the timer that triggers the scheduled delivery is not persisted, it will be reinitialized during plugin activation on node startup. Obviously, having only one copy of scheduled messages in the cluster means that losing that node or disabling a plugin on it will lose messages residing on that node.
The current design of the plug-in is not suitable for scenarios with a large number of delayed messages (such as tens of thousands or millions), and another source of variability of the plug-in is that it relies on timers, and a certain number of long messages are used in the system Erlang. After the time timer, they start competing for scheduler resources and the time drift keeps accumulating.
Reply to "rabbitMQ" to get the source code!
That’s all for today’s content. If you have different opinions or better ideas, please contact Ah Q.
Ah Q will continue to update articles on java combat. If you are interested, you can pay attention to Ah Q or come to the technical group Discuss the problem, the friend who likes it is worthy of deep friendship!