1. Idempotency
1.1 Concept
The results of one request or multiple requests initiated by the user for the same operation are consistent, and there will be no side effects due to multiple clicks.
Taking payment as an example, the user pays after purchasing the product, and the deduction is successful, but when the result is returned, the network is abnormal, and the money has been deducted at this time; the user clicks the button again, and the second deduction will be performed at this time, and the returned result is successful, and the user Checked the balance and found that more money was deducted, and the transaction records became two. In the previous single-application system, we only need to put the data operation into the transaction, and immediately roll back when an error occurs, but there may be network interruption or exception when responding to the client.
1.2 Repeated message consumption
When the consumer consumes the message in MQ, MQ has sent the message to the consumer, and the network is interrupted when the consumer returns ack to MQ, so MQ does not receive the confirmation message, and the message will be resent to other consumers. Or send it to the consumer again after the network is reconnected, but in fact the consumer has successfully consumed the message, which causes repeated consumption of the message.
1.3 Solutions
The idempotent solution of MQ consumers generally uses a global ID or a unique identifier such as a timestamp, UUID, or the message in MQ consumed by an order consumer can also be judged by using the id of MQ, or can generate one according to its own rules A globally unique id, which is used to determine whether the message has been consumed each time a message is consumed.
1.4 Guarantee of idempotence on the consumer side
During the peak period of business when a large number of orders are generated, there may be repeated messages on the production side. At this time, the consumer side must achieve idempotence, which means that our messages will never be consumed multiple times, even if we receive the same message. news.
The mainstream idempotence in the industry has two operations:
Unique ID+fingerprint code mechanism, using the database primary key to deduplicate
Use the atomicity of redis to achieve (recommended)
1.4.1 Unique ID+fingerprint code mechanism
Fingerprint code: the unique information code given by some of our rules or timestamps plus other services. It is not necessarily generated by our system. It is basically spliced by our business rules, but the uniqueness must be guaranteed. Then use the query statement to judge whether the id exists in the database.
Its advantage is that it is simple to implement, and it is only necessary to query to determine whether it is repeated after splicing; the disadvantage is that in high concurrency, if it is a single database, there will be a write performance bottleneck. Not our most recommended way.
1.4.2 Redis Atomicity
Using redis to execute the setnx command is naturally idempotent. So as to achieve non-repeated consumption.
2. Priority queue
2.1 Usage Scenarios
For example, there is an order reminder scenario in our system. Our customer places an order on Tmall, and Taobao will push the order to us in time. For us, tmall merchants must distinguish between large and small customers. For example, big merchants like Apple and Xiaomi can generate a lot of profits for us in a year, so their orders must be processed first. This is the scenario of order reminders.
In the past, our backend system used redis to store regular polling. Redis can only use List as a simple message queue, and cannot implement a priority scenario. Therefore, when the order volume is large, we need to use RabbitMQ is modified and optimized. If it is found that the order of a large customer is given a relatively high priority, otherwise it is the default priority.
2.2 Implementation
The things that need to be done to make the queue realize the priority are as follows: the queue needs to be set as a priority queue, the message needs to be set with the priority of the message, and the consumer needs to wait for the message to be sent to the queue before consuming it, so as to have the opportunity to process the message Sort.
2.2.1 Producer
public class Producer {
private static final String QUEUE_NAME="hello";
public static void main(String[] args) throws Exception {
try (Channel channel = RabbitMqUtils.getChannel();) {
//给消息赋予一个 priority 属性
AMQP.BasicProperties properties =
new AMQP.BasicProperties().builder().priority(5).build();
for (int i = 1; i <11; i++) {
String message = "info"+i;
if(i==5){
channel.basicPublish("", QUEUE_NAME, properties, message.getBytes());
}else{
channel.basicPublish("", QUEUE_NAME, null, message.getBytes());
}
System.out.println("发送消息完成:" + message);
}
}
}
}2.2.2 Consumers
public class Consumer {
private static final String QUEUE_NAME="hello";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMqUtils.getChannel();
//设置队列的最大优先级 最大可以设置到 255 官网推荐 1-10 如果设置太高比较吃内存和 CPU
Map<String, Object> params = new HashMap();
params.put("x-max-priority", 10);
channel.queueDeclare(QUEUE_NAME, true, false, false, params);
System.out.println("消费者启动等待消费......");
DeliverCallback deliverCallback=(consumerTag, delivery)->{
String receivedMessage = new String(delivery.getBody());
System.out.println("接收到消息:"+receivedMessage);
};
channel.basicConsume(QUEUE_NAME,true,deliverCallback,(consumerTag)->{
System.out.println("消费者无法消费消息时调用，如队列被删除");
});
}
}3. Lazy queue
3.1 Usage Scenarios
RabbitMQ has introduced the concept of lazy queues since version 3.6.0. The lazy queue will store the message on the disk as much as possible , and will be loaded into the memory when the consumer consumes the corresponding message. One of its important design goals is to be able to support longer queues, that is, to support more message storage. Inert queues are necessary when consumers cannot consume messages for a long time due to various reasons (such as consumer offline, downtime, or shutdown due to maintenance, etc.).
By default, when a producer sends a message to RabbitMQ, the message in the queue will be stored in memory as much as possible, so that the message can be sent to the consumer more quickly. Even persistent messages have an in-memory copy as they are written to disk. When RabbitMQ needs to release the memory, it will page the messages in the memory to the disk. This operation will take a long time and will also block the operation of the queue, so that it cannot receive new messages. Although RabbitMQ developers have been upgrading related algorithms, the effect is not ideal, especially when the message volume is particularly large.
3.2 Declare the queue as a lazy queue
When declaring the queue, you can set the mode of the queue through the "x-queue-mode" parameter, and the values are "default" and "lazy". The following example demonstrates the declaration details of a lazy queue:
Map<String, Object> args = new HashMap<String, Object>();
args.put("x-queue-mode", "lazy");
channel.queueDeclare("myqueue", false, false, false, args);3.3 Memory overhead comparison
In the case of sending 1 million messages and each message occupies about 1KB, the memory occupied by the ordinary queue is 1.2GB, while the lazy queue only occupies 1.5MB. Because the lazy queue saves a large number of messages on disk.