7. Springboot 2.x integrates RabbitMQ to realize message sending confirmation and message receiving confirmation to realize 100% message delivery

Springboot2.x integrates RabbitMQ to realize message sending confirmation and message receiving confirmation to realize 100% message delivery

Preface

• How does the message guarantee 100% delivery success?
• Detailed explanation of the concept of idempotence
• How to avoid the problem of repeated consumption of messages during the peak business period when massive orders are generated?
• Confirm confirmation message, Return return message

How does the message guarantee 100% delivery success?

1.1 What is reliable delivery on the production side?

• Guarantee the success of the message
• Ensure the successful reception of MQ nodes
• The sender receives the confirmation response from the MQ node (Broker)
• Complete message compensation mechanism

The first three steps may not guarantee 100% delivery of the message. So add the fourth step

BAT/TMD Internet giant's solution:

• Message drop database, marking the message status
  When sending a message, you need to persist the message in the database and set a status (not sent, sending, arriving) for the message. When the message status changes, a change needs to be made to the message. Do a round-robin operation for messages that have not arrived and resend them. There is also a limit of 3-5 times for the number of rotations. Ensure that the message can be sent successfully.

• Delayed delivery of messages, second confirmation, callback check

Which scheme to adopt depends on the amount of concurrent services and messages.

1.2 The first scheme:

Production side-reliable delivery

For example: send an order message as follows.

step1: Store the order message (create an order), the business data is stored in the database, and the message is also stored in the database. Disadvantages: Need to persist twice. (Status:0)
step2: On the premise that step1 is successful, send a message.
Step3: After the Broker receives the message, confirm to our production end. Confirm Listener asynchronously monitors the messages sent back by Broker.
step4: Grab the specified message and update (status=1), indicating that the message has been delivered successfully.

step5: Distributed timing task gets the message status, if it is equal to 0, then grab the data.
step6: resend the message
step7: set the retry limit 3 times. If the message fails after 3 retries, then (status=2), the message is considered a failure.

Inquiring why these messages failed may require manual inquiry.

Assume that step 2 is executed successfully, and step 3 is due to network flash. Then confirm will never receive the message, then we need to set a rule:
for example: when the message is stored in the database, set a critical value timeout=5min, when it exceeds 5min, the data will be captured.
Or write a timed task to grab the status=0 messages every 5 minutes. There may be a small problem: the message is sent out, the timed task is just executed, and the Confirm has not been received, the timed task will be executed, which will cause the message to be executed twice.
More refined operation: message timeout tolerance limit. If confirm does not receive the message within 2-3 minutes, it will be resent.

Note: In the face of small-scale applications, transactions can be added to ensure transaction consistency. However, in the face of high concurrency in large factories, no transactions are added. The performance splicing of transactions is very serious, but compensation is made.

To ensure MQ, we think if the first type of reliable delivery is appropriate in the context of high concurrency?

The first scheme has two storage of data, one storage of business data, and one storage of messages. This is a bottleneck for data storage.
In fact, we only need to store the business.

Delayed delivery of messages, second confirmation, callback check

This method does not necessarily guarantee 100% success, but it can also guarantee 99.99% of messages. If you encounter a particularly extreme situation, then you can only need to manually compensate, or timed tasks to do it.
The second method is mainly to reduce the operation of the database.

1.2 The second scheme:

step1: After the business message is successfully stored in the database, the first message is sent.
step2: Similarly, after the message is successfully stored in the database, the second message is sent, and the two messages are sent at the same time. The second message is delayed check, which can be set to 2min or 5min to delay sending.
step3: The consumer monitors the specified queue.
step4: After the consumer has processed the message, a new message send confirm is generated internally. Delivery to MQ Broker.
step5: Callback Service The callback service monitors the MQ Broker. If the message sent by the Downstream service is received, it can be determined that the message is sent successfully, and the execution message is stored in the MSG DB.
step6: Check Detail to check and monitor the delayed delivery message of step2. At this time, the two monitoring queues are not the same. After 5 minutes, the Callback service receives the message and checks the MSG DB. If you find that the previous message has been delivered successfully, you do not need to do other things. If the check fails, Callback compensates and actively sends RPC communication. Notify the upstream producer to resend the message.

The purpose of doing so: One less DB storage. The focus is not on 100% delivery success, but on performance.

2. The concept of idempotence

2.1 What is idempotence?

Idempotent (idempotent, idempotence) is a mathematical and computer science concept commonly found in abstract algebra, namely f(f(x)) = f(x). Simply put, the result of an operation executed multiple times is consistent with the result of one execution .

• We can learn from the optimistic locking mechanism of the database:
• For example, we execute a SQL statement to update the inventory:
• UPDATE T_REPS SET COUNT = COUNT - 1,VERSION = VERSION + 1 WHERE VERSION = 1

Use the method of adding the version number to ensure idempotence.

2.2 Consumer end-idempotence guarantee

How to avoid the problem of repeated consumption of messages during the peak business period when massive orders are generated?

In the case of high concurrency, a large number of messages will arrive at MQ, and the consumer needs to monitor a large number of messages. In this case, repeated delivery of messages, network flashes, etc. will inevitably occur. If you do not do idempotence, there will be repeated consumption of messages.
-The realization of idempotency on the consumer side means that our messages will never be consumed multiple times, even if we receive multiple identical messages, they will only be executed once.

Look at the mainstream idempotent operations of major Internet companies:
-Unique ID + fingerprint mechanism, using database primary keys to remove duplicates.
-Use Redis's atomicity to achieve
-Other technologies to achieve idempotence

2.2.1 Unique ID + fingerprint code mechanism

• Unique ID + fingerprint mechanism, using the database primary key to remove duplicates.
  Guarantee uniqueness
• SELECT COUNT(1) FROM T_ORDER WHERE ID = unique ID + fingerprint code
  If the query does not exist, add it. There is no need to do anything, and the consumer does not need to consume messages.
• Benefit: Simple to implement
• Disadvantages: performance bottleneck for database writes under high concurrency
• Solution: Follow up the ID to sub-database and sub-table to perform algorithmic routing to
  allocate traffic pressure.

2.2.2 Redis atomic feature implementation

The easiest to use Redis's self-increment.

• Issues that need to be considered when using Redis for idempotence.
• First: Do we need data to be stored in the database? If it is, the key problem to be solved is how to achieve atomicity in the database and cache?
  Adding transactions does not work. Redis and database transactions are not the same, and there is no guarantee of simultaneous success and failure. Do you have a better plan?
• Second: How to set the timing synchronization strategy if the storage is not performed, then all are stored in the cache?
  How to achieve the stability of cached data?

3. Confirm confirmation message

Understand the Confirm message confirmation mechanism:

• Message confirmation means that after the producer delivers the message, if the Broker receives the message, it will give us a response to the producer.
• The producer receives the response to determine whether the message is sent to the Broker normally. This method is also the core guarantee for the reliable delivery of the message!

3.1 Implement Confirm confirmation message

spring:
  application:
    name: zoo-plus-rabbitmq
  rabbitmq:
    virtual-host: /
    host: localhost
    port: 5672
    username: guest
    password: guest
    publisher-confirm-type: correlated #必须配置这个才会确认回调

Producer:

    /**
     * 测试消息确认回调（必须在yml配置publisher-confirm-type: correlated）
     */
    @GetMapping("sendConfirmCallback")
    public Resp sendConfirmCallback() {
    
    
        rabbitTemplate.setConfirmCallback((correlationData, ack, cause) -> {
    
    
            System.out.println("ack:" + ack);
            if (!ack) {
    
    
                //可以进行日志记录、异常处理、补偿处理等
                System.out.println("异常处理");
            }else{
    
    
                //更新数据库，可靠性投递机制
            }
        });
        amqpTemplate.convertAndSend("test-queue", "测试ack确认模式");
        return Resp.success("ok", null);
    }

4. Return message mechanism

• Return Listener is used to process some unroutable messages!
• Our message producer, by specifying an Exchange and Routingkey, sends the message to a certain queue, and then our consumer listens to the queue for consumption processing operations!
• But in some cases, if the current exchange does not exist or the specified routing key cannot be routed when we send a message, at this time, if we need to listen for such unreachable messages, we must use Return Listener!

There is a key configuration item in the basic API:

• Mandatory: If it is true, the listener will receive the message that the route is unreachable, and then perform subsequent processing. If it is false, the broker will automatically delete the message!

4.1 Implement Confirm confirmation message

Configuration

spring:
  application:
    name: zoo-plus-rabbitmq
  rabbitmq:
    virtual-host: /
    host: localhost
    port: 5672
    username: guest
    password: guest
    publisher-returns: true #支持发布返回

Code

    /**
     * 启动消息失败返回，比如路由不到队列时触发回调
     * 测试发布回调（必须在yml配置publisher-returns: true）
     */
    @GetMapping("sendReturnCallback")
    public Resp sendReturnCallback() {
        RabbitTemplate.ReturnCallback returnCallback = (message, replyCode, replyText, exchange, routingKey) -> {
            System.out.println("========returnCallback=============");
        };
        rabbitTemplate.setReturnCallback(returnCallback);
        amqpTemplate.convertAndSend("test-", "测试发布回调模式");
        return Resp.success("ok", null);
    }

Source address: https://gitee.com/zoo-plus/springboot-learn/tree/2.x/springboot-middleware/rabbitmq