Introduce RocketMQ transaction message principle and code

1. The origin of the transaction message

1. Case

Quoting the official shopping case:

When Xiao Ming purchases a 100 yuan item, the account deducts 100 yuan and he needs to ensure that the downstream points system adds 100 points to Xiao Ming's account. The account system and the points system are two independent systems, one should be reduced by 100 yuan, and the other should be increased by 100 points. As shown below:

2. The problem

• The account service deduction was successful, and the notification point system was also successful, but it failed when the points were increased, and the data was inconsistent.
• The account service deduction was successful, but the notification point system failed, so the points will not increase, and the data is inconsistent.

3. Scheme

RocketMQ's solution to the first problem is: if the consumption fails, it will automatically retry. If the consumption fails after a few retries, then this situation needs to be solved manually, such as putting it in the dead letter queue and then Manually check the cause and deal with it.

RocketMQ's solution to the second problem is: if your deduction succeeds, but fails when writing a message to mq, then RocketMQ will roll back the message, and at this time we can also roll back our deduction operation.

Second, the principle of transaction messages

1. Principle diagram

2. Detailed process

• 1. The Producer sends a half message (Half Message) to the broker.

I really want to complain about why it is called a half-message, which is difficult to understand. In fact, this is a prepare message, pre-sent message.

• After the Half Message is sent successfully, the local transaction will be executed.
• If the local transaction is executed successfully, it returns commit, if the execution fails, it returns rollback. (This is determined by the developer himself in the callback method of the transaction message commit or rollback)
• Producer sends the commit of the previous step or rollback to the broker. There are two situations here:

1. If the broker receives a commit/rollback message:

• If a commit is received, the broker believes that the entire transaction is okay and the execution is successful. Then it will send a message to the Consumer end for consumption.
• If a rollback is received, the broker thinks that the local transaction execution has failed, and the broker will delete the Half Message and not send it to the Consumer.

2. If the broker does not receive the message (if the execution of the local transaction suddenly goes down, the execution result of the equivalent local transaction returns unknow, it will be treated as if the broker did not receive the confirmation message.):

• The broker will periodically review the execution results of the local transaction: if the review result is that the local transaction has been executed, it returns commit, if not, it returns rollback.
• The results of the Producer's backcheck are sent to the Broker. If the broker receives a commit, the broker considers the entire transaction to be executed successfully. If it is a rollback, the broker considers the local transaction to fail, and the broker deletes the Half Message and does not send it to the consumer. If the broker does not receive the result of the back-check (or if it finds unknow), the broker will repeat the back-check regularly to ensure that the final transaction result is found. Both the time interval and the number of repeated back-checks can be configured.

Three, transaction message realization process

1. Implementation process

To put it simply, the transaction message is a listener with a callback function. In the callback function, we perform business logic operations, such as giving the account -100 yuan, and then sending a message to the mq of the points. At this time, if the account -100 succeeds, And if the message is successfully sent to mq, the message status is set to commit. At this time, the broker will send the half message to the real topic. At the beginning, it was sent to the semi-message queue, and it did not exist in the real topic queue. Only after confirming the commit will it be transferred.

2. Remedial plan

If the transaction fails to respond immediately due to interruption or other network reasons, RocketMQ treats it as UNKNOW. RocketMQ transaction messages also provide a remedy: query the transaction status of the transaction messages regularly. This is also a callback function. Compensation can be done in it. The compensation logic developer writes it himself. If it succeeds, it returns to commit and it's done.

Four, code examples

1. Code

package com.chentongwei.mq.rocketmq;

import org.apache.rocketmq.client.producer.LocalTransactionState;
import org.apache.rocketmq.client.producer.TransactionListener;
import org.apache.rocketmq.client.producer.TransactionMQProducer;
import org.apache.rocketmq.client.producer.TransactionSendResult;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageExt;

import java.util.Date;

/**
 * Description:
 *
 * @author TongWei.Chen 2020-06-21 11:32:58
 */
public class ProducerTransaction2 {
    public static void main(String[] args) throws Exception {
        TransactionMQProducer producer = new TransactionMQProducer("my-transaction-producer");
        producer.setNamesrvAddr("124.57.180.156:9876");

        // 回调
        producer.setTransactionListener(new TransactionListener() {
            @Override
            public LocalTransactionState executeLocalTransaction(Message message, Object arg) {
                LocalTransactionState state = null;
                //msg-4返回COMMIT_MESSAGE
                if(message.getKeys().equals("msg-1")){
                    state = LocalTransactionState.COMMIT_MESSAGE;
                }
                //msg-5返回ROLLBACK_MESSAGE
                else if(message.getKeys().equals("msg-2")){
                    state = LocalTransactionState.ROLLBACK_MESSAGE;
                }else{
                    //这里返回unknown的目的是模拟执行本地事务突然宕机的情况（或者本地执行成功发送确认消息失败的场景）
                    state = LocalTransactionState.UNKNOW;
                }
                System.out.println(message.getKeys() + ",state:" + state);
                return state;
            }

            /**
             * 事务消息的回查方法
             */
            @Override
            public LocalTransactionState checkLocalTransaction(MessageExt messageExt) {
                if (null != messageExt.getKeys()) {
                    switch (messageExt.getKeys()) {
                        case "msg-3":
                            System.out.println("msg-3 unknow");
                            return LocalTransactionState.UNKNOW;
                        case "msg-4":
                            System.out.println("msg-4 COMMIT_MESSAGE");
                            return LocalTransactionState.COMMIT_MESSAGE;
                        case "msg-5":
                            //查询到本地事务执行失败，需要回滚消息。
                            System.out.println("msg-5 ROLLBACK_MESSAGE");
                            return LocalTransactionState.ROLLBACK_MESSAGE;
                    }
                }
                return LocalTransactionState.COMMIT_MESSAGE;
            }
        });

        producer.start();

        //模拟发送5条消息
        for (int i = 1; i < 6; i++) {
            try {
                Message msg = new Message("transactionTopic", null, "msg-" + i, ("测试，这是事务消息！ " + i).getBytes());
                producer.sendMessageInTransaction(msg, null);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }
}

2. Results

msg-1,state:COMMIT_MESSAGE
msg-2,state:ROLLBACK_MESSAGE
msg-3,state:UNKNOW
msg-4,state:UNKNOW
msg-5,state:UNKNOW

msg-3 unknow
msg-3 unknow
msg-5 ROLLBACK_MESSAGE
msg-4 COMMIT_MESSAGE

msg-3 unknow
msg-3 unknow
msg-3 unknow
msg-3 unknow

3. Control console

4. Result analysis

• Only msg-1 and msg-4 were successfully sent. msg-4 is ahead of msg-1 because msg-1 succeeded first, and msg-4 succeeded only after back-checking. Comes in reverse chronological order.
• First output five results, corresponding to five messages

msg-1,state:COMMIT_MESSAGE
msg-2,state:ROLLBACK_MESSAGE
msg-3,state:UNKNOW
msg-4,state:UNKNOW
msg-5,state:UNKNOW

• Then entered the review, msg-3 was still unknow, msg-5 rolled back, and msg-4 submitted the transaction. So at this time msg-4 can be seen in the control console.
• After a while, I checked msg-3 again and found that it was still unknow, so I kept checking back.

The time interval and the number of back-checks are configurable. The default is that if the check-back fails 15 times, the message will be discarded.

Five, questions

Question: Are Spring transactions and regular distributed transactions not good? Is Rocketmq's affairs superfluous?

MQ is used for decoupling. Previously, distributed transactions directly operated the account system and points system. But the two of them are the existence of strong coupling. If an mq is inserted in the middle, the account system will send a message to the mq after the account system is operated. At this time, as long as the sending is guaranteed to be successful, it will be submitted, and the sending will be rolled back if the sending fails. How to guarantee this step depends on the transaction. And there are quite a lot of people using RocketMQ for distributed transactions.