RocketMQ source code analysis - distributed transaction solution

Message queue solves transaction problems

If a message queue is used to resolve the transaction, at which stage will the message be sent to MQ?

1. Deduct the money first and then
   send the message to RocketMQ. Deduct the money first and then send the message. If the message sending times out (it may succeed or fail in MQ), then the status will be difficult to judge.

2. Send a message to RocketMQ first and then deduct
   the money. The deduction success message is sent successfully, but if the local deduction business fails, the message has been sent to MQ, and the second phase of adding money will be executed successfully.

So no matter which solution is adopted, there will be problems in handling it.

In fact, after careful analysis, the key point of the problem is that RocketMQ cannot change the transaction status of the message sender. Therefore, RocketMQ's distributed transaction solution has been optimized.

RocketMQ’s distributed transaction solution

RocketMQ introduces semi-transaction and transaction review mechanisms in distributed transactions.

Semi-transaction:
Send a message to rocketmq, but the message is only stored in the commitlog, but is not visible in the consumeQueue, that is, the consumer (subscriber) cannot see the message.

Transaction review:

RocketMq will periodically traverse the semi-transaction messages in the commitlog. This transaction review mechanism can participate in the transaction of the message sender from the perspective of RocketMQ.

RocketMQ distributed transaction case code

This is the producer of distributed transactions and completes the sending of half transactions.

Through transaction review, if in the executeLocalTransaction method of the TransactionListenerImpl class, if the local transaction is executed successfully, commit_message is submitted, and the consumer can consume the message.

If there are some time-consuming operations that cannot be confirmed at this step, you can submit UNKNOW and leave it to the scheduled task review for processing.

Distributed transaction source code analysis

From the process of distributed transactions, it can be analyzed from three aspects: message sending, confirmation/rollback, and review.

Message sending source code analysis

Producer

Broker

RocketMQ uses Netty to handle the network. When the broker receives a request to write a message, it will enter the processRequest method in the SendMessageProcessor class.

Finally enter the asyncPutMessage method in the DefaultMessageStore class to store the message

Combining the diagram and the code, we can see that when a transaction message is sent, the topic where the message is actually stored is a system topic: RMQ_SYS_TRANS_HALF_TOPIC

At the same time, the message stores information and queues related to the original topic of the message.

Confirm/rollback source code analysis

Producer

DefaultMQProducerImpl class sendMessageInTransaction method

Broker

EndTransactionProcessor class

Back-check source code analysis

Producer

During transaction review, Producer is the server, so it needs to register for service processing.

DefaultMQProducerImpl class checkTransactionState method

DefaultMQProducerImpl class processTransactionState method

Broker

When the Broker is started, it acts as a client and regularly accesses the client for transaction review.

Transaction review is a scheduled network call initiated by the Broker (every 60s), so the transaction review is not necessarily an interval of 60s for the first time when the client is started, and is generally less than 60s (because the transaction review is initiated by the broker) , it is not initiated regularly by the client)

Summarize

RocketMQ supports distributed transactions, mainly through the two-phase commit protocol. In the first phase, the system sends a prepared message to MQ. If the prepared message fails to be sent, the operation is canceled directly and is not executed. If the message is sent successfully, then execute the local transaction (executeLocalTransaction). If successful, tell MQ to send a confirmation message. If it fails, tell MQ to send a rollback message. In the second phase, if a confirmation message is sent, system B will receive the confirmation message and then execute the local transaction. If the confirmation or rollback message fails to be sent, the broker has a polling mechanism to query the local transaction status based on the unique ID. MQ will automatically poll all prepared messages at regular intervals and call back your interface (checkLocalTransaction) to ask whether the message is local. Transaction processing failed. For all messages that have not been confirmed, should we continue to retry or roll back?