MQ transaction message usage scenarios
The "transaction" in the message queue mainly solves the problem of data consistency between message producers and message consumers.
Take the e-commerce we are familiar with as an example. Generally speaking, when a user is shopping on an e-commerce app, he first adds the product to the shopping cart, then places an order for several products together, and finally pays, completes the shopping process, and can happily wait for the delivery.
There is a step in this process that requires the use of message queues. After the order system creates an order, it sends a message to the shopping cart system to delete the ordered items from the shopping cart. Because the step of deleting the ordered product from the shopping cart is not a necessary step in the main process of the user placing an order and paying, it is more reasonable to use a message queue to asynchronously clean up the shopping cart.
For the order system, it actually performs two steps in the process of creating an order:
1 Insert an order data in the order library to create an order;
2 Send a message to the message queue, and the content of the message is the order just created .
The shopping cart system subscribes to the corresponding topic, receives the message of order creation, then cleans up the shopping cart and deletes the items in the order in the shopping cart.
In a distributed system, any of the steps mentioned above may fail. If no processing is done, there may be inconsistencies between the order data and the shopping cart data
. Clear the shopping cart;
the order was not successfully created, but the items in the shopping cart were cleared.
Then the problem we need to solve can be summarized as: in the case that any of the above steps may fail, we must also ensure the data consistency of the two libraries of the order library and the shopping garage.
For the operation of cleaning the shopping cart after the shopping cart system receives the order creation success message, the failure handling is relatively simple, as long as the shopping cart is cleaned up successfully and then the consumption confirmation can be submitted. If it fails, because the consumption confirmation has not been submitted, the message queue will Automatically retry.
The key point of the problem is the order system. The two steps of creating an order and sending a message are either successful or both fail. It is not allowed for one to succeed and the other to fail.
This is the problem that transactions need to solve.
How does the message queue implement distributed transactions?
Transactional messages require corresponding functions provided by the message queue to be realized. Both Kafka and RocketMQ provide transaction-related functions.
Back to the order and cart example.
First, the order system starts a transaction on the message queue. Then the order system sends a "half message" to the message server. This half message does not mean that the content of the message is incomplete, but the content it contains is the complete message content. Or, the message is invisible.
After the half message is sent successfully, the order system can execute the local transaction, create an order record in the order library, and submit the database transaction of the order library. Then decide to commit or rollback the transaction message according to the execution result of the local transaction. If the order is successfully created, submit the transaction message, and the shopping cart system can consume this message to continue the subsequent process. If the order creation fails, the transaction message is rolled back and the shopping cart system does not receive the message. In this way, the consistency requirement of "either all succeed or both fail" is basically realized.
In this implementation process, there is a problem that has not been resolved. What if there is a failure in committing the transaction message in the fourth step? For this problem, Kafka and RocketMQ give two different solutions.
Kafka's solution is relatively simple and rude, throwing an exception directly and letting the user handle it by themselves. We can retry the submission repeatedly in the business code until the submission is successful, or delete the previously created order to compensate. RocketMQ gives another solution
Distributed transaction implementation in RocketMQ
In the transaction implementation in RocketMQ, a transaction reverse check mechanism is added to solve the problem of transaction message submission failure. If the Producer is the order system, and a network exception occurs when a transaction message is submitted or rolled back, and the Broker of RocketMQ does not receive the request for submission or rollback, the Broker will regularly check the status of the local transaction corresponding to the transaction on the Producer, and then According to the results of the reverse check, it is decided to commit or roll back the transaction.
In order to support this transaction anti-check mechanism, our business code needs to implement an interface for anti-check local transaction status to inform RocketMQ whether the local transaction is successful or failed.
In our example, the logic of back-checking local transactions is also very simple. We only need to check whether the order exists in the order library according to the order ID in the message. If the order exists, it will return success, otherwise it will return failure.
RocketMQ will automatically commit or roll back the transaction message according to the result of the transaction reverse check.
The realization of this anti-check local transaction does not depend on the sender of the message, that is, any data on an instance node of the order service. In this case, even if the order service node that sent the transaction message is down, RocketMQ can still perform reverse checks through other order service nodes to ensure the integrity of the transaction.
Based on the implementation of general transaction messages mentioned above and RocketMQ's transaction anti-check mechanism, the process of using RocketMQ transaction message function to realize distributed transactions is as follows:
Many people often have a question: why not wait for the order to be created successfully before sending the order data to the message queue? In this way, there is no need to consider the situation where the order creation fails and the message is sent.
If the order is created first, the current service cannot work normally due to irresistible factors, and no message is sent to the shopping cart system. In this case, the order has been created and the shopping cart has not been emptied.
In the case of sending a half-message, you can periodically query the status of the transaction and then roll back the operation or resend the message according to the specific business
to ensure the final consistency of the data.
Code implementation and underlying code principle: RocketMQ source code analysis 9.3 9.4 9.5