Speaking of the transaction, you may naturally think of the database. Indeed, we use every day affairs of the scene, the vast majority are in operation when the database. Like MySQL, Oracle these mainstream relational database, also provides complete transaction realized. It also requires a transaction message queue Why do?
In fact, in many scenes, we "message" This process, often with the objective to inform another system or module to update the data, the message queue "transaction", the main problem is the problem of data consistency news producers and news consumers .
We still get familiar with the electricity suppliers to give an example. In general, when a user purchases the electricity supplier APP, first product added to the shopping cart, and then a few items together under a single, final payment to complete the shopping process, we can happily wait for a receipt.
This process there is a need to use step message queue, order system after the order is created, a message to the shopping cart system, which has been under a single commodity removed from the shopping cart. Because delete a single item from shopping cart at this step, the user is not a single payment under this major step process required to use asynchronous message queues to clean up cart is more reasonable design.
For ordering system, it is actually in the process of creating an order to perform the operation two steps:
- Order data in order to insert a database, create an order;
- A message to the message queue, the order is the content of the message you just created.
Shopping cart system subscribe to the appropriate subject matter, receiving an order to create a message, and then clean up the shopping cart, delete items in the order in the shopping cart.
In a distributed system, the steps mentioned above, any of the steps are likely to fail if no treatment, it may be inconsistent order data and shopping cart data occurs, for example:
- Created orders, not cleaned cart;
- Orders have not been created, inside the shopping cart of merchandise was cleared away.
We need to address that issue can be summarized as follows: In the case of any of the above steps are likely to fail, but also to ensure data consistency order store and shopping garage two libraries.
For the shopping cart system to create success message received orders to clean up cart this operation, the processing failure is relatively simple, as long as the successful implementation of the cart and then submit cleanup consumer to confirm, if it fails, because the consumer did not submit confirmation message queues automatic retry.
Key questions focused on the order system, create orders and send messages either of these two steps are operating succeed or fail are operating, but does not allow a successful case of another failure occurs.
That's affairs need to be resolved.
What is a distributed transaction?
What matters is that it? If we need to update some data, in order to ensure the integrity and consistency of these data, we hope that these update operations are either succeed or fail. As for the updated data is not limited to data in the database can be a file on disk, it can be a remote service, or data stored in other forms.
This is usually our understanding of the transaction. In fact, this description of the transaction is not too accurate nor complete, but it is easier to understand, in general, is correct. So I tend to like this in terms of the more abstract concept of "transaction."
A strict sense of the transaction implementation, should have four attributes: atomicity, consistency, isolation, durability. These four properties are usually called ACID properties.
Atomicity, a transaction refers to the indivisible operation, either succeed or fail, the case can not have half the success of failure of the half.
Consistency means that data until the transaction is completed the implementation of this point in time, the data must be read before the update, and then read the data must be updated, there should not be a moment, allowing users to read the update process The data.
Isolation refers to the execution of a transaction can not be other transactions interference. That is, data manipulation and the use of an internal affairs of other transactions in progress are isolated, between the various transactions to execute concurrently can not interfere with each other, this kind of like a copy of our online games to play, we played in the copy and strange out of the equipment, not associated with any other copies will not affect each other.
Persistence means that once a transaction is complete submission, subsequent failures of other operations and transactions will not have any impact on the results.
Most of the conventional single relational database to achieve the full ACID, however, for distributed systems, strict implementation of the four ACID properties is almost impossible, or that the cost of implementation is too large to us Can not accept.
Distributed transaction is to implement transactions in a distributed system. In distributed systems, to ensure the availability and performance without sacrificing serious premise, just to achieve the consistency of the data has been very difficult, so there are a lot of "residual blood version of" consistency, such as sequential consistency, The final consistency and so on.
Obviously achieve strict distributed transaction is even more impossible task. So, now we say distributed transactions, more often, is incomplete transactions in a distributed system implementation. In different application scenarios, different implementations, the goal is to solve practical problems through some compromises.
In practice, the more common Distributed transaction implementation has 2PC (Two-phase Commit, also known as two-phase commit protocol), TCC (Try-Confirm-Cancel) and transactional messages. Every implementation has its specific usage scenarios, also have their own problems, not the perfect solution.
Transaction message applies mainly those scenarios require an asynchronous update data, and the data is less demanding real-time scene. For example, the example we mentioned at the beginning, after you create an order, if brief appearance a few seconds, the shopping cart of goods not cleared in time, nor is it totally unacceptable, as long as the ultimate shopping cart of data and order data retention agreement on it.
2PC TCC is not our content and course of this discussion, do not start speaking up, interested students can learn on their own.
How to implement a message queue is distributed transaction?
Message Queuing provides transaction message requires the corresponding function can be achieved, Kafka and RocketMQ provides transaction-related functions.
Back orders and cart this example, we look at how together with message queues to implement a distributed transaction.
First, the order to open a transaction on the system message queue. Then the order system to the message server sends a "semi message", the semi message is not to say the message is incomplete, what it contains is complete message content, the only difference between semi message and common message is that before the transaction commits, for consumption are concerned, the news is not visible.
Semi message sent successfully, the order system can perform local transactions, and order records in order to create a library and database transaction orders submitted to the library. Then according to the results of local affairs decided to commit or roll back the transaction message. If the order is successfully created, then commit the transaction news, shopping cart system ready to spend this message to continue the follow-up process. If the order creation fails, then roll back the transaction news, shopping cart system will not receive this message. This basic realization of the "either succeed or fail," the conformance requirements.
If you are careful enough, you may have discovered, this implementation process, there is a problem is not resolved. If it fails to commit the transaction when the message is in the fourth step how to do? For this problem, Kafka and RocketMQ given two different solutions.
Kafka solution is relatively simple and crude, direct throw an exception, allowing users themselves. We can try again to submit the business code repeatedly, until the submission is successful, or orders created before deleting compensate. RocketMQ then given another solution.
RocketMQ Distributed transaction implementation
In the transaction implementation RocketMQ in, increasing the transaction pegging mechanism to solve the problem of failure to submit transaction message. If the Producer is an order system, occurs when the committed or rolled back transaction message network anomalies, RocketMQ the Broker does not receive a request to submit or rollback, Broker will go to pegging the transaction corresponding to the local affairs of the Producer status on a regular basis, then It decided to commit or roll back the transaction according to anti-check the results.
In order to support this transaction pegging mechanism, our business code needs to implement a reverse lookup interface to the local state of affairs, told RocketMQ local transaction success or failure.
In our case, pegging local transaction logic is very simple, as long as we are in order according to the message ID, the library in order to query the existence of this order can, if successful return order exists, otherwise fail. RocketMQ automatically commit or roll back the transaction message according to the results of the anti-affairs investigation.
The reverse lookup for local affairs, does not depend sender of the message, which is an instance of any data on the node order services. In this case, even if the transaction message is sent that order service node goes down, RocketMQ still reverse lookup can be performed by other nodes in the order of service, ensure the integrity of transactions.
And RocketMQ achieve the above general affairs speak integrated transaction message of reverse lookup mechanisms using RocketMQ transactional messaging features to achieve a distributed transaction processes as shown below:
summary
We use an example of shopping cart orders, learn the four ACID properties of a transaction, and how to use message queues to implement a distributed transaction.
Then we give a solution to several existing distributed transaction, including transaction messages, but these types of programs can not solve all the problems in a distributed system, each program has its limitations and specific application scenarios .
最后，我们一起学习了 RocketMQ 的事务反查机制，这种机制通过定期反查事务状态，来补偿提交事务消息可能出现的通信失败。在 Kafka 的事务功能中，并没有类似的反查机制，需要用户自行去解决这个问题。
但是，这不代表 RocketMQ 的事务功能比 Kafka 更好，只能说在我们这个例子的场景下，更适合使用 RocketMQ。Kafka 对于事务的定义、实现和适用场景，和 RocketMQ 有比较大的差异，后面的课程中，我们会专门讲到 Kafka 的事务的实现原理。