Spring--implementation method and alternative of microservice distributed transaction

Microservices-distributed transaction implementation methods and alternatives

In the past two days, I am working on a solution for distributed transactions in the microservice architecture, and make a small summary as a memo. If there are errors, please correct me!

Conceptual clarification

• Transaction compensation mechanism : In any forward transaction operation in the transaction chain, there must be a reversible transaction that fully conforms to the rollback rule.
• CAP theory : CAP (Consistency, Availability, Partition Tolerance), which elaborates the three main aspects of a distributed system, which can only be achieved by choosing two at the same time. Common CP systems, AP systems.
• Idempotency : Simply put, business operations support retries without adverse effects. Common implementation: add a unique ID to the message.
• BASE (Basically avaliable, soft state, eventually consistent): is a theoretical standard for distributed transaction implementation.

Flexible transactions vs. rigid transactions

Rigid transactions refer to transactions that strictly follow the ACID principle, such as database transactions in a stand-alone environment.

Flexible transactions refer to transactions that follow the BASE theory, and are usually used in a distributed environment.The common implementation methods are: two-phase commit (2PC), TCC compensated commit, message-based asynchronous guarantee, and best effort notification.

Generally, rigid transactions are used for local transactions, and flexible transactions are used for distributed transactions.

First come to the conclusion, and then introduce various implementation methods of distributed transactions.

• If business scenarios require strong consistency, try to avoid placing them in different services, that is, use local transactions as much as possible, and avoid using distributed transactions with strong consistency.
• If the business scenario can accept the final consistency, then it is best to use a message-based final consistency solution (asynchronous guarantee type) to solve.
• If business scenarios require strong consistency and can only deploy distributed services , it is best to use the TCC solution instead of the 2PC solution.

Note: Each of the following solutions has different applicable occasions, and needs to be selected according to the actual business scenario.

1. Best Practice

    Try to avoid putting a transaction in different services, that is, try to use local transactions, avoid distributed transactions with strong consistency, each microservice theoretically only returns results, which is the best method. Of course, the transactions in different microservices cannot be avoided, please use the following method.

Two, two-phase submission (2PC)

Two Phase Commit (2PC), with strong consistency, is a typical implementation of CP system.

Two-phase submission, the common standards are XA, JTA, etc. For example, Oracle database supports XA.

The following figure is a schematic diagram of the two-phase submission:

The first half of the figure is a demonstration of two-phase submission success, and the second half is a demonstration of two-phase submission failure. There are many classic explanations about the two-phase submission online.

Disadvantages

• In the second phase of the two-phase commit, the coordinator needs to wait for all participants to issue a yes request, or a participant to issue a no request before performing the commit or interrupt operation. This will cause multiple resources to be locked at the same time for a long time, resulting in performance The bottleneck , if the participant has a long operation, the performance loss will be more obvious.
• The implementation is complicated, which is not conducive to the expansion of the system, and it is not recommended.

3. Compensatory TCC (Try-Confirm-Cancle) type

TCC is an implementation of an AP system based on compensatory transactions and has ultimate consistency.

The following uses the payment operation when the customer purchases the goods as an example to explain:

• Try:
  Complete all business checks (consistency), and reserve the necessary business resources (quasi-isolation);
  in this example, it is to confirm that the customer account balance is sufficient to pay (consistency), lock the customer account, and the merchant account (quasi Isolation).
• Confirm:
  Use the business resources reserved in the Try stage to execute the business (the business operation must be idempotent). If the execution is abnormal, a retry is required.
  Here, the customer account deduction and the merchant account debit operation are performed.
• Cancle:
  release the business resources reserved in the Try phase, here is to release the lock of the customer account and the merchant account;
  if any of the sub-businesses cannot perform the operation successfully in the Confirm phase, it will cause the response to the business activity manager to time out, at this time To perform compensatory transactions for other businesses. If the compensatory operation is also abnormal, it must be retried. If it is impossible to execute successfully, the transaction manager must be able to sense the failed operation and perform log (for post-manual compensation (Transaction operations or middleware take over and perform compensatory transaction operations later).

advantage

Compared with the two-stage submission method mentioned above, there are two major advantages:

• TCC can separately lock, commit and release each resource in a distributed transaction . For example, suppose there are two operations AB, and if A takes a short time, then A can complete its own try-confirm-cancel faster Process, release resources. No need to wait for operation B. If problems occur afterwards, additional compensatory transactions can be performed.
• TCC is bound to each sub-business (except for the global rollback operation in cancle), that is, each service can be executed "asynchronously in parallel" to some extent.

Precautions

• The transaction manager (coordinator) node must be deployed in a highly available cluster (HAC) with synchronous replication semantics.
• The transaction manager (coordinator) also needs to use majority algorithm to avoid the split brain problem of the cluster.

Applicable scene

• Strict consistency
• Short execution time
• High real-time requirements

Examples: red envelopes, collection and payment business.

Fourth, asynchronous guarantee type (message transaction + final consistency)

By changing a series of synchronous transaction operations to asynchronous operations based on messages , the impact of synchronous blocking operations in distributed transactions is avoided.

This solution truly realizes the decoupling of the two services.The key to the decoupling is asynchronous messaging and compensating transactions.

Here is an example as an explanation:

The steps are as follows:

1. The MQ sender sends remote transaction messages to the MQ Server;
2. The MQ Server responds, indicating that the transaction message has successfully reached the MQ Server.
3. MQ sender Commit local transaction.
4. If the local transaction Commit succeeds, the MQ Server is notified to allow the corresponding transaction message to be consumed; if the local transaction fails, the MQ Server is notified that the corresponding transaction message should be discarded.
5. If the MQ sender does not provide feedback to the MQ Server on the execution status of the local transaction, the MQ Servfer needs to actively check the transaction status to the MQ sender to determine whether the transaction message can be consumed.
6. When it is known that the local transaction is successfully executed, MQ Server allows MQ subscribers to consume this transaction message.

One additional point is that after the transaction message is delivered to the MQ subscriber, it may not be successfully executed. The MQ subscriber needs to actively give consumer feedback (ack)

• If the MQ subscriber successfully executes the remote transaction, the successful ack is given to the consumer, then the MQ Server can safely remove the transaction message;
• If the execution fails, MQ Server needs to re-deliver the message until the consumption is successful.

Precautions

• Message middleware plays an important role in the system, and all transaction messages need to be communicated through it, so the message middleware also needs to support HAC to ensure that transaction messages are not lost.
• Depending on the specific implementation of the business logic, you may also need to add other requirements to the message middleware such as message non-repetition and no disorder.

Applicable scene

• Long execution cycle
• Real-time requirements are not high

E.g:

• Interbank transfer / remittance business (the two services are in different banks)
• Return / refund business
• Finance, billing statistics business (first sent to the message middleware, then batch accounting)

V. Best effort notification type

This is one of the lowest requirements in distributed transactions, and can also be implemented through message middleware. The difference from the previous asynchronous guarantee operation is that after the message is delivered by the MQ Server to the consumer, after the maximum number of retries is allowed End the transaction normally .

Applicable scene

Notification of transaction result messages, etc.

summary

Whether it is the transaction manager (coordinator) in synchronous transactions or the message middleware used in asynchronous transactions, to achieve consistency guarantees, you need to use the highly available and highly reliable features provided by HAC with synchronous replication semantics. These are all at the cost of performance and undoubtedly become one of the typical performance bottlenecks in SOA architecture.