how to fix distributed transation
There are many concepts in distributed transactions and many solutions.
How to understand, how to use
Knowledge point
Hard transactions: stand-alone database transactions, distributed database transactions. Such as OceanBase, TiDB.
Flexible transactions: For hard transactions, a solution that avoids the difficulty and problems of hard transaction implementation can achieve final consistency. Such as XA multi-phase commit, Sags long transaction, local transaction table, external transaction table (reliable message).
Recently, in the process of corporate SOA, we encountered the problem of distributed transactions. There are many concepts involved in distributed transactions, such as ACID, CAP, and BASE.
Solution: XA multi-phase commit, Sags long transaction, local transaction table, external transaction table (reliable message).
However, it is difficult to get a definite answer on how to use it in the project.
OK, let me try to analyze
It's easier to explain with a cooking scene (I won't admit that I'm a foodie).
Suppose I want to cook the "distributed transaction" dish now, what if I don't do this dish, the normal idea is to look for recipes, and then we find recipes for pork, beef, chicken, and fish.
But due to the complexity of distributed transactions, the "distributed transaction" in this pot is a mixture of pork, beef, chicken, and fish. Although there is a recipe, do you know how to make this dish?
Writing this, I talk about my understanding of cooking. Encountering this situation is a test of a chef's understanding of the ingredients, the chef's experience, and then deduce the cooking method.
Now let's try to analyze the ingredient "distributed transaction"
understanding of food
The birth of distributed transactions
Let's start the derivation from the emergence of distributed.
Why there is a distributed appearance, because the hardware of a single machine causes the disk, CPU, and network IO of a single machine to be limited. In the reality that a single machine cannot expand infinitely, and the computing demand grows infinitely, a compromise has to be made.
The principle of the original stand-alone transaction is to write to the cache and then submit it to the disk after successful processing.
The emergence of distributed design has led to the isolation of data space, the original stand-alone
The transaction scheme of DB cannot be used, resulting in distributed transactions.
The evolution scheme of ACID in distributed
Distributed database: RDMA solution of shard disk, high technical difficulty
XA two-phase commit is implemented by blocking, which has a great impact on performance.
stand-alone
DB's transaction scheme cannot be used
to process transactions, and existing distributed ACID solutions are flawed and cannot be used.
At present, there is no better general solution for distributed transactions, and the industry proposes consistency requirements for different services and adopts different methods.
Cooking methods
Here try to do an analysis of common transaction processing schemes (Sags long transaction, local transaction table, external transaction table (reliable message)).
When I understand these solutions, I have the following questions
When are these programs used?
With so many solutions, is there a way to understand them and find something in common?
Because of the isolation of the data space, transactions under the distributed transaction design can only be processed by RPC.
In this case, there are only the following ways to deal with it
- negotiate and deal with each other
- mediator
- failed retry
at this time
Businesses requiring strong consistency: Sags, TCC, reliable message mode
Business requires eventual consistency: best effort delivery, external event table (write temporary data, driven by scheduled tasks).
Business requires weak consistency: message notification
Then design a solution according to the strength of the business requirements.
Usually the processing flow of a transaction is as follows
Data inconsistency occurs -> catch programmatic retry through catch, or give it to an external program to automatically retry -> service accepts request for reprocessing, or repair.
Data inconsistency occurs: an event with data inconsistency occurs, which can be referred to as an event source for short
Capture programmatic retry by catch, or give it to an external program to automatically retry: how actions are implemented
Service accepts request for reprocessing, or repair: repair data
Through the above analysis
Elements of Distributed Transactions Can Be Concluded
event: event source action: action fix: fix
level: Consistency requirement level
event
- To handle exceptions
- Throw an exception
- Discover data inconsistencies through active calculation
action:
- Programmatically: try-catch
- Timing: Scheduled tasks
- Delayed Actions: Delayed Tasks, Delayed Messages
- Message queue: mq message
fix:
- Retry
- rollback
level:
- powerful
- middle
- weak
Implementation process
:
The event source (event) is generated due to inconsistency, the action (action) processes the event source to repair (fix), and then selects the appropriate implementation scheme according to the level of business consistency requirements.
Idempotency and asynchrony are often mentioned in distributed transactions
Asynchronous
Asynchrony is the optimization of program processing logic. During the SOA process, many locally invoked methods become asynchronous.
In the distributed transaction solution,
an asynchronous method is adopted according to the requirements of business consistency.
idempotent
idempotent implementation
No idempotent required, no impact on other services
External transaction table: write a piece of data with a state identifier, and update the state by retrying
Implement queryable mode: rely on
unique ID + storage,
the specific logic includes direct query, pre-state judgment, and write primary key duplicate exception judgment.
Distributed locks: rely on
unique ID + storage. Implementation methods include zookeeper, redis, db, etc.
How to understand, how to use
I have asked a lot of children's shoes, and most of the solutions for distributed transactions answer retry + idempotency. why?
Retry + idempotency is one of the ways to deal with distributed transactions. It is easy to use and easy to understand. So most people answer the same.
During the evolution of the system, idempotency is a solution used when a method is extracted from multiple method calls to make a service, without considering the overall business requirements for consistency.
As long as you think about it, you will find that retry + idempotency cannot solve the problem used, and it is especially important to understand the principle behind it
Corresponding to my proposed EAF theory,
Idempotency is the complement of fix in EAF, and retry is the implementation of Action.
Chef's experience
Scenes:
Data synchronization: write temporary data, driven by timed tasks
Transfer money between two banks: reliable news
Paying bills: TCC
Case:
In the process of placing an order, the point service and coupon service are called. If an exception occurs, the rollback interface is called, and MQ, point service, and coupon service are sent to check after receiving the MQ.
The data synchronization scheme between the order production library and the order display library, write a temporary data, identify the status flag=0, driven by a scheduled task, the synchronization is successful, flag=1.
主生产流程中 生成多条任务数据，定时任务驱动 主流程写订单生产，分支写订单中心。
订单生产中间项目MQ写订单状态记录。关联系统订阅。
总结：
写在最后： 解决分布式事务重要的是 理解原理，不断积累项目经验，本人水平有限，写此文章，目的在于抛砖引玉，文中如有谬误，请不吝赐教。