Business, not just ACID | JD Logistics Technical Team

1. What is a transaction?

When the application is running, the database and hardware may fail, the network connection between the application and the database is disconnected, or multiple clients modify data concurrently, resulting in unexpected data coverage problems. In order to improve the reliability of the application and the consistency of the data , the transaction came into being.

Conceptually, a transaction is a form in which an application combines multiple read and write operations into a logical unit , so that all read and write operations are performed as a single operation, either successfully committed or failed rollback, There are no partial successes and partial failures. Today, almost all relational databases and some non-relational databases support transactions.

1.1 ACID

Transactions use ACID to ensure safe operations, which are Atomicity , Consistency , Isolation, and Durability . The proposal of ACID aims to establish a precise term for the database fault tolerance mechanism, but it is not the same in different databases, let's explain it one by one.

atomicity

The characteristic of the definition of atomicity is: a transaction must be regarded as an indivisible unit of work , all operations in the entire transaction are either all committed successfully, or all failed and rolled back, and it is impossible to execute only a part of them.

consistency

Consistency is a "redundant" existence in ACID. It is different from atomicity, isolation, and persistence. Consistency is an attribute of the application , while the other three are attributes of the database. Applications may rely on atomicity and isolation to ensure consistency, but sometimes the guarantee of consistency does not depend solely on the database.

The embodiment of consistency depends on the application's constraints on the data. For example, in an accounting system, the transaction revenue and expenditure of all accounts must be balanced. If a transaction starts in a balanced state, it will remain in balance after the transaction is executed and committed. Conceptually, consistency is a set of specific constraints on data that must always be established . This is guaranteed by the application, because the logic of these writes and modifications is determined by the application, and the database is only responsible for these operations. implement.

isolation

In actual work, most databases are accessed by multiple clients at the same time, which may cause the client to modify the same piece of data concurrently, causing concurrency problems . As shown in the example below:

User1 and User2 need to increase the operation counter in the database at the same time. Each user first reads the value, adds 1, and then writes. In theory, the value of the counter should end up being 44, but due to concurrency issues, the final value is 43.

Generally speaking, isolation makes the modification made by a transaction invisible to other transactions before it is finally committed, and it solves the concurrency problem. If one transaction does multiple writes, the other transaction either sees all of the writes or nothing. So in the above example, the value read by User2 when modifying the counter should be the result of 43 after modification by User1, and then add 1 to make the final result 44.

Persistence

Persistence is a promise that the transaction is successfully committed, and even if a hardware failure or database crash occurs, any data written will not be lost. In a single-node database, it usually means that the data has been written to the hard disk or SSD; In a node database, durability may mean that the data has been successfully replicated to some nodes. However, if the hard disk and backup are destroyed, then obviously there is no database that can retrieve the data, so perfect durability does not exist .

2. The problem of data inconsistency caused by concurrency

Often due to the competition between the operation objects between transactions and the uncertain timing relationship between concurrent transactions, various strange results will appear on these objects with competition. Let’s take a look below These common questions.

2.1 Dirty write

Two transactions try to update the same object in the database at the same time. If the previous write is part of an uncommitted transaction, the subsequent write will overwrite an uncommitted value. This situation is called dirty write .

2.2 Dirty reads

If A transaction has written some data into the database, but A transaction has not been committed or aborted, and now another B transaction query is started, then B transaction can see the uncommitted data in A transaction, which is a dirty read .

We consider a situation where once transaction A is rolled back, transaction B is likely to submit uncommitted data to the database, so the resulting problems will make it difficult for people to troubleshoot.

2.3 Non-repeatable read

Let's take an example where Alice has $1,000 in savings in the bank, divided into two accounts of $500 each. Now there is a transaction that transfers $100 from one of her accounts to another. If she checks her account balance in the middle of a transaction, she might see that the sending account has a balance of $400 after sending the transfer, while the receiving account still has a balance of $500. To Alice, it now appears that her account only has $900 in total, the $100 transferred seems to have disappeared out of thin air, and when the receiving account is read again, the balance becomes $600.

This situation is called non-repeatable read , also known as read deviation , that is, the results of two reads of the same data are inconsistent.

2.4 Lost updates

Two transactions perform a read-modify-write sequence at the same time, and one of the write operations directly overwrites the result of the other write operation without merging the changes of the other write operation, resulting in data loss. This situation is called Known as a lost update .

A more direct way to avoid lost updates is not to use the series of read-modify-write operations, but to perform atomic updates. Taking counters as an example, the SQL is as follows. Its principle is usually to obtain an exclusive lock on the object to be read, so that transactions are executed sequentially while modifying the same data.

update counters set value = value + 1 where key = 1;

If you cannot avoid the read-modify-write series of operations, you can avoid losing updates by explicitly locking (FOR UPDATE), so that any other transactions that want to read the same object are blocked until the first A transaction that acquires the lock is executed.

BEGIN TRANSACTION;

SELECT * FROM xxx FOR UPDATE;

-- 执行业务逻辑
UPDATE xxx SET ...;

COMMIT;

Compare and set (CAS) is a relatively common optimistic operation to avoid lost updates. When the data is updated, the value in the data table will be compared with the read value, and only if there is no change, the update is allowed, otherwise the transaction needs to be retried. Generally, CAS is realized by adding a timestamp column in the data table in the work .

2.5 Phantom read and write deviation problems

Phantom reading can be summed up in one sentence: the writing of one transaction changes the search query results of another transaction .

Transaction A queries qualified data, checks whether it meets the business requirements, and decides whether to continue the business based on the check results. If transaction B modifies the data at this time and meets the query conditions of transaction A, then after transaction A completes the write operation, it will find different results if it executes the query again, which is likely to cause write bias . select select select

A write skew problem is when two transactions read the same objects and then update some of them with unexpected exceptions. It is distinguished from dirty writes and lost updates because two transactions are updating two different objects. As shown in the example below, Alice and Bob are the two doctors on duty, and both feel unwell, so they both decide to take time off. Unfortunately, they hit the button off work at exactly the same time:

phantom reading example.png

This resulted in no physicians on duty, violating the operational requirement of having at least one physician on duty.

It is troublesome to solve the problem of writing deviation, because it involves multiple objects, and the method of single-object atomic operation cannot be solved. Usually, it is solved by changing the isolation level to serializable or by locking.

However, the locking method is not applicable in all cases. For example, if multiple people book a conference room at the same time period, because the conference room reservation record for this time period has not been generated yet, when multiple people read the reservation record, they do not read the corresponding result value, so there is no way to lock it. It will result in the result that multiple people book the meeting room at the same time. In order to solve this situation, you can create another data table to manage the time period of the meeting room. When someone wants to reserve a meeting room for a certain period of time, the data of this period will be locked, and then when other users come to query , will be blocked, this method is called materialization conflict .

3. Isolation level

Databases have always tried to solve concurrency problems through transaction isolation . Serializable isolation level guarantees that transactions are executed serially, which means that concurrency issues cannot occur. However, in actual production, in order to ensure the performance of the system, this isolation level is often not used, but some weaker isolation levels are used. They may not guarantee data consistency in some cases, but they can make the performance of the system better. Below we introduce these isolation levels:

3.1 Read uncommitted

This isolation level is relatively weaker and can only avoid dirty writes .

3.2 Read Committed

This isolation level is very popular because it avoids dirty reads and dirty writes .

The most common case is to use row locks to prevent dirty writes: when a transaction wants to modify the same object, it must wait until the first transaction commits or rolls back to acquire the lock for the row to continue.

Dirty reads can also be avoided by adding read locks, but this method will cause read requests to be blocked when there are long-term write transactions holding locks to read data, so this method is not effective in practice good. Another way to avoid it is that the database remembers the old value that has been written. Even if a new write transaction occurs and is not completed, the read request reads the old value only when the write transaction is committed. to read the new value.

3.3 Repeatable read

Repeatable reads can avoid dirty writes, dirty reads, non-repeatable reads, and phantom reads in read-only queries. Snapshot isolation is a common solution for repeatable reads. Each transaction reads from a consistent snapshot of the database, which means that the transaction sees all the data that was committed in the database when the transaction started. Even if the data is subsequently changed by a new transaction, the transaction still reads the old data at the beginning of the transaction. This approach is very useful for long-running read-only queries, because if the data is constantly changing during the query, there is no way to analyze the data.

Read Committed without snapshot isolation cannot achieve repeatable read because it only remembers two versions of the data .

Snapshot isolation also uses write locks to avoid dirty writes, and the way to avoid dirty reads does not need to be locked, but by reading the corresponding version of the data object maintained in the database. Its key principle is that reading does not block writing, and writing does not blocking read . This means that while the database is processing long queries on consistent snapshots, it can process writes concurrently without lock contention.

The implementation of snapshot isolation by MySQL using the InnoDB engine is MVCC multi-version concurrency control , which maintains multiple versions of a single object at the same time to provide multiple data states at different time nodes. Let's take a brief look at its implementation below. principle.

For InnoDB engine tables, its clustered index records contain two necessary hidden columns: trx_id and roll_pointer

trx_id: Every time a transaction modifies a clustered index record, it assigns the transaction ID of the transaction to trx_id
roll_pointer: Every time a clustered index record is modified, the old version will be written into the undo log. This hidden column is equivalent to a pointer, and the information before the modification of the record can be found through it. Let's take an example to understand it, assuming that the table hero contains only one record:

mysql> select * from hero;
+-----+-----+
|id   |name |
+-----+-----+
|1    |刘备 |
+-----+-----+

Specify the transaction ID for inserting this record as 80. At this time, open two transactions to modify this record. Each modification will generate an undo log, and each log also has trx_id attribute and roll_pointer attribute. Multiple undo logs can be connected into a linked list through the roll_pointer attribute, as shown in the following figure:

This linked list is called a version chain. The head node of the version chain is the current latest record. The version chain of this record can be used to control the behavior of concurrent transactions when accessing the same record. This method is called multi-version concurrency control .

When the transaction executes the first query, it will generate a consistent snapshot (Read View) , which can be used to determine which version in the version chain is visible to the current transaction. Read View contains 4 more important contents as follows:

m_ids: When generating Read View, the id list of active read and write transactions in the current system, which is used to ensure that even if these active transactions are committed, their writes will be ignored by the current transaction
min_trx_id: the minimum transaction id of active read and write transactions in the current system
max_trx_id: The transaction id that the system should assign to the next transaction
creator_id: the transaction id of the transaction that generated the Read View

With Read View, you only need to follow the steps below to determine whether a version in the record is visible:

If the trx_id attribute value of the accessed version is the same as the creator_trx_id value in the creator in Read View, it means that the current transaction is accessing the content modified by itself, and this version can be accessed by the current transaction
If the trx_id attribute value of the accessed version is less than the min_trx_id value in Read View, it indicates that the transaction that generated this version has been committed before the current transaction generates Read View, and these versions can be accessed by the current transaction
If the trx_id attribute of the accessed version is greater than or equal to the max_trx_id value in Read View, it means that the transaction that generates this version is opened after the current transaction generates Read View, then this version cannot be accessed by the current transaction
If the value of the trx_id attribute of the accessed version is between the min_trx_id and max_trx_id of the Read View, it is necessary to determine whether the trx_id is in the m_ids list. If it is, it means that the transaction that generated this version was still active when the Read View was created, so this version is not visible; if it is not, it means that the transaction that generated this version when the Read View was created has been submitted, and this version can be accessed

That is to say, if the record is to be visible to the current read transaction, the transaction that needs to create the record has been committed before the current read transaction starts .

3.4 Serializable

Serializability is generally considered the strongest isolation level , able to avoid all data inconsistencies we appeal. It can guarantee that even though transactions can be executed in parallel, the final result is the same, as if they were executed one after the other without any concurrency. That is, the database prevents all possible race conditions.

Most serializable implementation technologies adopt one of the following three methods: serialized execution of transactions , two-phase locking or serializable snapshot isolation .

Serialized execution of transactions

The implementation of this technology must require each transaction to be small and fast . If there is a slow transaction among them, it will naturally slow down other transactions. In addition, this approach is limited to cases where the active data set fits in memory , and the system will also become very slow if the data needs to be accessed from disk within a transaction. Write throughput must be low enough to be processed on a single CPU core, and if not, transactions need to be partitioned into a single partition without coordination across partitions. Of course, cross-partition transactions can be implemented, but its execution efficiency will be very low. Therefore, the scalability of serialized execution transactions is poor.

Two-stage locking (2PL, two pahse locking)

The meaning of two-phase commit is: the lock (shared lock/exclusive lock) is acquired when the transaction is executed in the first phase, and the lock is released when the transaction execution is completed in the second phase. It requires that multiple transactions can read the same object when there is no write, but as long as there is write, it will have exclusive access , read blocks write, and write also blocks read , which is different from snapshot isolation, so two-stage locking can avoid competition conditions To achieve serialization.

Mysql's InnoDB engine implements the serializable isolation level using the 2PL mechanism.

The performance of two-phase locking is poor, not only because of the overhead of acquiring and releasing locks, but also because of the reduction in concurrency, because if two transactions modify the same object, the second transaction must wait for the first transaction to execute until the end. In addition, the serializable isolation implemented by 2PL also has frequent deadlocks.

Serializable snapshot isolation (SSI, serializable snapshot isolation)

Serializable snapshot isolation is an optimistic concurrency control technique that builds on snapshot isolation by adding an algorithm to detect serialization conflicts between writes and determine which transactions to terminate.

Optimism means not blocking a transaction if there is a potential danger, but continuing to execute it in the hope that everything will be fine. When a transaction wants to commit, the database checks to see if something bad happened (i.e. whether isolation was violated), and if so, the transaction is aborted and has to be retried. Optimistic concurrency control tends to perform better than pessimistic concurrency control when contention is not high.

The transaction reads some data from the database and makes conditional judgments based on the data. When executing business logic, under the conditions of snapshot isolation , the previous query results are often not up-to-date, because the data may be modified after the data query. Therefore, the executed business logic may be abnormal. Therefore, judging whether the previously read data has changed when the transaction is committed requires two verifications:

Check for uncommitted writes before reads
Check for writes after reads

Only after passing these checks can it be guaranteed that the data used when the transaction is committed is new.

Compared with serial execution, serializable snapshot isolation is not limited to the throughput of a single CPU core, so its scalability is better; serializable snapshot isolation is compared with two-phase locking , its biggest advantage is that a transaction does not need to block and wait for the lock held by another transaction, just like under snapshot isolation, reading does not block writing, and writing does not block reading, which is very suitable for business scenarios with more reading friendly.

The performance of serializable snapshot isolation is reflected in the abort rate. If there are many long-term read and write transactions, conflicts are likely to occur frequently and cause transaction aborts. Therefore, in the case of relatively short transactions, serializable snapshot isolation performs better.

4. Timeliness and completeness

ACID transactions usually guarantee strong consistency , that is, writers wait until the transaction commits, and after the write is complete, the result of the write is visible to all readers. In the semantics of strong consistency, there are two aspects that are particularly worth considering:

Timeliness : This means ensuring that users observe the latest state of the system. If it is not strong consistency but eventual consistency, then users may read stale data, but this inconsistency is temporary and will eventually be resolved by waiting and simply retrying
Integrity : Integrity means that there is no loss, inconsistency or error in the data, i.e. no corruption. In particular, certain derived datasets (caches, search indexes, etc.) must be kept consistent with the underlying database. In ACID transactions, atomicity and durability are important principles to ensure integrity

What's interesting is: based on the distributed transaction implemented by the asynchronous stream processing system, it can separate the timeliness from the integrity, and only guarantee the integrity, but not the timeliness, unless we explicitly construct a transaction that explicitly waits before the transaction commits and returns the result. The consumer to which a particular message arrives.

Let's look at an example of implementing distributed transactions based on a stream processing system to deepen our understanding of timeliness and integrity.

4.1 Using log-based message queues to ensure integrity

Let's take the transfer as an example, for example, there are three partitions: one contains the request ID, one contains the payee account, and the other contains the payer account. If, in the traditional approach to databases, executing this transaction would require an atomic commit across three partitions, then the distributed transaction would need to be coordinated, so throughput would likely suffer. But in fact, a stream processing system implemented using a log-based message queue can achieve equivalent data integrity without the need for atomic commit . The example execution process is as follows:

The client provides a unique request ID for the transfer request from account A to account B, and appends it to the corresponding message queue according to the request ID, and persists the message
Consumers read request logs. For each request message, it sends two messages to the output stream: the payer's debit instruction (A partition), the payee's credit instruction (B partition), and the sent message will carry the original request ID
Subsequent consumer consumption debit and credit instructions, deduplication according to ID, and apply the changes to the balance of the account

In order to ensure data integrity between multiple partitions and avoid coordination of distributed transactions (protocols such as 2PC), we first need to persist the things to be done by this transaction into a single record, and then derive the credit from this message record debit and debit instructions. In almost all data systems, single-object writes are atomic : the request either appears in the log or it doesn't.

If stream processing crashes at step 2, it resumes processing from the last archive point so it doesn't skip any messages, but may generate multiple duplicate debit/credit instructions, but since it is deterministic , so all it generates is the same instruction, the processor in step 3 can easily de-duplicate by the ID value.

In the above example, we split an operation into stream processors spanning multiple stages. The consumption of message records is asynchronous . The sender will not wait for the message to be consumed and processed, and the message and the processing result of the message are processed by Decoupling , so we did not guarantee timeliness, but integrity.

Generally, we can guarantee integrity without coordinating distributed transactions or adopting other atomic commit protocols when using a reliable stream processing system. The mechanisms included are as follows:

Represents the content of the write operation as a single message, which guarantees the atomicity of the write
Derive other required state changes from this message
Pass the request ID generated by the client through all processing layers, so as to achieve the purpose of deduplication and guarantee idempotency
Guarantees that messages are immutable and allows derived data to be reprocessed at any time, which makes recovering from errors easier

4.2 The Importance of Integrity

Whether it is an ACID transaction or a distributed transaction based on a stream processing system, they all guarantee data integrity. Because a violation of timeliness can be confusing, but only temporarily, but a violation of integrity can be catastrophic. Violation of consistency, eventual consistency; violation of integrity, never consistency , is the best generalization.

shoulders of giants

"Data-Intensive Application System Design": Chapter 7 Transactions, Chapter 12 Future of Data Systems
Replication (Part 2): Transactions, Consistency and Consensus
"How MySQL Works" Chapter 21
"High Performance MySQL Fourth Edition" Chapter 1

Author: JD Logistics Wang Yilong

Source: JD Cloud Developer Community