Author: Wonder , senior development engineer Ali cloud database

The role of database concurrency control

1.1 The concept of affairs

Before introducing concurrency control, you first need to understand transactions. The database provides several basic operations such as addition, deletion, modification and checking. Users can flexibly combine these operations to achieve complex semantics. In many scenarios, users want a group of operations to take effect together as a whole. This is a transaction. A transaction is the basic unit of database state changes and contains one or more operations (such as multiple SQL statements).

The classic transfer transaction includes three operations: (1) Check whether the balance of A account is sufficient. (2) If enough, deduct 100 yuan from A. (3) Add 100 yuan to the B account.

The transaction has a basic characteristic: this group of operations will either take effect together or not at all . If an error is encountered during the execution of the transaction, all operations that have been performed must be withdrawn. This is the atomicity of the transaction .

If after the failure, partially effective transactions cannot be withdrawn, then the database enters an inconsistent state, which is contrary to the real world facts. For example, the transfer transaction fails after deducting 100 yuan from the A account, and the B account has not yet increased the amount. If the deduction operation of the A account has not been withdrawn, the world will inexplicably lose 100 yuan. Atomicity can be achieved by logging (the value before the change), and some databases cache transaction operations locally, and in case of failure, directly discard the operations in the cache.

As long as the transaction is committed, its results cannot be changed. Even if the system is down, the state of the database after restart is the same as before the downtime. This is the durability of the transaction .

As long as data is stored in non-volatile storage media, downtime will not cause data loss. Therefore, the database can use the following methods to ensure durability: (1) Before the transaction is completed, all changes are guaranteed to be stored on disk. Or (2) Before the commit is completed, the change information of the transaction is stored on the disk in the form of a log, and the memory state of the database system is restored according to the log during the restart process. Generally speaking, the database will choose method (2), the reason is left to the reader to think.

In order to improve resource utilization and transaction execution efficiency and reduce response time, the database allows transactions to be executed concurrently. However, if multiple transactions operate on the same object at the same time, there will inevitably be conflicts. The intermediate state of the transaction may be exposed to other transactions, causing some transactions to write the wrong value to the database based on the intermediate state of other transactions. There is a need to provide a mechanism to ensure that the execution of transactions is not affected by concurrent transactions, so that users feel that currently only transactions initiated by themselves are being executed, which is isolation .

Isolation allows users to focus on the logic of a single transaction, regardless of the impact of concurrent execution. The database guarantees isolation through a concurrency control mechanism. Because isolation requires a higher order of transaction execution, many databases offer different options, and users can sacrifice some isolation to improve system performance. These different options are transaction isolation levels .

The database reflects the real world. There are many restrictions in the real world. For example: no matter how the account is transferred, the total amount will not change and other practical constraints; age cannot be negative, and the gender can only have three types: male, female and transgender Integrity constraints such as options. Transaction execution cannot break these constraints and ensure that the transaction is transferred from one correct state to another correct state. This is consistency .

The difference and the first three properties are completely guaranteed by the database implementation. Consistency depends not only on the database implementation (atomicity, durability, isolation is also to ensure consistency), but also on the transaction logic written on the application side.

1.2 How to ensure transaction isolation

In order to ensure isolation, one way is to execute all transactions serially so that transactions do not interfere with each other. However, serial execution efficiency is very low. In order to increase throughput and reduce response time, the database usually allows multiple transactions to be executed simultaneously. Therefore, the concurrency control module needs to ensure that the effect of concurrent execution of transactions is exactly the same as the effect of serial execution of transactions (serializability) in order to meet the requirements of isolation.

In order to facilitate the description of how concurrency control guarantees isolation, we simplify the transaction model. A transaction is composed of one or more operations, and all operations can eventually be split into a series of reads and writes. A batch of simultaneous transactions, all read and write execution order, is defined as a schedule, for example:

T1 and T2 are executed at the same time, a possible schedule:

T1.read(A),T2.read(B),T1.write(A),T1.read(B),T2.write(A)

If the schedule effect of concurrent transaction execution is equivalent to the serial execution (serial schedule), serializability can be satisfied. A schedule continuously changes the order of read and write operations, and it will always become a serializable schedule, but some exchanges may result in different results of transaction execution. In a schedule, the position of two adjacent operations change the result of the transaction, then these two operations are in conflict . The conflict needs to meet the following conditions at the same time:

These two operations come from different transactions
At least one is a write operation
Operate the same

Therefore, common conflicts include:

Read and write conflicts. Transaction A first reads a row of data, transaction B changes the row of data, and transaction B first modifies a row of transactions and transaction A reads the row record. Transaction A reads different results. This conflict may lead to non-repeatable read vision and dirty read vision.
Write and read conflicts. The same reason as the read-write conflict. This conflict may lead to dirty reading vision.
Write conflict. Two operations write an object in sequence, and the result of the latter operation determines the final result of the write. This conflict may cause updates to lose vision.

As long as the database guarantees that the schedule of concurrent transactions keeps the execution order of conflicting operations unchanged, only swapping non-conflicting operations can become a serial schedule, and they can be considered equivalent.

This method of equivalence judgment is called conflict equivalent: the conflicting sequence of the two schedules is the same . For example, in the example below, T1 write (A) conflicts with T3 read (A), and T1 occurs before T3. T1 read (B) and T2 write (B) conflict, and T2 precedes T1, so the schedule executed by the transaction on the left is equivalent to the serial schedule (right) executed by T2, T1, and T3. The execution sequence in the left figure satisfies conflict serializablity.

.png")

Let us analyze a counterexample: T1 read (A) conflicts with T2 write (A) and T1 precedes T2, and T2 write (A) conflicts with T2 write (A) and T2 precedes T1. The following schedule cannot be equivalent to any serial schedule. It is an execution sequence that does not meet the conflict serializablity and will cause the update to be lost.

.png")

In general, serializability is a relatively strict requirement. In order to improve the concurrent performance of database systems, many users are willing to reduce the isolation requirements to seek better performance. Database systems often implement multiple isolation levels for users to choose flexibly. For the transaction isolation level, please refer to this article .

The requirements of concurrency control are clear, how to achieve it? The following article will introduce the common implementation methods of concurrency control one by one according to the optimism of conflict detection.

Concurrency control based on two-stage lock

2.1
Since 2PL has to ensure that operations are performed in the correct order, the easiest way to think of is to lock and protect the access object. The lock manager module of the database system is specifically responsible for locking and releasing locks to access objects, ensuring that only the transaction holding the lock can operate the corresponding object . Locks can be divided into two categories: S-Lock and X-Lock. S-Lock is a shared lock used for read requests, and X-Lock is an exclusive lock used for write requests. Their compatibility is as follows: operating on the same object, only two read requests are compatible with each other and can be executed at the same time, and read and write operations will be executed serially due to lock conflicts. 2PL (Two-phase locking) is the most common lock-based concurrency control protocol for databases. As the name implies, it contains two phases:

Phase one: Growing, the transaction requests all the locks it needs from the lock manager (there may be a failure to lock).
Phase 2: Shrinking, the transaction releases the lock acquired in the Growing phase, and no new locks are allowed.

Why should locking and releasing locks be clearly divided into two stages? The purpose of 2PL concurrency control is to achieve serializable. If concurrency control does not apply all the required locks in advance, but after releasing the lock, it is allowed to apply for the lock again. There may be two operations within the transaction between the same object, other transactions modify this An object (as shown in the figure below), and then unable to reach conflict serializable, there is inconsistency (the following example is lost update). 2PL can guarantee conflict serializability, because the transaction must get all the required locks to execute. For example , the transaction A that is being executed conflicts with transaction B, and transaction B is either already executed or is still waiting. Therefore, the execution order of those conflicting operations is the same as the execution order of the conflicting operations when BA or AB is executed serially. So, as long as the database uses 2PL to ensure consistency and isolation? Let's take a look at this example: the above execution order is in line with 2PL, but T2 reads uncommitted data. If T1 is rolled back at this time, it will cause a cascade rollback (T1 changes cannot be seen by any transaction). Therefore, the database often uses an enhanced version of S (trong) S (trict) 2PL, which is a little different from 2PL: in the shrinking phase, the lock can only be released after the transaction is completed, completely eliminating the uncommitted data of the transaction Was read.

2.2 Deadlock handling

Locking and releasing locks for concurrent transactions will inevitably circumvent a problem-deadlock: transaction 1 holds B locks such as A lock, and transaction 2 holds A locks such as B lock. There are currently two solutions to the deadlock problem:

Deadlock Detection：

The database system records the waiting relationship of the transaction according to the waits-for graph, where the dot represents the transaction and the directed edge represents the transaction is waiting for another transaction to release the lock. When a ring appears in the waits-for graph, it means that a deadlock has occurred. The background of the system will periodically detect the waits-for graph. If a ring is found, a proper transaction abort needs to be selected.

Deadlock Prevention：

When a transaction requests a lock that is already held, the database system kills one of the transactions to prevent deadlocks (generally the longer the transaction lasts, the higher the reservation priority). This precautionary approach does not require waits-for graphs, but increases the rate at which transactions are killed.

2.3 Intention lock

If there are only row locks, then if a transaction needs to update 100 million records, it needs to acquire 100 million row locks, which will consume a lot of memory resources. We know that locks are used to protect the internal access objects of the database. These objects may be: Attribute, Tuple, Page, Table. The corresponding locks can be divided into row locks and pages Locks, table locks (no one implements attribute locks. For OLTP databases, the smallest unit of operation is a row). For transactions, it is of course the best to obtain the smallest number of locks, such as updating 100 million records, perhaps adding a table lock is sufficient. Higher-level locks (such as table locks) can effectively reduce the use of resources and significantly reduce the number of lock checks, but will severely limit concurrency. The lower-level locks (such as row locks) are conducive to concurrent execution, but in the case of many transaction request objects, a large number of lock checks are required. In order to solve the problem of high-level lock limiting concurrency, the database system introduces the concept of Intention lock:

Intention-Shared (IS): indicates that one or more objects in it are protected by S-Lock. For example, if a table is added with IS, at least one row in the table is protected by S-Lock.
Intention-Exclusive (IX): indicates that one or more objects in it are protected by X-Lock. For example, a table plus IX, at least one row in the table is protected by X-Lock.
Shared + Intention-Exclusive (SIX): indicates that at least one object inside is protected by X-Lock, and itself is protected by S-Lock. For example, an operation requires a full table scan, and changes a few rows in the table, you can add SIX to the table. Readers can think about why there is no XIX or XIS

The compatibility relationship between intent locks and ordinary locks is as follows: The advantage of intent locks is that when IX is added to the table, it means that there are rows in the table that are being modified. (1) At this time, the DDL operation is initiated on the table, and the X lock of the table needs to be requested. Then, when the table holds IX, it waits directly without checking whether the rows in the table hold row locks one by one, effectively reducing the inspection overhead . (2) At this time, there are other read and write transactions. Since the table is added IX instead of X, it will not prevent the read and write requests for the row (first add IX to the table, and then add S / X to the record) If the transaction does not involve the X-locked row, it can be executed normally, increasing the concurrency of the system. 03

Concurrency control based on Timing Order (T / O)

Assign a timestamp to each transaction, and use this to determine the transaction execution order. When the timestamp of transaction 1 is less than transaction 2, the database system should ensure that transaction 1 is executed before transaction 2. Timestamp distribution methods include: (1) physical clock; (2) logical clock; (3) mixed clock.

3.1 Basic T/O

Based on T / O concurrency control, no lock is required for reading and writing, and each row of record marks the timestamp of the transaction that last modified and read it . When the timestamp of the transaction is less than the recorded timestamp (cannot read the "future" data), it needs to be aborted and re-executed. Suppose that record X is marked with two read and write timestamps: WTS (X) and RTS (X), the transaction timestamp is TTS, and the visibility judgment is as follows: read:

TTS <WTS (X): The object is not visible to the transaction, abort transaction, take a new timestamp and start again.
TTS> WTS (X): This object is visible to the transaction, update RTS (X) = max (TTS, RTS (X)). In order to satisfy repeatable read, the value of X is copied by the transaction.
In order to prevent dirty data from being read, special marks can be made on the record. The read request needs to wait for the transaction to be submitted before reading.

write:

TTS <WTS (X) || TTS <RTS (X): abort transaction, restart.
TTS> WTS (X) && TTS> RTS (X): Transaction update X, WTS (X) = TTS.

The reason why TTS> RTS (X) is required here is to prevent the following situation: the timestamp of the read request is rts, X has been read, and the timestamp is set to RTS (X) = rts. ), And the update is successful, the rts read request is read again to see the new changes, which violates repeatable read, so this is to avoid read and write conflicts. The last read and write time is stored in the record, which can ensure that conflict serializable can also avoid write skew. For example: the initial state, X and Y records, X = -3, Y = 5, X + Y> 0, RTS (X) = RTS (Y) = WTS (X) = WTS (Y) = 0. The time stamp of transaction T1 is TTS1 = 1, and the time stamp of transaction T2 is TTS2 = 2. .png ") Its defects include:

Long transactions are prone to starvation, because the timestamp of long transactions is too small, and there is a high probability that the updated data will be read after a period of execution, resulting in abort.
Read operations also generate writes (write RTS).

Concurrency control based on Validation (OCC)

During the execution process, each transaction maintains its own write operation (Basic T / O writes data to the DB during transaction execution) and the corresponding RTS / WTS, and judges whether its own changes and existing ones in the database are submitted. Data conflicts, write to DB if there is no conflict. OCC is divided into three stages:

Read & Write Phase: the read and write phase, the transaction maintains the read results and upcoming changes, as well as the RTS and WTS written records.
Validation Phase: Check whether the transaction conflicts with the data in the database.
Write Phase: Write without conflict, conflict, abort, restart.

The Read & Write Phase is over, and entering the Validation Phase is equivalent to completing the transaction preparation and entering the commit phase. The time to enter the Validation Phase is selected as the timestamp of the record line to sequence. The reason why the transaction start time is not used is that: the transaction execution time may be longer, and the transaction that starts later may be submitted first, which will increase the probability of transaction conflict. Transactions with smaller time stamps will be written to the database after the transaction, and will definitely abort.

Validation process

Assuming that there are only two transactions T1 and T2, and the same data row is modified, the timestamp of T1 <the timestamp of T2 (that is, the order of validation: T1 <T2, for users, T1 occurs first in T2), there are as follows Situation: (1) T1 is in the validate phase, and T2 is still in the Read & Write Phase. At this time, as long as there is no conflict between the reading and writing that has occurred in T1 and T2, you can submit.

If WS (T1) ∩ (RS (T2) ∪ WS (T2)) = ∅, it means that there is no conflict between the records written by T2 and T1, and the verification can be written.
Otherwise, there is a read-write conflict or a write-write conflict between T2 and T1, and T1 needs to be rolled back. Read and write conflicts: T2 reads the version before T1 was written. After T1 commits, it may read the version written by T1, which cannot be read repeatedly. Write and write conflicts: T2 may be updated based on the old version and written again, causing the update of T1 to be lost.

(2) T1 completes the validate phase and enters the write phase until the submission is completed, which is already irreversible. T2's reading and writing before T1 enters the write phase must not conflict with T1's operation (because T1 validation passed). The read and write operations continued after T2 may conflict with the operation to be submitted by T1, so T2 enters the validate stage:

If WS (T1) ∩ RS (T2) = ∅, it means that T2 has not read the record written by T1, the verification is passed, and T2 can write. (Why not verify WS (T2)? WS (T1) has been submitted, and its timestamp is less than WS (T2). There must be no conflict in the previous part of WS (T2), and the following part, because it has not been read The object written by T1 is okay to write in, and will not overwrite the writing of WS (T1))
Otherwise, there is a read-write conflict and a write-write conflict between T2 and T1, and T2 needs to be rolled back. Read and write conflicts: T2 reads the version before T1 was written. After T1 commits, it may read the version written by T1, which cannot be read repeatedly. Write and write conflicts: T2 may be updated based on the old version and written again, causing the update of T1 to be lost.

Concurrency control based on MVCC

The database maintains multiple physical versions of a record. When a transaction is written, a new version of the written data is created, and the read request obtains the latest version of data that already exists at the time based on the snapshot information at the beginning of the transaction / statement. The most direct benefits it brings are: writing does not block reading, reading does not block writing, and read requests will never fail due to conflicts (such as single version T / O) or wait (such as single version 2PL). For database requests, there are often more read requests than write requests. Almost all mainstream databases use this optimization technology. MVCC is an optimization technology for read and write requests. It does not completely solve the database concurrency problem. It needs to be combined with the aforementioned several concurrency control technologies to provide complete concurrency control capabilities. Common types of concurrency control technologies include: MV-2PL, MV-T / O and MV-OCC, their characteristics are as follows: .png ") MVCC also has two key points to consider: multi-version data storage and redundant Recycling of version data. Multi-version data storage methods can be roughly divided into two categories: (1) Append only method, old and new versions are stored in the same table space, such as LSM-Tree-based storage engine. (2) Main table space Record the latest version of the data, the previous image is recorded in other table spaces or data segments, such as InnoDB's multi-version information is recorded in the undo log. Multi-version data collection is also known as garbage collection (GC), those that have no chance to be acquired by any read request The old version records should be deleted in time. 06

to sum up

This article introduces transactions based on locks (preventing conflicts before the start of transactions), T / O (judging conflicts during transaction execution), and Validation (verifying conflicts when transactions are committed) based on the timing of conflict handling (optimism). Concurrency control mechanism. Different implementations are suitable for different workloads, and workloads with low concurrency conflicts are certainly suitable for more optimistic concurrency control methods. MVCC can solve the problem of mutual blocking between read-only transactions and read-write transactions, improve the concurrent reading of transactions, and is adopted by most mainstream database systems.