[Database] Detailed Explanation of MVCC Principles

foreword

The principle of MVCC implementation is a very high-frequency interview question. Recently, the friends in the technical discussion group have been discussing it. Let’s chat together while we are free during the May Day holiday.

1. Review of relevant database knowledge points

1.1 What is a database transaction and why there is a transaction

A transaction consists of a limited sequence of database operations that are either all executed or not executed at all, and are an indivisible unit of work.

If A transfers 100 yuan to B, first deduct 100 yuan from A's account, and then add 100 yuan to B's account
. If A’s 100 yuan has been deducted, but there is no time to add it to B, the banking system is abnormal, and finally A’s balance decreases, but B’s balance does not increase. So you need a transaction, roll back A's money, it's that simple.

Why should there be a business? It is to ensure the final consistency of the data.

1.2 What features does a transaction include?

The four typical characteristics of transactions are ACID, Atomicity, Consistency, Isolation, and Durability.

• Atomicity: The transaction is executed as a whole, and the operations on the database included in it are either all executed or none of them are executed.
  Consistency: It means that the data will not be destroyed before the transaction starts and after the transaction ends. If account A transfers 10 yuan to account B, the total amount of A and B will remain the same regardless of success or failure.
• Isolation: When multiple transactions access concurrently, the transactions are isolated from each other. One transaction should not be interfered by other transactions, and multiple concurrent transactions should be isolated from each other. .
• Persistence: Indicates that after the transaction is committed, the operational changes made by the transaction to the database will be persisted in the database.

1.3 Problems with transaction concurrency

Concurrency of transactions can cause dirty reads, non-repeatable reads, and phantom reads.

1.3.1 Dirty reads

If a transaction reads data modified by another uncommitted transaction, we say that a dirty read has occurred.

Suppose there are two transactions A and B:

• Suppose the balance of A is now 100, and transaction A is preparing to query Jay's balance
• Transaction B deducts Jay's balance first, deducting 10, but has not yet committed
• The last balance read by A is 90, which is the balance after deduction

Because transaction A reads uncommitted data from transaction B, this is a dirty read.

1.3.2 Non-repeatable read

In the same transaction, read multiple times before and after, and the content of the read data is inconsistent

Suppose now there are two transactions A and B:

• Transaction A first checks Jay's balance, and the result is 100
• At this time, transaction B deducts Jay's account balance, after deducting 10, submit the transaction
• Transaction A checks Jay's account balance and finds that it has become 90

Transaction A is interfered by transaction B! In the scope of transaction A, two identical queries read the same record but return different data, which is non-repeatable read.

1.3.3 Phantom read

If a transaction first queries some records according to certain search conditions, and when the transaction is not committed, another transaction writes some records (such as insert, delete, update) that meet those search conditions, which means that phantom reading has occurred .

Suppose there are two transactions A and B:

• Transaction A first queries the account records with id greater than 2, and gets two records with id=2 and id=3
• At this time, transaction B opens, inserts a record with id=4, and submits
• Transaction A executes the same query again, but gets 3 records with id=2,3,4.

Transaction A queries a range of result sets, another concurrent transaction B inserts new data into this range, and submits the transaction, and then transaction A queries the same range again, but the result sets read twice are different. It is phantom reading.

1.4 Four major isolation levels

In order to solve the problems of dirty reads, non-repeatable reads, and phantom reads in concurrent transactions, Uncle Database designed four isolation levels. They are read uncommitted, read committed, repeatable read, serializable (Serializable).

1.4.1 Read Uncommitted

The read uncommitted isolation level only restricts that two data cannot be modified at the same time, but when the data is modified, even if the transaction is not committed, it can be read by other transactions. This level of transaction isolation includes dirty reads and repeated reads , The problem of phantom reading;

1.4.2 Read Committed

Read committed isolation level, the current transaction can only read the data submitted by other transactions, so the isolation level of this transaction solves the problem of dirty reads, but there are still problems of repeated reads and phantom reads;

1.4 3 Repeatable read

The repeatable read isolation level limits the reading of data and cannot be modified, so it solves the problem of repeated reading, but when reading range data, data can be inserted, so there will still be phantom reading problems;

1.4.4 Serialization

The highest isolation level for transactions, under which all transactions are executed serially. All concurrent problems of dirty reads, non-repeatable reads, and phantom reads can be avoided. However, under this transaction isolation level, transaction execution consumes a lot of performance.

1.4.5 What concurrency problems will exist in the four isolation levels?

isolation level	dirty read	non-repeatable read	Phantom reading
read uncommitted	√	√	√
read committed	×	√	√
repeatable read	×	×	√
Serialization	×	×	×

1.5 How does the database guarantee the isolation of transactions?

The database implements transaction isolation through locking. It's like, if you want to be alone and not disturbed by others, you can put a lock on the door.

Locking is really easy to use and can ensure isolation. For example, the serialization isolation level is implemented by locking. However, frequent locking makes it impossible to modify the data when it is read, and it cannot be read when the data is modified, which greatly reduces the performance of the database.

So, how to solve the performance problem after locking?

The answer is, MVCC multi-version concurrency control! It realizes reading data without locking, and allows reading data to be modified at the same time. It can be read while modifying data.

2. What is MVCC?

MVCC, ie Multi-Version Concurrency Control （多版本并发控制）. It is a concurrency control method, generally in the database management system, to achieve concurrent access to the database, in the programming language to achieve transactional memory.

In layman's terms, there are multiple versions of data in the database at the same time, not multiple versions of the entire database, but multiple versions of a certain record exist at the same time. When a transaction operates on it, you need to check this one The transaction version id of the hidden column of the record, compare the transaction id and judge which version of the data to read according to the transaction isolation level.

The database isolation level read committed and repeatable read are all implemented based on MVCC. Compared with the simple and crude way of locking, it uses a better way to deal with read and write conflicts, which can effectively improve the concurrency performance of the database.

3. Key knowledge points of MVCC implementation

3.1 Transaction version number

Before each transaction is started, a self-increasing transaction ID will be obtained from the database, and the execution order of the transactions can be judged from the transaction ID. This is the transaction version number.

3.2 Implicit fields

For the InnoDB storage engine, each row of records has two hidden columns trx_id and roll_pointer. If there is no primary key and non-NULL unique key in the table, there will be a third hidden primary key column row_id.

column name	Is it necessary	describe
row_id	no	Monotonically increasing row ID, not required, takes 6 bytes.
trx_id	yes	Record the transaction ID of the transaction that operates the data
roll_pointer	yes	This hidden column is equivalent to a pointer to the undo log of the rollback segment

3.3 undo log

undo log, rollback log, used to record the information before the data is modified. Before the table records are modified, the data will be copied to the undo log first. If the transaction is rolled back, the data can be restored through the undo log.

Picture
It can be considered that when a record is deleted, a corresponding insert record will be recorded in the undo log, and when a record is updated, it will record a corresponding update record.

What is the use of undo log?

1. Atomicity and consistency are guaranteed when a transaction is rolled back.
2. Used for MVCC snapshot reads.

3.4 Version chain

When multiple transactions operate a row of data in parallel, the modification of the row of data by different transactions will generate multiple versions, and then through the rollback pointer (roll_pointer), they will be connected into a linked list. This linked list is called a version chain. as follows:

In fact, through the version chain, we can see the relationship between the transaction version number, the hidden columns of the table and the undo log. Let's analyze it again.

1. Assuming that there is a core_user table now, there is a piece of data in the table, the id is 1, and the name is Sun Quan:
   

2. Now start a transaction A: execute update core_user set name = "Cao Cao" where id=1 on the core_user table, the following process will be performed

• First get a transaction ID=100
• Copy the data of the core_user table before modification to the undo log
• Modify the data with id=1 in the core_user table, and change the name to Cao Cao
• Change the modified data transaction Id=101 to the current transaction version number, and point the roll_pointer to the undo log data address.
  

3.5 Snapshot read and current read

Snapshot Read: Reads the visible version (with older versions) of record data. Without locking, ordinary select statements are all snapshot reads, such as:

select * from core_user where id > 2;

Current read: the latest version of the record data is read, and the current read is explicitly locked

select * from core_user where id > 2 for update;
select * from account where id>2 lock in share mode;

3.6 Read View

• What is Read View? It is the read view generated when the transaction executes the SQL statement. In fact, in innodb, each SQL statement will get a Read View before execution.
• What is the use of Read View? It is mainly used for visibility judgment, that is, to judge which version of data is visible in the current transaction~

How does Read View ensure visibility judgment? Let's take a look at several important properties of Read view

• m_ids: Active (uncommitted) read and write transaction IDs in the current system, its data structure is a List.
• min_limit_id: Indicates the smallest transaction id among active read and write transactions in the current system when Read View is generated, that is, the minimum value in m_ids.
• max_limit_id: Indicates the id value that should be assigned to the next transaction in the system when Read View is generated.
• creator_trx_id: Create the transaction ID of the current Read View

Read view matching condition rules are as follows:

1. If the data transaction ID trx_id < min_limit_id, it means that the transaction that generated this version has been committed before generating Read View (because the transaction ID is incremented), so this version can be accessed by the current transaction.
2. If trx_id>= max_limit_id, it means that the transaction that generates this version is generated after ReadView is generated, so this version cannot be accessed by the current transaction.
3. If min_limit_id =<trx_id< max_limit_id, 3 situations need to be discussed

(1). If m_ids contains trx_id, it means that the transaction has not been submitted at the time of Read
View generation, but if the trx_id of the data is equal to creator_trx_id, it indicates that the data is generated by itself, so it is visible.
(2) If m_ids contains trx_id, and trx_id is not equal to creator_trx_id, then
when Read View is generated, the transaction is not committed, and it is not produced by itself, so the current transaction is also invisible;
(3). If m_ids does not contain trx_id, it means Your transaction has been submitted before Read View is generated, and the result of the modification can be seen by the current transaction.

4. Analysis of the principle of MVCC implementation

4.1 What is the process of querying a record based on MVCC

1. Get the version number of the transaction itself, that is, the transaction ID
2. Get Read View
3. Query the obtained data, and then compare the transaction version number in Read View.
4. If the visibility rules of Read View are not met, the historical snapshot in the Undo log is required;
5. Finally, return the data that meets the rules

InnoDB implements MVCC through Read View + Undo Log. Undo Log saves historical snapshots. Read View visibility rules help determine whether the current version of data is visible.

4.2 Read committed (RC) isolation level, the analysis history of non-repeatable read problems

1. Create a core_user table and insert an initialization data, as follows:
   

2. The isolation level is set to read committed (RC), and transaction A and transaction B perform query and modification operations on the core_user table at the same time.

事务A: select * fom core_user where id=1
事务B: update core_user set name =”曹操”

The execution process is as follows:

Finally, the query result of transaction A is the record of name=Cao Cao. Let’s analyze the execution process based on MVCC:

(1). A starts a transaction, and first obtains a transaction ID of 100

(2).B starts the transaction and gets the transaction ID as 101

(3). Transaction A generates a Read View, and the values ​​corresponding to the read view are as follows

variable	value
m_ids	100，101
max_limit_id	102
min_limit_id	100
creator_trx_id	100

Then go back to the version chain: start picking visible records from the version chain:

It can be seen from the figure that the content of the column name in the latest version is Sun Quan, and the value of trx_id in this version is 100. Start to execute read view visibility rule verification:

min_limit_id(100)=<trx_id（100）<102;
creator_trx_id = trx_id =100;

From this, it can be seen that the current transaction is visible for the record of trx_id=100. So it is found that the name is Sun Quan's record.

(4). Transaction B performs a modification operation and changes the name to Cao Cao. Copy the original data to the undo log, then modify the data, and mark the transaction ID and the address of the previous data version in the undo log.

(5) Submit the transaction

(6) Transaction A executes the query operation again, and generates a new Read View, and the corresponding values ​​of the Read View are as follows

variable	value
m_ids	100
max_limit_id	102
min_limit_id	100
creator_trx_id	100

Then go back to the version chain again: pick the visible records from the version chain:

It can be seen from the figure that the content of the column name in the latest version is Cao Cao, and the trx_id value of this version is 101. Start to perform Read View visibility rule verification:

min_limit_id(100)=<trx_id（101）<max_limit_id（102);
但是,trx_id=101，不属于m_ids集合

Therefore, the record trx_id=101 is visible to the current transaction. So the SQL query is the record whose name is Cao Cao.

To sum up, under the Read Committed (RC) isolation level, in the same transaction, two identical queries read the same record (id=1), but returned different data (the first time it was found out is Sun Quan, the second time I found out it was Cao Cao's record), so the RC isolation level has non-repeatable read concurrency issues.

4.3 Repeatable read (RR) isolation level, analysis to solve the problem of non-repeatable read

Under the RR isolation level, how to solve the non-repeatable read problem? Let's take a look again,

It is still the process in section 4.2, or this transaction A and transaction B, as follows:

4.3.1 Under different isolation levels, Read View works differently

In fact, the working methods of Read view under various transaction isolation levels are different. RR can solve the non-repeatable read problem, which is related to the working method of Read view.

• Under the read committed (RC) isolation level, in the same transaction, each query will generate a new Read View copy, which may cause the problem that the data read before and after the same transaction may be inconsistent (non-repeatable read concurrency problem) .
  begin ||
  :–|:–
  select * from core_user where id =1 |generate a Read View
  /| /
  / |/
  select * from core_user where id =1 |generate a Read View

• Under the Repeatable Read (RR) isolation level, only one read view is obtained in a transaction, which is shared by replicas, so as to ensure that the data in each query is the same.

begin
select * from core_user where id =1	Generate a Read View
/
/
select * from core_user where id =1	Share a copy of Read View

4.3.2 Example analysis

Let's go back to the 4.2 example, and then execute the second query:

Transaction A executes the query operation again and reuses the old copy of Read View. The corresponding values ​​of Read View are as follows

variable	value
m_ids	100，101
max_limit_id	102
min_limit_id	100
creator_trx_id	100

Then go back to the version chain again: pick the visible records from the version chain:

It can be seen from the figure that the content of the column name in the latest version is Cao Cao, and the trx_id value of this version is 101. Start to execute read view visibility rule verification:

min_limit_id(100)=<trx_id（101）<max_limit_id（102);
因为m_ids{
   
   100,101}包含trx_id（101），
并且creator_trx_id (100) 不等于trx_id（101）

Therefore, the record trx_id=101 is not visible to the current transaction. At this time, the version chain roll_pointer jumps to the next version, and the record trx_id=100 is checked again to see if it is visible:

min_limit_id(100)=<trx_id（100）< max_limit_id（102);
因为m_ids{
   
   100,101}包含trx_id（100），
并且creator_trx_id (100) 等于trx_id（100）

Therefore, the record trx_id=100 is visible to the current transaction, so the results of the two queries are the record with name=Sun Quan. That is, under the repeatable read (RR) isolation level, the old Read View copy is reused to solve the problem of non-repeatable read.

4.4 According to the legend of the Internet, does MVCC solve the phantom reading problem?

There is a legend in the Internet world that the RR isolation level of MVCC solves the problem of phantom reading. Let's analyze it together.

4.4.1 Under RR level, an example of snapshot read, there is no phantom read problem

It can be seen from the figure that there is no change in the query result set of step 2 and step 6. It seems that the RR level has solved the problem of phantom reading~

4.4.2 Under RR level, an example of current reading

Suppose there is an account table now, and there are 4 pieces of data in the table, RR level.

Start transaction A, execute the current read, and query all records with id>2.
Then start transaction B and insert a piece of data with id=5.
The process is as follows:

Obviously, when transaction B performs the insert operation, it is blocked ~ because when transaction A executes select ... lock in share mode (current read), it not only adds a lock on the two records with id = 3, 4, but also adds a lock on the id > 2 Gap locks are also added to this range.

Therefore, we can find that under the RR isolation level, locked select, update, delete and other statements will use gap lock + adjacent key lock to lock the range between index records and avoid inserting records between ranges to avoid phantoms Line records, that is to say, the RR isolation level solves the problem of phantom reading?

4.4.3 In this special scenario, there seems to be a problem of phantom reading

In fact, in transaction A in the above figure, update account set balance=200 where id=5 is added; this operation, the same transaction, the same sql, the result set detected is different, this result is in line with the phantom reading definition~

This question, dear friend, do you think it is a phantom reading problem, so the RR isolation level still has the phantom reading problem? Welcome everyone to leave a message in the comment area.

References

[1] Database foundation (4) Innodb MVCC implementation principle: https://zhuanlan.zhihu.com/p/52977862