Interviewer: How does MySQL solve phantom reading?

Introduction

As we all know, under different isolation levels, the following problems will occur.

√ means it will happen, × means it will not happen

Isolation level	Dirty read	Non-repeatable	Phantom reading
read uncommitted	√	√	√
read committed	×	√	√
repeatable read	×	×	√
serializable	×	×	×

I don’t know how these problems arise. You can read the following article " Interviewer: Dirty Reading, Non-repeatable Reading, How Does the Phantom Reading Happen?" 》

So how does mysql avoid dirty reads, non-repeatability, and phantom reads? There are actually two options

Solution 1: Use multi-version concurrency control (MVCC) for read operations, and only locks for write operations

As mentioned before, mvcc will generate a ReadView every time a transaction is opened, and then find the version on the version chain that is visible to the current transaction. The historical version of the read record and the latest version of the change record do not conflict, so when using MVCC, the read and write will not conflict

Scheme 2: Read and write operations are all locked

Dirty read refers to when a transaction is accessing data, and the data is modified, and this modification has not been committed to the database, at this time, another transaction also accesses the data, and then uses the data.

If another transaction locks the record while modifying the record, and releases the lock after the transaction commits, then the current transaction cannot obtain the lock while reading the record, and there will be no dirty read

Non-repeatable read means that in transaction 1, a piece of data is read, and when transaction 1 has not yet ended, transaction 2 also accesses the data, modifies the data, and commits it. Immediately afterwards, transaction 1 reads this data again. Due to the modification of transaction 2, the data read twice in transaction 1 may be different, so it is called non-repeatable read.

If the current transaction locks the record when reading the record, then another transaction cannot modify the record, and naturally there will be no non-repeatable reading.

The phantom read refers to when a transaction is reading a record in a certain range, another transaction inserts a new record in the range, and when the previous transaction reads the record in the range again, it will read Records inserted by other transactions, these new records are phantom records (The InnoDB storage engine solves the phantom reading problem in this situation through multi-version concurrency control (MVCC) and gap locks )

We will talk about MVCC and gap lock later.

How to lock in this case? Because the phantom record does not exist when the record is read for the first time. See you later

Locks in MySQL

There are three levels of locks in MySQL, table locks, row locks, and page locks

Table lock : low overhead, fast locking; no deadlock; large locking granularity, the highest probability of lock conflicts, and the lowest concurrency. Will happen in: MyISAM, memory, InnoDB, BDB and other storage engines

Row locks : high overhead and slow locking; deadlocks will occur; the smallest locking granularity, the lowest probability of lock conflicts, and the highest degree of concurrency. Will happen: InnoDB storage engine

Page locks : The overhead and lock time are between table locks and row locks; deadlocks will occur; the locking granularity is between table locks and row locks, and the degree of concurrency is average. Will happen: BDB storage engine

Locks in InnoDB

There are the following two types of row-level locks in the InnoDB storage engine

1. Shared lock (Shared Lock, referred to as S lock), when the transaction needs to read a record, you need to obtain the S lock to change the record first
2. Exclusive lock (Exclusive Lock, referred to as X lock), when a transaction wants to change a record, the X lock of the record needs to be acquired first

If transaction T1 obtains the S lock of a record, transaction T2 will also access this record. If transaction T2 wants to acquire the S lock of this record again, it can succeed. This situation is called lock compatibility. If transaction T2 wants to acquire the X lock of this record again, then this operation will be blocked until the S lock is committed after transaction T1 is committed. Release

If transaction T1 acquires the X lock of a record, no matter whether transaction T2 then wants to acquire the S lock or X lock of the record, it will be blocked until transaction 1 commits. This situation is called lock incompatibility.

Multiple transactions can read records at the same time, that is, shared locks are not mutually exclusive, but shared locks will block exclusive locks. Mutual exclusion between exclusive locks

The compatibility relationship between S lock and X lock is as follows

compatibility	X lock	S lock
X lock	Not compatible	Not compatible
S lock	Not compatible	compatible

We can lock the read records through the following statement.

Add S lock to the read record

select .. lock in share mode;

Add X lock to the read record

select ... for update

Table lock

Table level S lock, X lock

When executing select, insert, update, or delete statements on a table, the InnoDB storage engine will not add table-level S locks or X locks for this table.

When some DDL statements such as ALTER TABLE and DROP TABLE are executed on the table, X locks will be added to this table, so other transactions on this table will block the execution of statements such as SELECT INSERT UPDATE DELETE

When the system variable autocommit=0 and innodb_table_locks=1, manually obtain the S lock or X lock of the table t provided by the InnoDB storage engine, you can write like this

Add table-level S lock to table t

lock tables t read

Add table-level X lock to table t

lock tables t write

If a transaction adds an S lock to the table, then

• Other transactions can continue to obtain the S lock of the table
• Other transactions can continue to obtain S locks for some records in the table
• No other transaction can continue to obtain the X lock of the table
• Other transactions cannot continue to obtain X locks on some records in the table

If a transaction adds an X lock to the table, then

• Other transactions can not continue to obtain the S lock of the table
• Other transactions cannot continue to obtain S locks on some records in the table
• No other transaction can continue to obtain the X lock of the table
• Other transactions cannot continue to obtain X locks on some records in the table

Therefore, you must be careful when modifying online tables, because it will block a large number of transactions . There are currently many mature ways to modify online tables, so I won’t go into details.

Table-level IS lock, IX lock

Why are there table-level IS locks and IX locks?
Let’s use the example of a teaching building and a classroom to compare

Each of us can go to the classroom to study. A classroom can accommodate multiple people to study. One person can learn to hang an S lock at the door, and the classroom can hang multiple S locks (equivalent to row-level S locks). When the classroom was under maintenance, no other work could be carried out. An X lock (equivalent to a row-level X lock) was hung at the door of the classroom.

A leader comes to inspect the teaching building. Students can study normally, but no classrooms are under maintenance. Hang an S lock (equivalent to a table-level S lock) at the entrance of the teaching building. Students can study normally when they see the S lock in the teaching building, and the repairman waits until they see the X lock in the teaching building.

The school will occupy the teaching building for examinations, students are not allowed to study, and repairs are not allowed. Hang an X lock (equivalent to an X lock at the meter level) at the entrance of the teaching building. At this time, students and repairmen have to wait

There are two problems with this

1. If you want to lock the upper floor of the teaching building, you must ensure that there are no classrooms under maintenance
2. If you want to lock X upstairs, you must ensure that there are no classrooms under maintenance and all classrooms are empty

Check the classrooms one by one, which is too inefficient. Can do this

• Students go to the classroom to study, first hang an IS lock at the door of the teaching building, and then hang an S lock at the door of the classroom
• The maintenance worker goes to the classroom to study, first hang an IX lock at the entrance of the teaching building, and then hang an X lock at the entrance of the classroom

In this way, if you want to lock the S on the teaching building, you only need to see if there is an IX lock at the entrance of the teaching building. If you
want to lock the X on the teaching building, you only need to see if there is an IX and IS lock at the entrance of the teaching building

Using the InnoDB storage engine, you need to add an IS lock at the table level before adding an S lock to the table records. Before adding an X lock to a table record, you need to add an IX lock at the table level. IS locks and IX locks are only to determine whether there are locked records in the table when subsequent table-level S locks and X locks are added, and to avoid traversing to see if there are locked records in the table.

Table level AUTO-INC lock

In the process of using MySQL, we can add the AUTO_INCREMENT attribute to a column of the table, and then when inserting the record, you can not specify the value column of the column, and the system assigns him an incremental value

As in the following table

CREATE TABLE t (
    id INT NOT NULL AUTO_INCREMENT,
    c VARCHAR(100),
    PRIMARY KEY (id)
) Engine=InnoDB CHARSET=utf8;

INSERT INTO t(c) VALUES('aa'), ('bb');

After inserting 2 records, the result is as follows

mysql> SELECT * FROM t;
+----+------+
| id | c    |
+----+------+
|  1 | aa   |
|  2 | bb   |
+----+------+
2 rows in set (0.00 sec)

The principle of MySQL automatically incrementing and assigning values ​​to AUTO_INCREMENT modified columns is mainly in the following two ways

1. Use AUTO-INC lock, add an AUTO-INC lock at the table level when inserting a statement, and then assign an incremental value to each column to be inserted in the AUTO_INCREMENT modified column of the record to be inserted, and then release the AUTO-INC lock after the insert statement is executed. Lost (the lock is released after the insert statement is executed, not at the end of the transaction ). When such a transaction is holding the AUTO-INC lock, the insert statements of other transactions must be blocked, which can ensure that the assigned increment value is continuous
2. Use a lightweight lock to obtain the lightweight lock when generating the value of the AUTO_INCREMENT modified column for the insert statement, and release the lightweight lock after generating the value of the AUTO_INCREMENT column required by the insert statement. You don't need to wait until the entire insert statement is executed before inserting the lock

Finally, summarize the compatibility of table-level locks

compatibility	IS	IX	S	X	AUTO_INC
IS	compatible	compatible	compatible	Not compatible	compatible
IX	compatible	compatible	Not compatible	Not compatible	compatible
S	compatible	Not compatible	compatible	Not compatible	Not compatible
X	Not compatible	Not compatible	Not compatible	Not compatible	Not compatible
AUTO_INC	compatible	compatible	Not compatible	Not compatible	Not compatible

Row lock

CREATE TABLE `girl` (
  `id` int(11) NOT NULL,
  `name` varchar(255),
  `age` int(11),
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

insert into girl values
(1, '西施', 20),
(5, '王昭君', 23),
(8, '貂蝉', 25),
(10, '杨玉环', 26),
(12, '陈圆圆', 20);

There are the following three locks in InnoDB

1. Record Lock: Lock a single record
2. Gap Lock: Gap lock, lock the gap in front of the record, do not allow the record to be inserted
3. Next-key Lock: Lock the gap between the data and the data at the same time, that is, the gap between the data and the data is not allowed to insert records

Record Lock

Lock a single record

For example, add a Record Lock to the data with the id value of 8, the diagram is as follows.

Record Lock also has S lock and X lock, and the compatibility is the same as the previous description.

What kind of lock is added to SQL execution is restricted by many conditions, such as the isolation level of the transaction and the index used during execution (eg, clustered index, non-clustered index, etc.), so I won't analyze it in detail, and give a few simple examples.

-- READ UNCOMMITTED/READ COMMITTED/REPEATABLE READ 利用主键进行等值查询
-- 对id=8的记录加S型Record Lock
select * from girl where id = 8 lock in share mode;

-- READ UNCOMMITTED/READ COMMITTED/REPEATABLE READ 利用主键进行等值查询
-- 对id=8的记录加X型Record Lock
select * from girl where id = 8 for update;

Gap Lock

Lock the gap in front of the record, do not allow the record to be inserted

MySQL can solve the problem of phantom reading through MVCC and locking under the repeatable read isolation level (more details will be introduced later)

But how to lock it? Because these locks do not exist when the read operation is performed for the first time, we have no way to add Record Lock. At this time, it can be solved by adding Gap Lock, that is, lock the gap.

If a transaction adds a gap lock to the record with id=8, it means that other transactions are not allowed to insert new records in the gap before the record with id=8, that is, records with id values ​​in the interval (3, 8) are not allowed Allow immediate insertion. Until the transaction with the gap lock is committed, the record with the id value in the interval (3, 8) can be committed

-- REPEATABLE READ 利用主键进行等值查询
-- 但是主键值并不存在
-- 对id=8的聚集索引记录加Gap Lock
SELECT * FROM girl WHERE id = 7 LOCK IN SHARE MODE;

Since the record with id=7 does not exist, in order to prohibit the phenomenon of phantom reading (to avoid the record of id=7 in the result set obtained by executing the same statement under the same transaction), we must prevent other transactions from inserting id before the current transaction is committed. =7 records, add a Gap Lock to the record with id=8 at this time, that is, no other transactions are allowed to insert new records with the id value in the range (5, 8)

Let me ask you a question. Gap Lock can only lock the gap in front of the record. How to lock the gap after the last record?
In fact, mysql data is stored in pages, each page has 2 pseudo records

1. Infimum record, which means the smallest record on the page
2. upremum record, which means the largest record on the page

In order to prevent transactions from inserting records whose id value is in the range of (12, +∞), we can add a gap lock to the Supremum record on the page where the id=12 record is located. At this time, other transactions can be prevented from inserting the id value in (12, +∞). , +∞) new record in this interval

Next-key Lock

At the same time, lock the gap between the data and the data, that is, the gap between the data and the data is not allowed to insert records

-- REPEATABLE READ 利用主键进行范围查询
-- 对id=8的聚集索引记录加S型Record Lock
-- 对id>8的所有聚集索引记录加S型Next-key Lock（包括Supremum伪记录）
SELECT * FROM girl WHERE id >= 8 LOCK IN SHARE MODE;

To solve the problem of phantom reading, it is necessary to prohibit other transactions from inserting records with id>=8, so

• Add S-type Record Lock to the clustered index record with id=8
• Add S-type Next-key Lock to all clustered index records with id>8 (including Supremum pseudo records)

How does MySQL solve phantom reads at the repeatable read level?

As we all know, under different isolation levels, the following problems will occur.
√ means it will happen, × means it will not happen

Isolation level	Dirty read	Non-repeatable	Phantom reading
read uncommitted	√	√	√
read committed	×	√	√
repeatable read	×	×	√
serializable	×	×	×

As mentioned earlier, we need to set the isolation level of the database to be serializable to solve the phantom reading, but the phantom reading problem has been solved under the MySQL repeatable isolation level. So how does it solve it?

This has to mention two ways to read data in MySQL

Snapshot read

Use multi-version concurrency control (MVCC) to read the version that is visible to the current transaction on the version chain. The implementation of MVCC can be seen in the following article. I will not introduce the " Interviewer: How is MVCC implemented?" 》

MVCC avoids dirty reads, non-repeatable reads, and phantom reads by reading the visible records on the version chain. After all, reads and writes will not conflict, which can greatly improve concurrency. Because the data that may be read is the previous data, it is called snapshot read. But in some scenarios, users need to read the latest records in the database. This requires the database to support locking statements, even for select read-only operations , this does not mention the current read we are going to talk about below

Current reading

To operate on the latest record of the database, the statement is as follows

select ... lock in share mode;
select ... for update;
insert; 
update; 
delete;

Reference blog

"How MySQL Works : Understanding MySQL from the Basics "
[1]https://blog.csdn.net/Saintyyu/article/details/91269087
[2]https://www.toutiao.com/a6838563153626792451/
mvcc And gap lock
[3] https://www.huaweicloud.com/articles/f571bafcbe55475cd94d1f2f65e729a9.html
statement lock analysis
[4] https://mp.weixin.qq.com/s/Lavoo9sgulOzxQ22GRAamw