Phantom reading and gap lock (Gap Lock)

Phantom reading

What is phantom reading

for example:

CREATE TABLE `t` (
  `id` int(11) NOT NULL,
  `c` int(11) DEFAULT NULL,
  `d` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `c` (`c`)
) ENGINE=InnoDB;
insert into t values(0,0,0),(5,5,5),
(10,10,10),(15,15,15),(20,20,20),(25,25,25);

Note: InnoDB's default transaction isolation level is repeatable read, so the parts that are not specifically explained in this article are set under the repeatable read isolation level.

Q3 The phenomenon of reading the line id=1 is called "phantom reading". In other words, the phantom read refers to when a transaction queries the same range twice before and after, the latter query sees rows that the previous query did not see. and:

The phantom reading will only appear under "current reading".
The phantom read only refers to the "newly inserted row".

Because these three queries have added for update, they are all currently read. The current read rule is to be able to read the latest value of all submitted records. In addition, the two statements of session B and session C will be submitted after execution, so Q2 and Q3 should see the operational effects of these two transactions , and also see that this does not contradict the visibility rules of the transaction .

What's wrong with the phantom reading

1. Semantic issues:

Session A declared at T1, "I want to lock all rows with d=5, and no other transactions are allowed to perform read and write operations." In fact, this semantic is destroyed.

The second statement of session B and session C destroys the lock statement of Q1 statement in session A to lock all d=5 rows.

2. The problem of data consistency:

The locking semantics of update is the same as select …for update, so it is reasonable to add this update statement at this time. Session A stated that "a lock should be added to the statement with d=5" in order to update the data. The newly added update statement is to modify the d value of the row it considers to be locked to 100 .

Let's see what will be the result in the database:

After T1, the line id=5 becomes (5,5,100). Of course, this result is finally officially submitted at T6;
After T2, the line id=0 becomes (0,5,5);
After T4 time, there is one more row in the table (1,5,5);
The other lines have nothing to do with this execution sequence and remain unchanged.

Looking at it this way, there is nothing wrong with the data, but let's take a look at the contents of the binlog at this time:

At T2, the session B transaction is committed and two statements are written;
At T4, the session C transaction is committed and two statements are written;
At T6, the session A transaction is committed, and the statement update t set d=100 where d=5 is written.

This sequence of sentences, whether you get the standby database to execute, or use binlog to clone a database later, the results of these three lines become (0,5,100), (1,5,100) and (5,5,100). That is to say, the two rows id=0 and id=1 have data inconsistency. This problem is very serious, and it won't work.

How to solve phantom reading:

Here, now you know, the reason for the phantom read is that the row lock can only lock the row, but for the action of inserting a new record, the "gap" between the records needs to be updated. Therefore, in order to solve the phantom read problem, InnoDB had to introduce a new lock, that is, a gap lock (Gap Lock).

Gap lock

What is a gap lock

As the name implies, gap lock, locks the gap between two values.

When you execute select * from t where d=5 for update, you will not only add row locks to the 6 existing records in the database , but also add 7 gap locks . This ensures that no new records can be inserted.

we all know. Row locks are divided into read locks and write locks, read-write exclusion, write-write exclusion, and conflicts between locks and locks.

However, gap lock is different. The conflicting relationship with gap lock is the operation of "insert a record into this gap". There is no conflict relationship between gap locks.

Session B will not be blocked because there is no record of c=7 in table t . Session A adds gap lock (5,10). And session B is also a gap lock added in this gap. They have the same goal, that is, to protect this gap and not allow values to be inserted. However, there is no conflict between them.

next-key lock

Gap lock and row lock are collectively called next-key lock, and each next-key lock is the interval before opening and closing.

After our table t is initialized, if we use select * from t for update to lock all the records of the entire table, 7 next-key locks are formed, which are (-∞,0], (0,5], ( 5,10], (10,15], (15,20], (20, 25], (25, +supremum].

The introduction of gap lock and next-key lock helped us solve the problem of phantom reading, but it also brought some "difficulties".

The business logic is like this: lock a row arbitrarily, insert if the row does not exist, and update its data if the row exists. Once this logic has concurrency, it will encounter a deadlock.

Session A executes the select… for update statement. Since the line id=9 does not exist, the gap lock (5,10) will be added;
Session B executes the select… for update statement, and gap locks (5, 10) are also added. There will be no conflict between gap locks, so this statement can be executed successfully;
Session B tried to insert a row (9,9,9), which was blocked by the gap lock of session A, and had to wait;
Session A tried to insert a row (9,9,9), which was blocked by the gap lock of session B.

Of course, InnoDB's deadlock detection immediately discovered this pair of deadlock relationships, causing the insert statement of session A to report an error and return. The introduction of gap locks may cause the same statement to lock a larger range, which actually affects the degree of concurrency . Actually, this is just a simple example

Note: At the beginning of the article, we emphasized that under the premise of repeatable read level, the gap lock will only take effect under the repeatable read isolation level.

If you set the isolation level to read commit , there is no gap lock. But at the same time, if you want to solve possible data and log inconsistencies , you need to set the binlog format to row . If the read submission isolation level is sufficient, that is, the business does not need repeatable read guarantees, so considering that the lock range of the operation data under read submission is smaller (no gap lock), this choice is reasonable .

Locking rules

Prerequisite: The later version of MySQL may change the locking strategy, so this rule is limited to the latest version up to now, that is, 5.x series <=5.7.24, 8.0 series <=8.0.13.

This locking rule contains two "principles", two "optimizations" and one "bug" :

Principle 1: The basic unit of locking is next-key lock, which is the interval between front opening and closing.
Principle 2: Objects accessed during the search process will be locked.
Optimization 1: Equivalent query on the index, when the unique index is locked, next-key lock degenerates into row lock.
Optimization 2: Equivalence query on the index. When traversing to the right and the last value does not meet the equivalence condition, next-key lock degenerates into a gap lock.
A bug: the range query on the unique index will access until the first value that does not meet the condition.

case study

The following cases are examples of the tables and data mentioned above.

Since there is no record with id=7 in table t, use the locking rules we mentioned above to judge:

According to principle 1, the locking unit is next-key lock, and the locking range of session A is (5,10);
At the same time, according to optimization 2, this is an equivalent query (id=7), and id=10 does not meet the query conditions, and the next-key lock degenerates into a gap lock, so the final lock range is (5,10).

Session A needs to add a read lock to the line c=5 on index c:

According to principle 1, the unit of locking is next-key lock, so next-key lock is added to (0,5).
c is an ordinary index, so only access to the record c=5 cannot stop immediately. You need to traverse to the right and give up when c=10. According to principle 2, all accesses must be locked, so add next-key lock to (5,10).
But at the same time, this conforms to optimization 2: equivalence judgment, traversing to the right, the last value does not meet the equivalence condition of c=5, so it degenerates into a gap lock (5, 10).
According to principle 2, only the accessed objects will be locked. This query uses a covering index and does not need to access the primary key index, so there is no lock on the primary key index. This is why the update statement of session B can be executed.

It should be noted that in this example, lock in share mode only locks the covering index , but it is different if it is for update. When executing for update, the system will think that you want to update the data next, so it will add row locks to the rows that meet the conditions on the primary key index by the way. In addition, this example shows that the lock is added to the index ; at the same time, it gives us the guidance that if you want to use lock in share mode to add a read lock to the row to prevent data from being updated, you must bypass the covering index Optimize, add fields that do not exist in the index in the query field.

The figure on the left is a unique index range query:

At the beginning of execution, it is necessary to find the first row with id=10, so it should be next-key lock(5,10]. According to optimization 1, the equivalent condition on the primary key id degenerates into a row lock with only id added =10 Row lock for this row.
The range search will continue to search, find the id=15 line and stop, so you need to add next-key lock(10,15].

The figure on the right is a non-unique index range query:

After the next-key lock (5,10) is added to the index c of session A, since index c is a non-unique index, there is no optimization rule, which means that it will not degenerate into a row lock, so the final lock added by sesion A is index The two next-key locks (5,10] and (10,15) on c .

Session A is a range query. According to principle 1, only the next-key lock (10,15) should be added to the index id, and because id is a unique key, the loop should stop when it reaches id=15. . But in terms of implementation, InnoDB will scan forward to the first behavior that does not meet the condition, that is, id=20. And because this is a range scan, the next-key lock (15,20) on the index id is also Will be locked.

5. In the above example, insert a new record into table t: insert into t values(30,10,30);

Now there are two rows with c=10 in the table, but their primary key value id is different (10 and 30 respectively), so there is a gap between the two records with c=10 . This time we use the delete statement to verify. Note that the lock logic of the delete statement is actually similar to select ... for update, which is the two "principles", two "optimizations" and one "bug" I summarized at the beginning of the article.

When session A is traversing, first visit the first record with c=10. Similarly, according to principle 1, here is the next-key lock (c=5,id=5) to (c=10,id=10) .
Then, session A looks to the right until it hits the line (c=15, id=15), and the loop does not end. According to optimization 2, this is an equivalent query, and the row that does not meet the condition is found to the right, so it will degenerate into a gap lock from (c=10,id=10) to (c=15,id=15).

There are dotted lines on the left and right sides of this blue area, indicating the open interval, that is, there is no lock on the two lines (c=5,id=5) and (c=15,id=15).

6. Continue to case 5 above

Limit 2 is added to the delete statement of session A. You know that there are actually only two records with c=10 in table t, so if limit 2 is added or not, the effect of deletion is the same, but the effect of locking is different. As you can see, the insert statement of session B has passed. Therefore, after traversing to the line (c=10, id=30), there are already two statements that meet the conditions, and the loop ends.

Therefore, the lock range on index c becomes from (c=5,id=5) to (c=10,id=30), which is the open and closed interval , as shown in the following figure:

7. Examples of deadlock

After session A starts the transaction, execute the query statement plus lock in share mode, and add next-key lock(5,10) and gap lock(10,15) to index c;
The update statement of session B should also add next-key lock(5,10] to index c to enter lock waiting;
Then session A should insert the line (8,8,8) again, which is locked by the gap lock of session B. Due to a deadlock, InnoDB lets session B roll back.

Since it is order by c desc, the first row to be located is the "rightmost" c=20 row on index c, so gap lock (20,25) and next-key lock (15,20] will be added.
Traverse to the left on index c, and stop until c=10, so next-key lock will be added to (5,10), which is the reason why the insert statement of session B is blocked.
During the scanning process, the three rows c=20, c=15, and c=10 all have values. Because it is select *, three row locks are added to the primary key id.

Therefore, the lock range of the select statement of session A is: (5, 25) on index c, and id=15 and 20 on the primary key index.

Content source: Lin Xiaobin "45 Lectures on MySQL Actual Combat"