Source: He Dengcheng

Link: http://hedengcheng.com/?p=771

Posted on December 13, 2013

background

The locking analysis of MySQL/InnoDB has always been a difficult topic. In the course of my work, I often have colleagues asking questions in this regard. At the same time, I often receive private messages related to MySQL locks on Weibo, asking me to help solve some deadlock problems. In this article, I am going to conduct a more in-depth analysis and discussion on the locking problem of MySQL/InnoDB, mainly to introduce an idea. Using this idea, if you get any SQL statement, you can completely analyze what will be added to this statement. Lock? What kind of use risks will there be? Even analyze a deadlock scenario online to understand the cause of the deadlock.

Note: MySQL is a database system that supports plug-in storage engines. All the following introductions in this article are based on the InnoDB storage engine, and the performance of other engines will be quite different.

MVCC：Snapshot Read vs Current Read

The MySQL InnoDB storage engine implements a multi-version-based concurrency control protocol - MVCC ( Multi-Version Concurrency Control ) (Note: As opposed to MVCC, it is lock-based concurrency control, Lock-Based Concurrency Control). The biggest advantage of MVCC, I believe, is also familiar: read without lock, read and write without conflict. In OLTP applications that read more and write less, it is very important that there is no conflict between reading and writing, which greatly increases the concurrent performance of the system. This is why at this stage, almost all RDBMSs support MVCC.

In MVCC concurrency control, read operations can be divided into two categories: snapshot read and current read. Snapshot read, reads the visible version of the record (possibly the historical version) without locking. The current read, reads the latest version of the record, and the record returned by the current read will be locked to ensure that other transactions will not concurrently modify this record.

In a system that supports MVCC concurrency control, which reads are snapshot reads? Which operations are currently read? Take MySQL InnoDB as an example:

Snapshot read: A simple select operation is a snapshot read and does not lock. (Of course, there are exceptions, which will be analyzed below)

select * from table where ?;

Current read: special read operations, insert/update/delete operations, belong to the current read and need to be locked.

All the above statements belong to the current read and read the latest version of the record. Moreover, after reading, it is also necessary to ensure that other concurrent transactions cannot modify the current record and lock the read record. Among them, in addition to the first statement, the S lock (shared lock) is added to the read record, and the X lock (exclusive lock) is added for other operations.

select * from table where ? lock in share mode;
select * from table where ? for update;
insert into table values (…);
update table set ? where ?;
delete from table where ?;

Why are insert/update/delete operations classified as current read? You can take a look at the following update operation, the execution process in the database:

From the figure, you can see the specific process of an Update operation. When Update SQL is sent to MySQL, MySQL Server will read the first record that meets the condition according to the where condition, and then the InnoDB engine will return the first record and lock it (current read). After MySQL Server receives the locked record, it will initiate an Update request to update the record. After a record operation is completed, the next record is read until there are no records that meet the conditions. Therefore, inside the Update operation, a current read is included. Similarly, the Delete operation is the same. The Insert operation is slightly different. In short, the Insert operation may trigger the conflict check of the Unique Key and also perform a current read.

Note: According to the interaction in the above figure, for a currently read SQL statement, the interaction between InnoDB and MySQL Server is carried out one by one, so the locking is also carried out one by one. First lock a record that meets the conditions, return it to MySQL Server, and do some DML operations; then read the next record and lock it until the reading is complete.

Cluster Index: Cluster Index

The data organization method of the InnoDB storage engine is a clustered index table: complete records are stored in the primary key index, and all columns of the record can be obtained through the primary key index. For the organization of clustered index tables, you can refer to MySQL's official documentation: Clustered and Secondary Indexes. This article assumes that the reader has a certain understanding of this, and will not give a specific introduction. In the next part, the two names of primary key index/clustered index will be mixed, so I hope readers will know.

2PL：Two-Phase Locking

A principle of traditional RDBMS locking is 2PL (two-phase lock): Two-Phase Locking. Relatively speaking, 2PL is easier to understand. It means that the lock operation is divided into two stages: the locking stage and the unlocking stage, and it is guaranteed that the locking stage and the unlocking stage do not intersect. Below, still taking MySQL as an example, let's take a brief look at the implementation of 2PL in MySQL.

As can be seen from the above figure, 2PL is to divide the locking/unlocking into two completely disjoint stages. Locking stage: only locks are added, no locks are released. Unlocking stage: only lock, not lock.

Isolation Level

Isolation Level: Isolation Level is also a key feature of RDBMS. I believe that friends who know something about databases have a deep understanding of the four isolation levels: Read Uncommited, Read Committed, Repeatable Read, and Serializable. This article does not intend to discuss how to define the meaning of these four isolation levels in database theory, but to introduce how MySQL/InnoDB defines these four isolation levels.

There are 4 isolation levels defined by MySQL/InnoDB:

Read Uncommited

can read uncommitted records. This isolation level, will not be used, ignored.
Read Committed (RC)

snapshot read ignore, this article does not consider. For the current read, the RC isolation level ensures that the read records are locked (record locks), and there is a phantom read phenomenon.
Repeatable Read (RR)

snapshot reads are ignored and are not considered in this article. For the current read, the RR isolation level ensures that the read record is locked (record lock), and at the same time, the read range is guaranteed to be locked. New records that meet the query conditions cannot be inserted (gap lock), and there is no phantom read. Phenomenon.
Serializable

degenerates from MVCC concurrency control to lock-based concurrency control. There is no difference between snapshot read and current read, all read operations are current read, read plus read lock (S lock), write plus write lock (X lock). Under the Serializable isolation level, the read and write conflicts, so the concurrency drops sharply, and it is not recommended to use it under MySQL/InnoDB.

A Simple SQL Locking Implementation Analysis

After introducing some background knowledge, this article will select several representative examples to analyze MySQL's locking processing in detail. Of course, let's start with the simplest example. Often a friend sends me a SQL, and then asks me, what lock is added to this SQL? Just like the following two simple SQLs, what locks do they add?

SQL1：select * from t1 where id = 10;
SQL2：delete from t1 where id = 10;

How to answer this question? One answer I can imagine is:

SQL1: No lock. Because MySQL uses multi-version concurrency control, reads are not locked.
SQL2: Add a write lock to the record with id = 10 (take the primary key index).

Is this answer correct? Can't tell. That may be true, it may be false, the known conditions are insufficient, and there is no answer to this question. If I were to answer this question, I must also know some of the following premises, and if the premises are different, the answer I can give will be different. What preconditions are missing to answer this question?

Premise 1: Is the id column a primary key?
Premise 2: What is the isolation level of the current system?
Premise 3: If the id column is not the primary key, is there an index on the id column?
Premise 4: If there is a secondary index on the id column, is this index a unique index?
Premise 5: What are the execution plans of the two SQLs? Index scan? Full table scan?

Without these prerequisites, it is very amateurish to directly give a SQL and then ask what locks will be added to this SQL. And when these questions have clear answers, it will be clear at a glance what locks will be added to a given SQL. Below, I will combine the answers to these questions, and then, in order from easy to difficult, will analyze each combination one by one, which locks will be added to the corresponding SQL?

Note: For the following combinations, I made a premise that when there is an index, the execution plan will definitely choose to use the index for filtering (index scan). But the actual situation will be much more complicated. The real execution plan still needs to be based on the output of MySQL.

Combination 1: id column is the primary key, RC isolation level
Combination 2: id column is a secondary unique index, RC isolation level
Combination 3: id column is a secondary non-unique index, RC isolation level
Combination 4: There is no index on the id column, RC isolation level
Combination five: id column is the primary key, RR isolation level
Combination six: id column is a secondary unique index, RR isolation level
Combination 7: id column is a secondary non-unique index, RR isolation level
Combination 8: There is no index on the id column, RR isolation level
Combination nine: Serializable isolation level

The permutations and combinations have not been listed completely, but it seems that there are already many. Is it really necessary to be so complicated? In fact, to analyze locking, it needs to be so complicated. But from another point of view, as long as you select a combination, which locks SQL needs to be added is actually determined. Next, let us analyze the SQL locking strategies under these 9 combinations one by one.

Note: Under the first eight combinations, that is, under the RC and RR isolation levels, SQL1: select operations are not locked, and snapshot reading is used, so they are ignored in the following discussion, mainly discussing SQL2: delete operations lock.

Combination 1: id primary key + RC

This combination is the simplest and easiest to analyze. id is the primary key, Read Committed isolation level, given SQL: delete from t1 where id = 10; Just add X lock to the record with id = 10 on the primary key. As shown below:

Conclusion: When id is the primary key, this SQL only needs to add X lock on the record with id=10.

Combination 2: id unique index + RC

In this combination, id is not the primary key, but a unique secondary index key value. Then under the RC isolation level, what lock does delete from t1 where id = 10; need to be added? See below:

In this combination, the id is the unique index and the primary key is the name column. At this time, the locking situation is different due to combination 1. Since id is a unique index, the delete statement will select the index of the id column to filter the where condition. After finding the record with id=10, it will first add X lock to the id=10 index record on the unique index, and at the same time, According to the read name column, return to the primary key index (clustered index), and then add X lock to the primary key index item corresponding to name = 'd' on the clustered index. Why are the records on the clustered index also locked? Imagine if a concurrent SQL is updated through the primary key index: update t1 set id = 100 where name = 'd'; At this point, if the delete statement does not lock the records on the primary key index, then the concurrent update will The existence of the delete statement will not be perceived, which violates the constraint that updates/deletes on the same record need to be executed serially.

Conclusion: If the id column is a unique column, there is a unique index on it. Then SQL needs to add two X locks, one corresponding to the record with id = 10 on the id unique index, and the other lock corresponding to the record of [name='d', id=10] on the clustered index.

Combination 3: id non-unique index + RC

Compared with combination 1 and 2, combination 3 has changed again, the isolation level is still the same as RC, but the constraint on the id column has been reduced, the id column is no longer unique, and there is only one common index. Suppose the delete from t1 where id = 10; statement still selects the index on the id column to filter the where condition, then what locks will be held at this time? Also see the image below:

According to this figure, it can be seen that, first of all, on the id column index, records that satisfy the query condition of id = 10 have been locked. At the same time, the records on the primary key index corresponding to these records are also locked. The only difference from combination 2 is that combination 2 has at most one record that satisfies the equivalent query, while combination 3 locks all records that meet the query conditions.

Conclusion: If there is a non-unique index on the id column, all the corresponding records that meet the SQL query conditions will be locked. At the same time, these records on the primary key index will also be locked.

Combination 4: id without index + RC

Compared with the previous three combinations, this is a rather special case. There is no index on the id column, where id = 10; this filter condition cannot be filtered by the index, so it can only be filtered by a full table scan. Corresponding to this combination, what lock will SQL add? Or in other words, what locks will be added during a full table scan? There are also many answers to this question: some people say that X locks will be added to the table; some people say that X locks will be added to the records with id = 10; selected on the clustered index. What about the actual situation? Please see the picture below:

Since there is no index on the id column, only the clustered index can be used for full scan. As can be seen from the figure, there are two records that meet the deletion conditions, but all records on the clustered index are added with X locks. Regardless of whether the record meets the conditions, all are locked with X. Neither table locks nor row locks are added to the records that meet the conditions.

Someone might ask? Why not only lock the records that meet the conditions? This is due to the implementation of MySQL. If a condition cannot be quickly filtered by the index, the storage engine layer will lock all records and return them, and then the MySQL Server layer will filter them. Therefore, all records are locked.

Note: In the actual implementation, MySQL has some improvements. After the MySQL Server filter conditions are found to be unsatisfied, the unlock_row method will be called to lock the records that do not meet the conditions (violating the constraints of 2PL). This ensures that only the lock on the record that meets the conditions will be held in the end, but the locking operation of each record cannot be omitted.

Conclusion: If there is no index on the id column, SQL will filter through the full scan of the clustered index, because the filtering is performed at the MySQL Server level. Therefore, each record, whether or not it meets the conditions, will be locked with X. However, for the sake of efficiency, MySQL has been optimized. For records that do not meet the conditions, the lock will be released after judgment. The final hold is the lock on the record that meets the condition, but the lock on the record that does not meet the condition/ The locking action will not be omitted. At the same time, the optimization also violates the constraints of 2PL.

Combination five: id primary key + RR

The above four combinations are all locking behaviors under the Read Committed isolation level, and the next four combinations are the locking behaviors under the Repeatable Read isolation level.

Combination 5, the id column is the primary key column, Repeatable Read isolation level, for delete from t1 where id = 10; this SQL, locking is consistent with combination 1: [id primary key, Read Committed].

Combination six: id unique index + RR

Similar to combination five, the locking of combination six is consistent with combination two: [id unique index, Read Committed]. Two X locks, one for the record whose id unique index satisfies the condition, and one for the record on the corresponding clustered index.

Combination 7: id non-unique index + RR

Remember the difference between the four isolation levels of MySQL mentioned earlier? The RC isolation level allows phantom reads, while the RR isolation level does not allow phantom reads. But in combination 5 and combination 6, the locking behavior is exactly the same as the locking behavior under RC. So under the RR isolation level, how to prevent phantom reads? The answer to the question is revealed in Combination Seven.

Combination 7, Repeatable Read isolation level, there is a non-unique index on id, execute delete from t1 where id = 10; Assuming that the index on the id column is selected for conditional filtering, what is the final locking behavior? Also look at the picture below:

In this figure, compared to the combination three: [non-unique lock on the id column, Read Committed] seems to be the same, but in fact there is a big difference. The biggest difference is that there is an additional GAP lock in this picture, and the GAP lock does not seem to be added to the record, but rather is to load the position between two records. What is the use of the GAP lock?

In fact, this extra GAP lock is the RR isolation level. Compared with the RC isolation level, there is no key to phantom reading. Indeed, the position locked by the GAP lock is not the record itself, but the GAP between the two records. The so-called phantom read is the same transaction, and the current read is performed twice in a row (for example: select * from t1 where id = 10 for update;), then the two current reads return exactly the same records (the number of records is the same, the records itself), the second current read will not return more records (phantoms) than the first.

How to ensure that two current reads return consistent records, then between the first current read and the second current read, other transactions will not insert new records that meet the conditions and commit. In order to realize this function, GAP lock came into being.

As shown in the figure, where are the new items that meet the conditions (id = 10) can be inserted. Considering the ordering of the B+ tree index, the items that meet the conditions must be stored continuously. Before the record [6,c], the record with id=10 will not be inserted;[10, aa] can be inserted between [6,c] and [10,b]; new [10,bb],[10,c] can be inserted between [10,b] and [10,d]; [10,d] and [11,f] can be inserted between [10,e], [10,z], etc. that meet the conditions; and records that meet the conditions will not be inserted after [11,f]. Therefore, in order to ensure that between [6,c] and [10,b], between [10,b] and [10,d], [10,d] and [11,f] will not insert new records that meet the conditions , MySQL chose to use GAP lock to lock these three GAPs.

Insert operations, such as insert [10,aa], will first locate between [6,c] and [10,b], and then check whether the GAP has been locked before inserting. If it is locked, Insert Records cannot be inserted. Therefore, through the current read of the first pass, not only the records that meet the conditions are locked (X lock), which is similar to combination three. At the same time, 3 GAP locks are added, and 3 GAPs that may be inserted into the records that meet the conditions are locked to ensure that subsequent Inserts cannot insert new records with id=10, which also prevents the second current read of the same transaction. Phantom situation.

If you see this, you may ask: Since preventing phantom reading requires the protection of GAP lock, why is combination 5 and combination 6, which are also RR isolation levels, but do not need to add GAP lock?

First, this is a good question. Secondly, the answer to this question is also very simple. The purpose of the GAP lock is to prevent two current reads of the same transaction from occurring phantom reads. In combination five, id is the primary key; in combination six, id is a unique key, both of which can guarantee uniqueness. An equal-value query can only return one record at most, and a new record with the same value will not be newly inserted, thus avoiding the use of GAP locks. In fact, for this problem, there is a more in-depth problem: if you combine five and six, for SQL: select * from t1 where id = 10 for update; the first query, no records that meet the query conditions are found, then Can GAP locks be omitted? This question is left for everyone to think about.

Conclusion: Under the Repeatable Read isolation level, there is a non-unique index on the id column, corresponding to SQL: delete from t1 where id = 10; First, locate the first record that satisfies the query conditions through the id index, and add the X lock on the record , add the GAP lock on the GAP, then add the record X lock on the primary key clustered index, and then return; then read the next one, and repeat. Until the first record [11,f] that does not meet the conditions is reached, at this time, there is no need to add the record X lock, but the GAP lock still needs to be added, and finally returns to the end.

Combination 8: id has no index + RR

Combination eight, the last case under the Repeatable Read isolation level, there is no index on the id column. At this point SQL: delete from t1 where id = 10; There is no other path to choose from, and only a full table scan can be performed. The final locking situation is shown in the following figure:

As the picture shows, this is a very scary phenomenon. First, all records on the clustered index are X-locked. Secondly, the gap (GAP) between each record of the clustered index is also added with a GAP lock. This sample table has only 6 records, a total of 6 record locks and 7 GAP locks are required. Just imagine, what if there are 10 million records in the table?

In this case, on this table, except for the snapshot degree without locking, any other locked concurrent SQL cannot be executed, updated, deleted, or inserted, and the entire table is locked.

Of course, similar to combination four: [ id has no index, Read Committed ], in this case, MySQL has also made some optimizations, which is the so-called semi-consistent read. When semi-consistent read is enabled, MySQL will release locks in advance for records that do not meet the query conditions. For the above use case, except for records [d, 10], [g, 10], all record locks will be released, and GAP locks will not be added. How semi-consistent read is triggered: either read committed isolation level; or Repeatable Read isolation level, and the innodb_locks_unsafe_for_binlog parameter is set. For a more detailed introduction to semi-consistent read, please refer to my previous blog: MySQL+InnoDB semi-consistent read principle and implementation analysis.

Conclusion: Under the Repeatable Read isolation level, if the current read of a full table scan is performed, all records in the table will be locked, and all GAPs in the clustered index will be locked at the same time, preventing all concurrent update/delete/insert operations. . Of course, you can also trigger semi-consistent read to alleviate the locking overhead and concurrency impact, but semi-consistent read itself will also bring other problems and is not recommended.

Combination Nine: Serializable

For the simple SQL mentioned earlier, the last case: Serializable isolation level. For SQL2: delete from t1 where id = 10;, the Serializable isolation level is exactly the same as the Repeatable Read isolation level, so it will not be introduced.

The Serializable isolation level affects SQL1: select * from t1 where id = 10; This SQL, under the RC and RR isolation levels, is a snapshot read without locking. However, at the Serializable isolation level, SQL1 will add a read lock, which means that the snapshot read no longer exists, and the MVCC concurrency control is downgraded to Lock-Based CC.

Conclusion: In MySQL/InnoDB, the so-called read-free lock does not apply to all situations, but is related to the isolation level. Serializable isolation level, read is no longer established without locking, all read operations are current reads.

a complex SQL

Written here, in fact, MySQL's locking implementation has also been introduced. As long as the analysis ideas above are used in this article, most of the SQL can analyze which locks will be added. And here, let's look at a slightly more complicated SQL to illustrate another logic of MySQL locking. The SQL use case is as follows:

As shown in the SQL, what locks will be added? It is assumed that it is under the Repeatable Read isolation level (the locking situation under the Read Committed isolation level is left to the reader to analyze.) At the same time, it is assumed that the SQL uses the idx_t1_pu index.

Before analyzing the locking situation of this SQL in detail, it is necessary to have a knowledge reserve, that is, how to split the where condition in a SQL? For a specific introduction, it is recommended to read my previous article: where conditions in SQL, extraction and application analysis in the database. Here, I directly give the results after analysis:

Index key: pubtime > 1 and puptime < 20. This condition is used to determine the range of SQL queries on the idx_t1_pu index.
Index Filter: userid = 'hdc' . This condition can be filtered on the idx_t1_pu index, but it does not belong to the Index Key.
Table Filter: comment is not NULL. This condition cannot be filtered on the idx_t1_pu index, but only on the clustered index.

After analyzing the composition of the SQL where condition, let's take a look at the locking situation of this SQL (RR isolation level), as shown in the following figure:

As can be seen from the figure, under the Repeatable Read isolation level, the range determined by the Index Key is added with a GAP lock; when the Index Filter locks a given condition (userid = 'hdc') to filter, it depends on the MySQL It depends on the version. Before MySQL version 5.6, Index Condition Pushdown (ICP) is not supported. Therefore, Index Filter filters at the MySQL Server layer. After 5.6, Index Condition Pushdown is supported, and then it is filtered on the index. If ICP is not supported, and records that do not satisfy Index Filter also need to add record X lock, if ICP is supported, records that do not satisfy Index Filter do not need to add record X lock (in the figure, the X lock marked with a red arrow, Whether to add it depends on whether ICP is supported); and the filter conditions corresponding to Table Filter are filtered at the MySQL Server level after being read in the clustered index, so X locks are also required on the clustered index. Finally, a record [8, hdc, d, 5, good] that satisfies the condition is selected, but the number of locks is much larger than the number of records that satisfy the condition.

Conclusion: Under the Repeatable Read isolation level, for a complex SQL, the where condition needs to be extracted first. The range determined by the Index Key needs to be added with GAP lock; the index filter filter condition depends on whether the MySQL version supports ICP, if it supports ICP, the records that do not meet the index filter, do not add X lock, otherwise X lock is required; Table Filter filter condition , no matter whether it is satisfied or not, X lock needs to be added.

Deadlock principle and analysis

The previous part of this article has basically covered all the locking rules of MySQL/InnoDB. In-depth understanding of how MySQL locks has two important functions:

According to the locking rules of MySQL, you can write SQL that will not cause deadlock;
According to the locking rules of MySQL, you can locate the cause of the deadlock on the line;

Let's take a look at two examples of deadlocks (one is two SQLs of two sessions that cause deadlocks; the other is one SQL that is two sessions that cause deadlocks):

The two deadlock use cases above. The first one is very easy to understand and is also the most common deadlock. Each transaction executes two SQLs, each holds a lock, and then adds another lock, resulting in a deadlock.

The second use case, although there is only one statement per session, still results in a deadlock. To analyze this deadlock, you must first use the MySQL locking rules mentioned earlier in this article. For Session 1, starting from the name index, the read [hdc, 1], [hdc, 6] all meet the conditions, not only the record X lock on the name index, but also the record X lock on the cluster index will be added. The locking sequence is [1,hdc,100] first, then [6,hdc,10]. For Session 2, starting from the pubtime index, [10,6], [100,1] all meet the filter conditions, and the record X lock on the cluster index will also be added, and the locking sequence is [6,hdc,10], After [1, hdc, 100]. It is found that there is no, which is exactly the opposite of the locking sequence of Session 1. If both Sessions happen to hold the first lock and request to add the second lock, a deadlock will occur.

Conclusion: The occurrence of deadlock does not depend on the number of SQL statements in the transaction. The key to deadlock is that the order of locking of two (or more) sessions is inconsistent. Using the above mentioned in this article, analyze the locking rules of each SQL statement in MySQL, analyze the locking order of each statement, and then check whether there are locks in the opposite order among multiple concurrent SQLs, you can analyze Various potential deadlock situations can be identified, and the causes of online deadlocks can also be analyzed.

Summarize

At this point, this article also comes to an end and makes a brief summary. To fully grasp the locking rules of MySQL/InnoDB, or even the locking rules of any other database, you need to have the following knowledge points:

Understand some basic theoretical knowledge of database: data storage format (heap-organized table vs clustered index table); concurrency control protocol (MVCC vs Lock-Based CC); Two-Phase Locking; database isolation level definition (Isolation Level);
Understand the execution plan of SQL itself (primary key scan vs unique key scan vs range scan vs full table scan);
Understand some implementation details of the database itself (filter condition extraction; Index Condition Pushdown; Semi-Consistent Read);
Understand the cause of deadlock and the method of analysis (inconsistent locking sequence; analyze the locking sequence of each SQL)

With these knowledge points, plus appropriate practical experience, it is not a problem to fully control the locking rules of MySQL/InnoDB.

Original link: MySQL lock processing analysis

MySQL lock processing analysis [transfer]