solving-dead-locks-three.html

This blog will have some common SQL statements locking analysis, we usually look at the implementation of those will add any SQL lock. Only we write SQL statements locking procedure well known to pour launched in the face of the deadlock problem is what the problem is caused by lock. In the previous blog we have already learned about the different types of lock mode and lock down MySQL, we should pay special attention to their compatibility matrix, which are familiar lock is not compatible, not compatible with these locks is often the culprit leading to deadlock. Overall, MySQL in the lock granularity can be divided into two: the table and row locks, table locks are: table-level read locks, table-level write locks, read locks intent, intent to write locks, increase self-locking; lock line are: reading record locking, record a write lock, the lock gap, Next-key locks, intent lock insert. Not surprisingly, the vast majority of the deadlock problems are caused by the conflict between these locks caused.

We know that different isolation level locking is not the same, for example, there is a gap locks and Next-key locks under RR isolation level, which is not under RC isolation level (there are exceptions), so be locked in for SQL Analysis , you must get to know the isolation level of the database. Since the RR and RC used more, so this blog only these two isolation levels for analysis.

This is the "road to resolve the deadlock" in a blog series, you can also read a few other:

Learning transaction isolation level
Learn about common types of locks
Lock master common analysis SQL statements
Analyze and solve the deadlock problem

First, the basic locking rules

Although MySQL variety of locks, but some basic principles remain the same lock, such as: a snapshot read is not locked, the update statement is definitely plus exclusive locks, RC isolation level is no gap locks and many more. These rules are summarized as follows, will not repeat it later introduced:

Locking common statement
- SELECT ... statement read under normal circumstances is a snapshot, not locked;
- SELECT ... LOCK IN SHARE MODE current read statement, plus S lock;
- SELECT ... FOR UPDATE statement for the current reading, plus X lock;
- Common DML statements (such as INSERT, DELETE, UPDATE) for the current reading, plus X lock;
- Common DDL statements (such as ALTER, CREATE, etc.) plus the table level lock, and the statement is an implicit commit these can not be rolled back;
Table lock
- Table lock (lock points S and X locks)
- Intention locks (IX by sub-IS)
- Increase self-locking (see generally, if only innodb_autoinc_lock_mode = 0 or Bulk inserts may be)
Row lock
- Record locking (lock points S and X locks)
- Gap lock (lock points S and X locks)
- Next-key lock (lock points S and X locks)
- Insert intent locks
Row lock analysis
- Row locks are added to the index, and eventually falls on the clustered index;
- Add row lock is a process of adding a record;
Lock conflicts
- S S lock and compatibility lock, X and X locks lock conflicts, X S lock and lock conflicts;
- Table and row locks conflict matrix see previous blog Learn about common types of locks ;
Locks at different isolation levels
- It says SELECT ... statement read under normal circumstances is a snapshot, not locked; but the current reading in the Serializable isolation level, plus S lock;
- There is no gap and Next-key lock latch (special case there will be: purge + unique key) under the isolation level RC;
- Lock difference under different isolation levels, see previous blog learning transaction isolation level ;

Locking analysis Second, the simple SQL

It Dengcheng predecessors in his blog "MySQL lock processing and analysis" in some common SQL locking conducted a detailed analysis of this blog can be said that online MySQL introduce the analysis of a model lock, lock analysis on almost all online the blog is a reference to this blog survey said the classic, highly recommended. I have here is no exception, but had some sort and summarize his basis.

We use this as an example of students the following table, where id primary key, no (Student ID) for the two unique index, name (names) and age (the age of) for the two non-unique index, score (credit) no index.

In this section we analyze the simplest kind of SQL, it contains a WHERE condition, the equivalent range queries or queries. Although SQL is very simple, but for different types of columns, we will still face a variety of situations:

Clustered indexes, a query hit: UPDATE students SET score = 100 WHERE id = 15;
Clustered indexes, a query miss: UPDATE students SET score = 100 WHERE id = 16;
Two unique index, the query hit: UPDATE students SET score = 100 WHERE no = 'S0003';
The only two index, query miss: UPDATE students SET score = 100 WHERE no = 'S0008';
Two non-unique index, the query hit: UPDATE students SET score = 100 WHERE name = 'Tom';
Two non-unique index, a query miss: UPDATE students SET score = 100 WHERE name = 'John';
无索引：UPDATE students SET score = 100 WHERE score = 22;
Clustered index range query: UPDATE students SET score = 100 WHERE id <= 20;
Two index range query: UPDATE students SET score = 100 WHERE age <= 23;
Modify the index values: UPDATE students SET name = 'John' WHERE id = 15;

2.1 clustered index, query hit

Statement UPDATE students SET score = 100 WHERE id = 15 lock as in the case of RC and RR isolation levels, this is a clustered index on id plus X lock, as follows:

2.2 clustered index, query misses

If the query misses record locking is not the same in RC and RR isolation level, because there are GAP RR lock. Statement UPDATE students SET score = 100 WHERE id = 16 lock case in RC and RR isolation levels as follows (RC unlocked):

2.3 two unique index, query hit

Statement UPDATE students SET score = 100 WHERE no = 'S0003' hit two unique index on a blog, we introduced the structure of the index, we know that two leaf nodes of the index holds the position of the primary key index, when locked to the secondary index, primary key index will be a and locked. This is no difference in the two kinds of RC and RR isolation levels:

So why record on the primary key index have locked it? Because there may be other transactions will be updated according to the primary key table of students, such as: UPDATE students SET score = 100 WHERE id = 20Imagine if the primary key index is not locked, then obviously there will be concurrent problems.

2.4 two unique index, query misses

If the query does not hit a record, and 2.2 the same situation, RR isolation level will increase GAP lock, RC no lock. Statement UPDATE students SET score = 100 WHERE no = 'S0008' lock as follows:

In this case only two locking index, not clustered index on the lock.

2.5 two non-unique index, query hit

If the query hit is the second non-unique index, in the RR isolation level, but also add GAP lock. Statement UPDATE students SET score = 100 WHERE name = 'Tom'lock as follows:

Why non-unique index will increase GAP lock, while the unique index do not add GAP lock it? The reason is simple, the effect of GAP lock to solve the phantom read, preventing other transactions from inserting records with the same value of the index, the only index and primary key constraints have to ensure that the index value is certainly only one record, so no need to add GAP lock.

Another point to note here, count on the right of the lock you may feel added a total of seven locks, the actual situation is not, to be noted that (Tom, 37) record lock on and lock it in front of the GAP together are a Next-key lock that added (Tom, 37) on this index, there is also a lock on the Next-key (Tom, 49). Then the rightmost GAP plus lock in what it? The right has no record ah. In fact, in InnoDb storage engine, each data page will have two virtual rows, used to define the record boundary, namely: Infimum Record and Supremum Record, Infimum are no records to be smaller than the value of the page, and Supremum the maximum recorded values bigger than the page, which will have two records, and will not be deleted when you create the page. GAP lock on the right above is added to the Supremum Record. Therefore, a total of two right above Next-key locks, a GAP locks, record locks 2, a total of five lock.

2.6 two non-unique index, query misses

If the query does not hit a record, and 2.2, 2.4, like the situation, RR isolation level will increase GAP lock, RC no lock. Statement UPDATE students SET score = 100 WHERE name = 'John' lock as follows:

2.7 No Index

If the WHERE condition can not take the index, MySQL will be how to lock it? Some people say that will add X lock on the table, it was said that it would be screened in accordance with WHERE condition records plus X lock on the clustered index, then exactly how we look:

In the absence of an index, can only take a clustered index, the records table full table scan. In isolation level will give RC plus all records row lock, in the RR isolation level, not only will add all the records row lock, the lock will be added GAP between all clustered indexes and clustered indexes.

Statements UPDATE students SET score = 100 WHERE score = 22 meet the conditions, although only one record, but all the records on the clustered index, have been added X lock. So why not just lock it on the record satisfies the conditions? This is due to the implementation of the decision of MySQL. If a condition can not be quickly filtered through an index level storage engine will return all records after the lock, and then filtered by the MySQL Server layer, and therefore put all the records are locked.

However, in actual implementation, MySQL there are some improvements, if the isolation level is RC filter condition found not satisfied, calls unlock_row method, the record does not satisfy the conditions of the release lock out (contrary to the constraints of 2PL) in MySQL Server. This will ensure that the final will meet the locks held on the recording conditions, but still locking operation of each record can not be omitted. If RR isolation level, under normal circumstances is not so optimized MySQL, unless the set innodb_locks_unsafe_for_binlog parameters, then the lock will be released ahead of time, and without GAP lock, which is called the semi-consistent read , on semi-consistent read can reference here .

2.8 clustered index, the scope of inquiry

Each case presented above are actually very common in SQL, they have a characteristic: all only a WHERE condition, and are equivalent queries. So the question is, if not the equivalent query but range queries, lock what would happen? One might think this is very simple, according to the above lock experience, we just give all the records within the scope of the query plus lock can be, if the isolation level is RR, plus the gap between the lock all records. The fact exactly how we look at the following chart:

SQL statement UPDATE students SET score = 100 WHERE id <= 20, and normally we only need to id = 20,18,15 three records can be locked, but see diagram on the right, under the RR isolation level, we also id = 30 this record as well (20, 30 the gap between] also locked up, it is clear that this is a Next-key locks. If the WHERE condition is id <20, this will be the id = 20 record locked. I do not know why this is so, the internet search for a long time, it was said that in order to prevent phantom read, but id is a unique primary key, (20, 30] is no longer possible to insert a id = 20, so still need specific reasons under analysis, if you know, but also please let us know.

So for range queries, with the proviso that if the WHERE id <= N, then the recording will be coupled with a Next-key locks after N; with the proviso that if id <N, then this record is N plus Next-key locks. Further, with the proviso that if the WHERE id> = N, N only to add record locks, and to lock greater than N records, a record is not locked to the former N; with the proviso that if id> N, not previous record locks, N even this record will not lock.

====================== No. 26 November complement ======================= ==

I was doing experiments, we found that the RR isolation level, that id> = 20, would sometimes record id <20 lock, sometimes not increase, the feeling can not find any law, subject to the actual situation . I do not quite understand the principle of locking range queries, followed by time to study carefully the next, and also welcome interested students together under discussion.

Here is a simple experiment I did, very simple table, only one primary key id:

mysql> show create table t1;

+-------+--------------------------------------------+

| Table | Create Table |

+-------+--------------------------------------------+

| t1 | CREATE TABLE `t1` ( |

| | `id` int(11) NOT NULL AUTO_INCREMENT, |

| | PRIMARY KEY (`id`) |

| | ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

+-------+--------------------------------------------+

A total of three data table:

mysql> select * from t1;

+----+

| id |

+----+

| 2 |

| 4 |

| 6 |

+----+

3 rows in set (0.00 sec)

Performs delete from t1 where id > 2 locking when the case is: (2, 4], ( 4, 6], (6, + ∞)
performs select * from t1 where id > 2 for update locking when the case is: (- ∞, 2], (2, 4], (4, 6] , (6, + ∞)
visible select for update and delete locked or differentiated, as to select for update Why add (-∞, 2] the lock, I was baffled. later inadvertently to the table t1 added a field a int (11) nOT NULL, even found select * from t1 where id > 2 for update not give (-∞, 2] locked up, really strange.

====================== December 3 Supplement ======================= ==

After several days of searching, finally found a decent explanation (but not to prove): When the data in the data table is very low, such as the example above, select ... [lock in share mode | for update] statement take full table scan, so that all records in the table will be locked, which is (-∞, 2] reasons locked while the delete statement and will not take a full table scan.

2.9 secondary indexes, range queries

We then applied the range query to two non-unique index up, SQL statements, as: UPDATE students SET score = 100 WHERE age <= 23, lock case as shown below:

As can be seen, and cluster index as range queries, in addition to the WHERE condition record locking range, will add back a record Next-key lock, where interesting to note that, despite the record age = 24 are satisfied two, but only the first is locked, the second is not locked, and between the first and second locking not.

2.10 modify the index value

This situation is more easily understood, the index WHERE part of the locking principle and as described above, and more are locked SET section. For example UPDATE students SET name = 'John' WHERE id = 15 , not only in the id = 15 record locking addition, also in the name = 'Bob' (original value), and name = 'John' on the lock (new value). Diagram below (<span style = 'color: red;'> misunderstood here, see the following comments section </ span>):

RC and RR no difference.

Third, the lock complex condition analysis

The preceding examples are very simple SQL, it contains only a WHERE condition, and is the equivalent of a query, when the SQL statement contains multiple conditions, the analysis of the index is quite important. Because we know the final row locks are added to the index, which index will be used to execute SQL statements even if we do not know, how to analyze the SQL added lock it?

MySQL indexes is a very complex topic, and even write a book came out. Here just to learn how to analyze the index before locking the analysis of complex SQL. For example, such as the following SQL:

1	`mysql> DELETE FROM students WHERE name = 'Tom' AND age = 22;`

Name and age in which two fields are indexed, then how to lock? It really depends on which index MySQL use. You can use EXPLAIN command analysis is how MySQL executes the SQL by this command can know which indexes MySQL will use the index and how to execute SQL, only the execution will be used in the index is likely to be locked, unused the index is not locked, there is an EXPLAIN the blog for reference. You can also use the MySQL optimizer_trace function to analyze the SQL, which supports SQL query execution plan tree of the record, this a little bit more difficult, interested students can study under. So MySQL is how to choose the right index it? In fact, MySQL will give each index as an indicator, called the selective index, the higher the value of this index is able to filter more records maximum, about this, but that's another story.

Of course, choose from two index to use an index, lock analysis of this situation and the one we discussed the situation did not essentially different, simply as that does not filter ordinary conditions with the index on the WHERE condition All right. Here we will use the index is called Index Key , and another condition called the Table the Filter . For example, if the index used here is age, so age is the Index Key, and name = 'Tom' is the Table Filter. Index Key is divided into First Key and the Last Key, if Index Key is the scope of the query words, as in the following example:

1	`mysql> DELETE FROM students WHERE name = 'Tom' AND age > 22 AND age < 25;`

First Key which is age> 22, Last Key to age <25.

So when we locked analysis can only determine the Index Key lock is applied to the recording between the First Key and the Last Key if the isolation level RR, there is also a gap lock. Note that, when the index is a composite index, Index Key may be multiple, He Dengcheng of this blog "where conditions SQL, database and extract in the Application of" details how from a complex WHERE conditions extracted Index Key, recommended reading. Here also one blog describes how MySQL uses the index.

When the index is a composite index, not only may have multiple Index Key , but may also have Index the Filter . The so-called Index Filter, except that a composite index Index Key other filtration conditions may be used. If MySQL version prior to 5.6, Index Filter and Table Filter no difference, all the records in the index Index First Key and Index Last Key range, back to the table to read a complete record, and then return to the MySQL Server filtering layer. And after MySQL 5.6, Index Filter and Table Filter separation, Index Filter drop in the index level of InnoDB filtering, reducing the back table records the interactive overhead and return MySQL Server layer, improve the efficiency of SQL, which is the legendary the ICP (Index for condition condition Pushdown) , using the filter condition is not satisfied Index filter recording, without locking.

One example cited here (He Dengcheng predecessors blog of Source ):

See pubtime> 1 and pubtime <20 and for the Index First Key Index Last Key, MySQL will add record lock and lock the gap within this range; userid = 'hdc' is the Index Filter, this filter criteria may be in the index level can filter out a record, so if the database supports ICP words, (4, yyy, 3) this record will not be locked; comment is not NULL for the Table filter, although this condition can also filter a record, but it can not filtered index level, but after reading the entire record according to the index it was filtered, and thus locking can not be omitted.

Four, DELETE statement lock analysis

In general, DELETE and SELECT FOR UPDATE of locking or UPDATE is not much difference, DELETE statements as these have the following situations:

Clustered index, the query hit: DELETE FROM students WHERE id = 15;
Clustered index, query miss: DELETE FROM students WHERE id = 16;
Two unique index, the query hit: DELETE FROM students WHERE no = 'S0003';
The only two index, query miss: DELETE FROM students WHERE no = 'S0008';
Two non-unique index, the query hit: DELETE FROM students WHERE name = 'Tom';
Two non-unique index, a query miss: DELETE FROM students WHERE name = 'John';
No index: DELETE FROM students WHERE score = 22;
Clustered index range query: DELETE FROM students WHERE id <= 20;
Two index range query: DELETE FROM students WHERE age <= 23;

For analysis of these cases and lock above the same, not repeat them here.

Then lock DELETE statements, and UPDATE statements in the end what difference will it make? We know that in MySQL database, execute DELETE statement did not actually delete records, but marked with a deletion mark on the record, then called the purge by a background thread to clean up. From this point of view, DELETE, and UPDATE indeed very similar. Indeed lock analysis, DELETE and UPDATE lock is almost the same, here to add a separate section to illustrate the DELETE statement, in fact, not because locking and other statements DELETE statement is different, but because DELETE statement causes more of a particular type of record: marked for deletion record for this type of record, its locking mechanism for locking and other records are not the same. So the title of this section is called marked as deleted records are locked analysis may be more appropriate.

So the question again: Under what circumstances would that have been marked as deleted records locked it? I conclude there are two issues: After blocking lock and a snapshot reading lock (own made name), the following were introduced.

After blocking lock as shown below, a transaction id = 18 A delete this record, and the transaction id = 18 B delete this record, it is clear that, id primary key, DELETE statements need to acquire X record locks, transaction B blocking. After submission of transaction A, id = 18 this record is marked as deleted, then transaction B needs to be locked to the deleted records.

Reading snapshot lock as shown below, a transaction id = 18 A delete this record and submit. A B transaction before the transaction is submitted once read snapshot id = 18, so delete id = 18 behind this record when you need to lock the deleted records. If this is not the beginning of a transaction read the snapshot, DELETE statement is simply to delete a record that does not exist.

Note that the above transaction B is not limited to DELETE, UPDATE or SELECT FOR UPDATE replaced by equally applicable. Locking online analysis of this deletion record is not much, through my own experiments, has been following these conclusions, if not the right place, welcome treatise. (Experimental environment, MySQL version: 5.7, isolation levels: RR)

Delete records clustered index
- After blocking Lock: X plus record locking (rec but not gap) on deleting records, and after deleting a record lock plus a gap (gap before rec)
- Snapshot lock reading: X plus record locking (rec but not gap) on deleting records
Delete recorded as a secondary index (index unique and non-unique indexes are applicable)
- After blocking Lock: add Next-key locks on deleting records, and a record lock plus a gap after deletion
- Snapshot lock reading: add Next-key locks on deleting records, and a record lock plus a gap after deletion

It should be noted that, where the isolation level RR, if the isolation level in the RC, the locking procedure should be different, interested students can experiment on their own. Lock on DELETE statement, He Dengcheng predecessors in his blog: One of the most incredible MySQL deadlock analysis which has a detailed analysis, and introduces the concept of page locks, restored with only a DELETE statement will cause deadlock the whole process, said very exciting.

Five, INSERT statement lock analysis

Locking analysis mentioned above, are for SELECT FOR UPDATE, UPDATE, DELETE, etc., then the locking procedure for INSERT and how is it? We continue to explore below.

Or to experiment with the students table, for example, we execute the following SQL:

1	`mysql> insert into students(no, name, age, score) value('S0008', 'John', 26, 87);`

Then we use the show engine innodb status\G query transaction locks situation:

---TRANSACTION 3774, ACTIVE 2 sec

1 lock struct(s), heap size 1136, 0 row lock(s), undo log entries 1

MySQL thread id 150, OS thread handle 10420, query id 3125 localhost ::1 root

TABLE LOCK table `sys`.`t3` trx id 3774 lock mode IX

我们发现除了一个 IX 的 TABLE LOCK 之外，就没有其他的锁了，难道 INSERT 不加锁？一般来说，加锁都是对表中已有的记录进行加锁，而 INSERT 语句是插入一条新的纪录，这条记录表中本来就没有，那是不是就不需要加锁了？显然不是，至少有两个原因可以说明 INSERT 加了锁：

为了防止幻读，如果记录之间加有 GAP 锁，此时不能 INSERT；
如果 INSERT 的记录和已有记录造成唯一键冲突，此时不能 INSERT；

要解决这两个问题，都是靠锁来解决的（第一个加插入意向锁，第二个加 S 锁进行当前读），只是在 INSERT 的时候如果没有出现这两种情况，那么锁就是隐式的，只是我们看不到而已。这里我们不得不提一个概念叫 隐式锁（Implicit Lock），它对我们分析 INSERT 语句的加锁过程至关重要。

关于隐式锁，这篇文章《MySQL数据库InnoDB存储引擎中的锁机制》对其做了详细的说明，讲的非常清楚，推荐一读。可以参考上一篇介绍的悲观锁和乐观锁。

锁是一种悲观的顺序化机制，它假设很可能发生冲突，因此在操作数据时，就加锁，如果冲突的可能性很小，多数的锁都是不必要的。Innodb 实现了一个延迟加锁的机制来减少加锁的数量，这被称为隐式锁。

隐式锁中有个重要的元素：事务ID（trx_id）。隐式锁的逻辑过程如下：

A. InnoDB 的每条记录中都有一个隐含的 trx_id 字段，这个字段存在于簇索引的 B+Tree 中;
B. 在操作一条记录前，首先根据记录中的 trx_id 检查该事务是否是活动的事务（未提交或回滚），如果是活动的事务，首先将隐式锁转换为显式锁（就是为该事务添加一个锁）；
C. 检查是否有锁冲突，如果有冲突，创建锁，并设置为 waiting 状态；如果没有冲突不加锁，跳到 E；
D. 等待加锁成功，被唤醒，或者超时；
E. 写数据，并将自己的 trx_id 写入 trx_id 字段。

隐式锁的特点是只有在可能发生冲突时才加锁，减少了锁的数量。另外，隐式锁是针对被修改的 B+Tree 记录，因此都是 Record 类型的锁，不可能是 Gap 或 Next-Key 类型。

INSERT 操作只加隐式锁，不需要显示加锁；

UPDATE、DELETE 在查询时，直接对查询用的 Index 和主键使用显示锁，其他索引上使用隐式锁。

理论上说，可以对主键使用隐式锁的。提前使用显示锁应该是为了减少死锁的可能性。INSERT，UPDATE，DELETE 对 B+Tree 们的操作都是从主键的 B+Tree 开始，因此对主键加锁可以有效的阻止死锁。

INSERT 加锁流程如下（参考）：

首先对插入的间隙加插入意向锁（Insert Intension Locks）
- 如果该间隙已被加上了 GAP 锁或 Next-Key 锁，则加锁失败进入等待；
- 如果没有，则加锁成功，表示可以插入；
然后判断插入记录是否有唯一键，如果有，则进行唯一性约束检查
- 如果不存在相同键值，则完成插入
- 如果存在相同键值，则判断该键值是否加锁
  - 如果没有锁，判断该记录是否被标记为删除
    - 如果标记为删除，说明事务已经提交，还没来得及 purge，这时加 S 锁等待；
    - If not marked for deletion, 1062 duplicate key error is reported;
  - If there is a lock, indicating that the record is being processed (add, delete or update), and the transaction has not been submitted, plus S lock wait;
Inserting records and record locking recording plus X;

The expression here is actually not accurate, interested students can go to read the source code analysis INSERT statement InnoDb specific locking procedure, I "read MySQL source code look INSERT locking process," a detailed description of this blog.

Locking common analysis SQL statement - to resolve the deadlock of the road

Transfer: https://www.aneasystone.com/archives/2017/12/solving-dead-locks-three.html

First, the basic locking rules

Locking analysis Second, the simple SQL

2.1 clustered index, query hit

2.2 clustered index, query misses

2.3 two unique index, query hit

2.4 two unique index, query misses

2.5 two non-unique index, query hit

2.6 two non-unique index, query misses

2.7 No Index

2.8 clustered index, the scope of inquiry

2.9 secondary indexes, range queries

2.10 modify the index value

Third, the lock complex condition analysis

Four, DELETE statement lock analysis

Five, INSERT statement lock analysis

reference

Guess you like