This article analyzes deadlocks in several scenarios of INSERT and its variants (REPLACE/INSERT ON DUPLICATE KEY UPDATE) and how to avoid them.

Author: Zhang Luodan, DBA database technology enthusiast~

Produced by the Aikeson open source community. Original content may not be used without authorization. Please contact the editor and indicate the source for reprinting.

This article has a total of 3200 words and is expected to take 10 minutes to read.

Said it in front

This article analyzes deadlocks in several scenarios of INSERTand its variants ( ) and how to avoid them:REPLACE/INSERT ON DUPLICATE KEY UPDATE

Scenario 1: INSERT unique key conflict
Scenario 2/3: REPLACE INTOUnique key conflict (from online business)
Scenario 4: INSERTPrimary key conflict (from official case)

In fact, if you Google it, there will be a lot of articles like this. This article only analyzes a few scenarios, but if you go through it, you INSERTwill have a grasp of the locking situation and how to cause deadlock. My personal ability is limited. If there are errors and omissions in the content of the article, you are welcome to point it out.

If you are interested, please continue reading~

Review row locks

Before that, let's briefly review the row lock types in InnoDB.

RECORD LOCK

Lock the index record.

Gap LOCK (also called range lock)

Locking the gap where the index record is located is used to solve the problem of phantom reading under the RR isolation level (in fact, under the RC isolation level, gap locks will also be generated).

S gap locks and X gap locks are compatible, and different transactions can lock in the same gap.

NEXT-KEY lock

Equivalent to RECORD LOCK + GAP LOCK.

INSERT INTENTION LOCK

A type of GAP lock. Before execution INSERT, if the next record to be inserted is locked with a GAP lock, the INSERTstatement will be blocked and an insertion intention lock will be generated.

Will only be blocked by GAP lock.

implicit lock

The newly inserted record does not generate a lock structure, but due to the existence of the transaction ID, it is equivalent to adding an implicit lock; before other transactions want to lock this record, they must first help it generate a lock structure, and then enter the waiting state. .

The key to deadlock here is the GAP lock. The GAP lock is used to solve the phantom read problem under the RR isolation level, but it is also used in duplicate key checks and foreign key checks under the RC isolation level.

Let’s briefly review INSERTthe statement locking types:

When blocked by a GAP lock, an insertion intention lock is generated.
When encountering a duplicate key conflict
- Primary key conflict generates S-type record lock (RR and RR isolation levels, in fact, GAP lock will still be requested during the INSERT phase ).
- Unique key conflict, resulting in S-type NEXT-KEY lock (RR and RR isolation levels).

Note: INSERTWhen the statement is executed normally, no lock structure will be generated.

Also, it's slightly different for INSERT ... ON DUPLICATE KEY UPDATEand :REPLACE

Different types of locks

INSERT ... ON DUPLICATE KEY UPDATEand REPLACEif a duplicate key conflict is encountered.

If there is a primary key conflict, add X-type record lock (RR and RR isolation levels, in fact INSERT, GAP lock will still be requested during the stage ).
If it is a unique key conflict, add an X-type NEXT-KEY lock (RR and RR isolation levels).

Lock range is different

INSERTand add NEXT-KEY lock on INSERT ... ON DUPLICATE KEY UPDATEthe inserted or UPDATErow.
REPLACEWhen adding NEXT-KEY lock, REPLACENEXT-KEY lock will be added to the record and the next record.

This is somewhat different from the official document description. As shown below, the official only said that REPLACENEXT-KEY lock will be added to the line being blocked, but after testing, NEXT-KEY lock will also be added to the next line. See the scenario below for details.

Finally, let’s briefly review the conditions and observation methods for deadlock:

Deadlock conditions

Two or more transactions wait for each other's locks and hold the locks required by the other party, resulting in cyclic waiting.

Deadlock observation method

performance_schema.data_locksView the lock structure information generated by the session.

SELECT ENGINE_TRANSACTION_ID, OBJECT_NAME, INDEX_NAME, LOCK_TYPE, LOCK_MODE, LOCK_STATUS, LOCK_DATA FROM performance_schema.data_locks;

show engine innodb statusView deadlock information.

Officially begin

Before we officially start, we still need to talk about some basic environmental information:

MySQL 8.0.32
transaction_isolation：READ-COMMITTED

Prepare data

These are the initial data for each case.

DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (
    id INT NOT NULL AUTO_INCREMENT,
    a INT NULL,
    b INT NULL,
    PRIMARY KEY (id),
    UNIQUE INDEX uk_a (a ASC)
);
INSERT INTO t1 (id, a, b) VALUES (1, 10, 0);
INSERT INTO t1 (id, a, b) VALUES (2, 20, 0);
INSERT INTO t1 (id, a, b) VALUES (3, 30, 0);
INSERT INTO t1 (id, a, b) VALUES (4, 40, 0);
INSERT INTO t1 (id, a, b) VALUES (5, 50, 0);

scene one

time	session1	session2
T1	BEGIN; INSERT INTO t1(a,b) VALUES (35,0);
T2		BEGIN; INSERT INTO t1(a,b) VALUES (35,0); --Blocked
T3	INSERT INTO t1(a,b) VALUES (33,0)
T4		DEADLOCK

The lock status held at different times is as follows:

Note: The schematic diagram only shows the lock on the unique index we analyzed. In fact, after locking the unique index, record locks will also be added to the corresponding clustered index, but the primary key index will not be reflected here. Below same.

Process explanation

T1 time

The record insertion in session1 is successful. At this time, the corresponding index record is protected by an implicit lock and no lock structure is generated.

T2 time

Inserting records in session2 detects a conflict between the inserted value and session1's unique key.

session2 helps session1 generate an explicit lock structure for the record a=35.
Session2 itself generates an S-type NEXT-KEY LOCK, and the request range is (30,35], but it can only obtain the GAP LOCK of (30,35), and is blocked by the record lock of a=35 of session1.

mysql> SELECT ENGINE_TRANSACTION_ID, OBJECT_NAME, INDEX_NAME, LOCK_TYPE, LOCK_MODE, LOCK_STATUS, LOCK_DATA FROM performance_schema.data_locks;
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
| ENGINE_TRANSACTION_ID | OBJECT_NAME | INDEX_NAME | LOCK_TYPE | LOCK_MODE     | LOCK_STATUS | LOCK_DATA |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
|               xxxxxx2 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx2 | t1          | uk_a       | RECORD    | S             | WAITING     | 35, 7     |
|               xxxxxx1 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx1 | t1          | uk_a       | RECORD    | X,REC_NOT_GAP | GRANTED     | 35, 7     |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
4 rows in set (0.01 sec)

T3 time

Session1 inserts a=33, which is blocked by the session2 (30,35) gap lock.

At this point, a closed-loop lock wait is formed, and the deadlock condition is reached:

Session1 holds the a=35 record lock required by session2, and requests the (30,35) GAP lock held by session2.
session2 holds the (30,35) GAP lock required by session1, and requests the record lock held by session1.

Below is the printed deadlock log.

How to avoid deadlock in this scenario:

In a transaction, INSERTinsert in ascending order of the primary key or unique key, that is, session1 can insert the record a=33 first, and then insert the record a=35, which can avoid the impact of GAP lock to a certain extent.
Only one row of records is inserted in a transaction and it is committed as soon as possible.

Scene 2

time	session1	session2	session3
T1	BEGIN; REPLACE INTO t1 (a, b) VALUES (40, 1);
T2		BEGIN; REPLACE INTO t1 (a, b) VALUES (30, 1); -- blocked
T3			BEGIN; REPLACE INTO t1 (a, b) VALUES (40, 1); -- blocked
T4	COMMIT;
T5		2 rows affected;	DEADLOCK,ROLLBACK;

The lock status held at different times is as follows:

Process explanation

T1 time

session1 detects a unique key conflict, and REPLACEadds an X-type NEXT-KEY lock to the record and the next record, that is, the lock range is (30,40], (40,50].

Note: This is INSERTdistinguished from . INSERTWhen encountering a unique key conflict and being blocked, the NEXT-KEY lock is added to the inserted record. Here REPLACEis the NEXT-KEY lock added to the inserted record and the next record (the official document description seems to be lacking. appropriate).

lock situation

mysql> SELECT ENGINE_TRANSACTION_ID, OBJECT_NAME, INDEX_NAME, LOCK_TYPE, LOCK_MODE, LOCK_STATUS, LOCK_DATA FROM performance_schema.data_locks;
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
| ENGINE_TRANSACTION_ID | OBJECT_NAME | INDEX_NAME | LOCK_TYPE | LOCK_MODE     | LOCK_STATUS | LOCK_DATA |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
|               xxxxxx1| t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx1| t1          | uk_a       | RECORD    | X             | GRANTED     | 40, 4     |
|               xxxxxx1| t1          | uk_a       | RECORD    | X             | GRANTED     | 50, 5     |
|               xxxxxx1| t1          | PRIMARY    | RECORD    | X,REC_NOT_GAP | GRANTED     | 4         |
|               xxxxxx1| t1          | uk_a       | RECORD    | X,GAP         | GRANTED     | 40, 10    |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
5 rows in set (0.00 sec)

T2 time

Session2 encounters a unique key conflict, REPLACEand adds an ) The locking is successful, but it is waiting for the record lock of session1 a=40.

mysql> SELECT ENGINE_TRANSACTION_ID, OBJECT_NAME, INDEX_NAME, LOCK_TYPE, LOCK_MODE, LOCK_STATUS, LOCK_DATA FROM performance_schema.data_locks;
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
| ENGINE_TRANSACTION_ID | OBJECT_NAME | INDEX_NAME | LOCK_TYPE | LOCK_MODE     | LOCK_STATUS | LOCK_DATA |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
|               xxxxxx2 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx2 | t1          | uk_a       | RECORD    | X             | GRANTED     | 30, 3     |
|               xxxxxx2 | t1          | PRIMARY    | RECORD    | X,REC_NOT_GAP | GRANTED     | 3         |
|               xxxxxx2 | t1          | uk_a       | RECORD    | X             | WAITING     | 40, 4     |
|               xxxxxx1 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx1 | t1          | uk_a       | RECORD    | X             | GRANTED     | 40, 4     |
|               xxxxxx1 | t1          | uk_a       | RECORD    | X             | GRANTED     | 50, 5     |
|               xxxxxx1 | t1          | PRIMARY    | RECORD    | X,REC_NOT_GAP | GRANTED     | 4         |
|               xxxxxx1 | t1          | uk_a       | RECORD    | X,GAP         | GRANTED     | 40, 10    |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
9 rows in set (0.00 sec)

T3 time

The lock type requested by session3 is the same as session1, and the lock range is (30,40], (40,50]. When acquiring the (30,40] NEXT-KEY lock, only the (30,40) GAP lock was acquired, waiting for session1 Record lock with a=40.

Note: No lock has been added to (40, 50] here. InnoDB row locks are acquired row by row. If they cannot be acquired, they will be blocked.

lock situation

mysql> SELECT ENGINE_TRANSACTION_ID, OBJECT_NAME, INDEX_NAME, LOCK_TYPE, LOCK_MODE, LOCK_STATUS, LOCK_DATA FROM performance_schema.data_locks;
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
| ENGINE_TRANSACTION_ID | OBJECT_NAME | INDEX_NAME | LOCK_TYPE | LOCK_MODE     | LOCK_STATUS | LOCK_DATA |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
|               xxxxxx3 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx3 | t1          | uk_a       | RECORD    | X             | WAITING     | 40, 4     |
|               xxxxxx2 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx2 | t1          | uk_a       | RECORD    | X             | GRANTED     | 30, 3     |
|               xxxxxx2 | t1          | PRIMARY    | RECORD    | X,REC_NOT_GAP | GRANTED     | 3         |
|               xxxxxx2 | t1          | uk_a       | RECORD    | X             | WAITING     | 40, 4     |
|               xxxxxx1 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx1 | t1          | uk_a       | RECORD    | X             | GRANTED     | 40, 4     |
|               xxxxxx1 | t1          | uk_a       | RECORD    | X             | GRANTED     | 50, 5     |
|               xxxxxx1 | t1          | PRIMARY    | RECORD    | X,REC_NOT_GAP | GRANTED     | 4         |
|               xxxxxx1 | t1          | uk_a       | RECORD    | X,GAP         | GRANTED     | 40, 10    |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
11 rows in set (0.01 sec)

T4 time

After session1 is submitted, the lock held is released.
session2 acquires the record lock of a=40. At this point, the locks held by session2 are (20,30], (30,40] NEXT-KEY locks; after session2 acquires the lock, it performs the insertion operation. Since the insertion gap is ( 20,40), is blocked by the (30,40) GAP lock of session3, generates an insertion intention lock, and enters the waiting state.

At this point, a closed-loop lock wait is formed, and the deadlock condition is reached:

Session2 holds the (20,30], (30,40] NEXT-KEY lock, and requests to insert the intention lock, which is blocked by the (30,40) GAP lock of session3.
Session3 holds the (30,40) GAP lock blocking session2, and requests the a=40 record lock held by session2.

Below is the printed deadlock log.

Scene three

time	session1	session2	session3
T1	BEGIN; SELECT * FROM t1 WHERE a=40 for UPDATE;
T2		BEGIN; REPLACE INTO t1 (a, b) VALUES (30, 1);-- blocked
T3			BEGIN; REPLACE INTO t1 (a, b) VALUES (40, 1); -- blocked
T4	COMMIT;
T5		2 rows affected;	DEADLOCK,ROLLBACK;

The lock status held at different times is as follows:

This scenario is basically the same as the deadlock situation in scenario 2, except that the lock type held by session1 is different, so I will not explain them one by one.

Below is the printed deadlock log.

How to avoid deadlocks in scenarios 2 and 3?

From the previous analysis, we can see that when a unique key conflict occurs, INSERTthe INSERT ... ON DUPLICATE KEY UPDATElocking range of , is REPLACEsmaller than the locking range of . In this scenario, you can use INSERT ... ON DUPLICATE KEY UPDATEinstead of REPLACEto avoid deadlock. If you are interested, you can test it yourself.

Scene four

illustrate

This case tests the case of primary key conflict. The unique key on the table is first deleted to avoid interference.
Deadlock will also occur in this scenario of unique key conflict. The deadlock situation is the same. If you are interested, you can verify it yourself.

time	session1	session2	session3
T1	BEGIN;INSERT INTO t1 (id,a, b) VALUES (6,60, 0);
T2		BEGIN;INSERT INTO t1 (id,a, b) VALUES(6,70, 0); --Blocked
T3			BEGIN;INSERT INTO t1 (id,a, b) VALUES(6,80, 0);-- blocked
T4	ROLLBACK;
T5		1 rows affected;	DEADLOCK,ROLLBACK;

lock situation

The lock situation in stages T1, T2, and T3 is as follows. There is no GAP lock at this time, but a record lock. The corresponding lock status is as follows:

mysql>  SELECT ENGINE_TRANSACTION_ID, OBJECT_NAME, INDEX_NAME, LOCK_TYPE, LOCK_MODE, LOCK_STATUS, LOCK_DATA FROM performance_schema.data_locks;
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
| ENGINE_TRANSACTION_ID | OBJECT_NAME | INDEX_NAME | LOCK_TYPE | LOCK_MODE     | LOCK_STATUS | LOCK_DATA |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
|               xxxxxx3 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx3 | t1          | PRIMARY    | RECORD    | S,REC_NOT_GAP | WAITING     | 6         |
|               xxxxxx2 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx2 | t1          | PRIMARY    | RECORD    | S,REC_NOT_GAP | WAITING     | 6         |
|               xxxxxx1 | t1          | NULL       | TABLE     | IX            | GRANTED     | NULL      |
|               xxxxxx1 | t1          | PRIMARY    | RECORD    | X,REC_NOT_GAP | GRANTED     | 6         |
+-----------------------+-------------+------------+-----------+---------------+-------------+-----------+
6 rows in set (0.00 sec)

T4 time

Session1 ROLLBACK, session2 and session3 all acquired the S lock. In the INSERTstage, a NEXT-KEY lock was generated, and the lock range was (5, supremum].

At this point, a closed-loop lock wait is formed, and the deadlock condition is reached: session2 and session3 respectively want to obtain the insertion intention lock in the insertion gap (5, supremum), but are blocked by the GAP lock held by the other party.

Below is the printed deadlock log.

After the deadlock is triggered, we look at the lock status.

At this time, session2 holds (5, supremum), and inserting records within this range will be blocked.

mysql>  SELECT ENGINE_TRANSACTION_ID, OBJECT_NAME, INDEX_NAME, LOCK_TYPE, LOCK_MODE, LOCK_STATUS, LOCK_DATA FROM performance_schema.data_locks;
+-----------------------+-------------+------------+-----------+--------------------+-------------+------------------------+
| ENGINE_TRANSACTION_ID | OBJECT_NAME | INDEX_NAME | LOCK_TYPE | LOCK_MODE          | LOCK_STATUS | LOCK_DATA              |
+-----------------------+-------------+------------+-----------+--------------------+-------------+------------------------+
|               xxxxxx2 | t1          | NULL       | TABLE     | IX                 | GRANTED     | NULL                   |
|               xxxxxx2 | t1          | PRIMARY    | RECORD    | S                  | GRANTED     | supremum pseudo-record |
|               xxxxxx2 | t1          | PRIMARY    | RECORD    | X,INSERT_INTENTION | GRANTED     | supremum pseudo-record |
|               xxxxxx2 | t1          | PRIMARY    | RECORD    | S,GAP              | GRANTED     | 6                      |
+-----------------------+-------------+------------+-----------+--------------------+-------------+------------------------+
4 rows in set (0.00 sec)

summary

From the previous experiments, we can see that no matter it is INSERTor REPLACE, in the case of high concurrency, due to the existence of the unique key, even under the RC isolation level, there is still a high probability of triggering a deadlock. Currently, fault tolerance can only be done on the business side. Here are some suggestions to reduce or avoid INSERTdeadlocks:

The RC isolation level has a smaller probability of deadlock than the RR isolation level, but it is still inevitable.
INSERT ... ON DUPLICATE KEY UPDATEREPLACEIt has less chance of deadlock and is safer and more efficient than .
Concurrent transactions process data in the same order.
Submit transactions as soon as possible to avoid large and long transactions.

In addition, through the previous experiments, you may have the following questions:

Why does the RC isolation level use GAP locks?
Why are primary keys and unique keys treated differently?
...？？？

Those who are interested can find the answer in the following article: http://mysql.taobao.org/monthly/2022/05/02/ For more technical articles, please visit: https://opensource.actionsky.com/

About SQLE

SQLE from the Axon open source community is a SQL audit tool for database users and managers that supports multi-scenario audits, standardized online processes, native support for MySQL audits and scalable database types.

SQLE get

type	address
Repository	https://github.com/actiontech/sqle
document	https://actiontech.github.io/sqle-docs/
release news	https://github.com/actiontech/sqle/releases
Data audit plug-in development documentation	https://actiontech.github.io/sqle-docs/docs/dev-manual/plugins/howtouse

How does MySQL avoid INSERT deadlock under RC isolation level?

Said it in front

Review row locks

RECORD LOCK

Gap LOCK (also called range lock)

NEXT-KEY lock

INSERT INTENTION LOCK

implicit lock

Different types of locks

Lock range is different

Deadlock conditions

Deadlock observation method

Officially begin

Prepare data

scene one

Process explanation

T1 time

T2 time

T3 time

Scene 2

Process explanation

T1 time

lock situation

T2 time

T3 time

lock situation

T4 time

Scene three

How to avoid deadlocks in scenarios 2 and 3?

Scene four

illustrate

lock situation

T4 time

summary

About SQLE

SQLE get

Guess you like