background

In DBS-Cluster List-More-Connection Query-Deadlock, I saw a database deadlock log on September 22. After investigation, I found that the database deadlock was caused by mysql optimization-index merge.

definition

Index merge: A technology for database query optimization, introduced after MySQL 5.1. It can query on multiple indexes and merge the results back.

Mysql database lock mechanism

Before troubleshooting the problem, first talk about the lock mechanism of the mysql database:

1 The basic unit of locking is next-key lock (record lock + gap lock). When record lock or gap lock can solve the problem of phantom reading, it will degenerate into record lock (row lock) and gap lock.

2 Locking adds locks to the index, not the data.

3 For the current read, the index is locked. The current read statement includes (select... from. ... for update, select... from ..... lock in share mode, update..., delete. ...).

4. Locking is distinguished according to unique index and non-unique index. According to the query conditions, it is divided into equal value query and range query. According to whether the data can be found, it is divided into record existence and non-existence.

The index used in this deadlock problem is the existence of records in the equivalent query of the non-unique index, so this article only introduces this situation in detail. For other situations, you can check the reference document 1 at the bottom:

The locking situation is: it will scan in sequence. First, it will scan the data that matches the condition, and add a next-key lock. Then it will scan the first record that does not match the data, add a gap lock, and finally find the primary key of the record. Add a record lock,

Three types of locks have been added for the above situation. The purpose of locking is to prevent phantom reads from occurring.

Analyze locks on secondary indexes:

Table Structure:

CREATE TABLE `jdi_roster_apply_detail` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键',
  `apply_id` varchar(100) NOT NULL COMMENT '申请单号',
  `status` tinyint(10) NOT NULL COMMENT '状态',
  PRIMARY KEY (`id`),
  KEY `idx_status` (`status`),
  KEY `idx_apply_id` (`apply_id`)
) ENGINE=InnoDB AUTO_INCREMENT=984483 DEFAULT CHARSET=utf8 COMMENT='黑白名单申请单明细'

Table data:

id	apply_id	status
959651	1695369220522068998	1
960738	1695369227576173690	1
961319	1695373047673903326	1
961365	1695373122447865228	1

b+ tree established through idx_apply_id:

Because the index is a secondary index, the data stored in the leaf nodes is the primary key value.

Execute sql:

select * from jdi_roster_apply_detail where apply_id='1695369227576173690' for update

Perform data scanning process

1 Find the record that meets the conditions and add the next-key lock, so the lock is (1695369220522068998,1695369227576173690]

2 Find the first data that does not match the record and increase the gap lock, so the lock is (1695369227576173690, 1695373047673903326)

3 Add a record lock to the primary key index that meets the conditions, so add a record lock to id=960738.

Phantom reading for three types of lock solutions:

1 If there is no next-key lock for the first item, and another transaction adds apply_id=1695369227576173690, and id<960738, the transaction will have one more record when querying, thus causing phantom reading.

2 If there is no second gap lock and another transaction adds apply_id=1695369227576173690, id>960738, the transaction will have one more record when querying, thus causing phantom reading.

3. If there is no third record lock and another transaction deletes a record with id=960738, there will be one less piece of data when the transaction queries, thus causing phantom reading.

Analysis of practical problems

Database deadlock log

The two transactions in the above log executed update statements respectively:

#事务1
update jdi_roster_apply_detail set `status` = 10 where `status` = 1 and apply_id = '1695369220522068998'
#事务2
update jdi_roster_apply_detail set `status` = 10 where `status` = 1 and apply_id = '1695369227576173690'

This sql is used to change the data pending approval of a certain application ID to approved.

Because the update statement cannot be executed in Taishan, the select statement is executed to check the index status:

explain select * from  jdi_roster_apply_detail  where `status` = 1 and apply_id = '1695369220522068998'

Execution result:

It can be seen from the results that both update statements use two indexes, namely idx_status and idx_apply_id. The found results are then merged, so during the simulation process, they can be split into two query statements.

Deadlock simulation

Transaction 1	Transaction 2	lock range
begin	begin
select * from jdi_roster_apply_detail where apply_id = '1695369220522068998' for update		idx_apply_id is locked (-∞, 1695369220522068998], (1695369220522068998, 1695369227576173690) The primary key id index is locked id=959651
	select * from jdi_roster_apply_detail where apply_id = '1695369227576173690' for update	idx_apply_id is locked (1695369220522068998, 1695369227576173690], (1695369227576173690, 1695373047673903326) The primary key id index is locked id=960738
select * from jdi_roster_apply_detail where status = 1 for update		Next-key locks and gap locks will be added to idx_status, but when record locks are added to primary keys 959651, 960738, 961319, and 961365, transaction 2 has already added record locks to 960738, so transaction 1 is blocked.
	select * from jdi_roster_apply_detail where status = 1 for update	Next-key locks and gap locks will be added to idx_status, but when record locks are added to primary keys 959651, 960738, 961319, and 961365, transaction 1 has already added record locks to 959651, so transaction 2 is blocked.
	deadlock

Two transactions each want record locks for two primary key IDs, causing each other to wait and form a deadlock.

The above is to execute the index query of idx_apply_id first and then the index query of idx_status. If the index query of idx_status is executed first and then the index query of idx_apply_id, a deadlock will also be caused due to the record lock of the primary key.

solution

1 Use force index(idx_apply_id) to force an index so that InnoDB will ignore index merge and avoid multiple indexes being locked at the same time.

2 Disable Index Merge. Use the command to disable Index Merge: SET GLOBAL optimizer_switch='index_merge=off,index_merge_union=off,index_merge_sort_union=off,index_merge_intersection=off';

3 Index Merge uses two independent indexes at the same time, so create a new joint index containing all fields of these two indexes, so that InnoDB will only use this separate joint index.

The third solution has better query performance than the first. Compared with the second solution, it only affects the table and has a smaller scope of impact. Therefore, this solution is also adopted this time.

Summarize

The deadlock problem was caused by the optimizer's use of a merged index, which was eventually solved by creating a new joint index.

Reference documentation:

1 https://www.xiaolincoding.com/mysql/lock/how_to_lock.html

Author: JD Industrial Li Xiaohui

Source: JD Cloud Developer Community Please indicate the source when reprinting

Analysis and solutions to database deadlock caused by MySQL index merge | JD Cloud technical team