1. Background
With the development of the company's business, the commodity inventory has become an independent system from the commodity center, undertaking the main site commodity inventory verification, order inventory deduction, and after-sales inventory release. Before going online, we conducted a stress test on the core interface. During the stress test, a MySQL 5.6.35 deadlock occurred. The log found that the deadlock was only a simple SQL. How did the deadlock occur? Carrying forward the fine tradition of technical personnel inquiring into the roots, a detailed investigation and summary of the cause of this deadlock were carried out. This article is a record of this process.
Before delving into the problem, let's first understand MySQL's locking mechanism.
Two, MySQL lock mechanism
The first thing to be clear is that MySQL locking is actually locking the index, not the data. Let's first look at the structure of the MySQL index.
MySQL index into primary key index (or clustered index) and a secondary index (or primary key index, the non-clustered index, secondary index, other indexes including all various external primary key index). Different storage engines have slightly different ways of organizing data.
For InnoDB, the primary key index and data are stored together to form a B+ tree (called an index-organized table). The primary key is located in the non-leaf node, and the data is stored in the leaf node. The schematic diagram is as follows:
MyISAM is a heap-organized table. The primary key index and data are stored separately. The leaf nodes store only the physical address of the data. The schematic diagram is as follows:
The organization of the secondary index is the same for InnoDB and MyISAM. The corresponding relationship between the secondary index and the primary key index is saved. The secondary index column is located in the non-leaf node, and the primary key value is located in the leaf node. The schematic diagram is as follows:
So under this index structure of MySQL, how do we find the data we need?
Take select * from t where name='aaa' as an example. After MySQL Server parses the sql, it finds that the name field has an index available, so first find the primary key id based on name='aaa' on the secondary index (Figure 2-2) =17, then according to the primary key 17 to the primary key index (Figure 2-1) to find the required record.
After understanding the principle of MySQL using indexes to organize and retrieve data, let's look at how MySQL yokes indexes.
What you need to understand is how the index is locked and the index type (primary key, unique, non-unique, no index) and isolation level (RC, RR, etc.). In this example, the isolation level is limited to RC. In the case of RR, it is basically the same as RC locking. The difference is that RC adds an additional gap lock to prevent phantom reading.
2.1 Update based on the primary key
update t set name='xxx' where id=29; only need to add X lock to the record with id=29 on the primary key (X lock is called mutual exclusion lock, after locking, this transaction can read and write, other transactions Reading and writing will be blocked). as follows:
2.2 Update based on unique index
update t set name='xxx' where name='ddd'; It is assumed that name is unique here. InnoDB now finds the index entry with name='ddd' (id=29) on the name index and adds X lock, and then finds the corresponding leaf node on the primary key index according to id=29 and adds X lock.
There are two locks, one is added to the unique index and the other is added to the primary key index. What needs to be explained here is that the lock is added step by step, and the unique index and the primary key index will not be locked at the same time. This step-by-step locking mechanism is actually one of the causes of deadlock. The schematic is as follows:
2.3 Update based on non-unique index
update t set name='xxx' where name='ddd'; It is assumed that the name is not unique, that is, multiple records (with different ids) can be found according to the name. Similar to the above unique index locking, the difference is that all eligible index items will be locked. The schematic is as follows:
There are a total of four locks, the steps to lock are as follows:
- Find the index entry of (ddd, 29) on the non-unique index (name), add X lock;
- According to (ddd, 29) to find the (29, ddd) record of the primary key index, add X lock;
- Find the index entry of (ddd, 37) on the non-unique index (name), add X lock;
- According to (ddd, 29), find the (37, ddd) record of the primary key index, and add X lock;
As can be seen from the above steps, InnoDB locks each eligible record step by step, that is, first add the secondary index and then the primary key index; secondly, it is locked by record, that is, after adding a record, Add another record until all eligible records are locked. So when will the lock be released? The answer is that all locks will be released when the transaction ends.
Summary: MySQL locks are related to index types. Locks are added record by record. In addition, locks are also related to isolation level.
Three, deadlock phenomenon and investigation
After understanding how MySQL locks indexes, let's move on to the topic and look at the deadlock phenomenon and its cause analysis in actual scenarios.
The deadlock this time is the inventory deduction interface. The main logic of the interface is to deduct the inventory of the ordered item in a warehouse after the user places an order. For example, a user orders an X50 mobile phone and an X30 headset on the vivo official website. After placing the order, first determine the shipping warehouse according to the user's receiving address, and then subtract an X50 inventory and an X30 from the warehouse. in stock. Before analyzing the deadlock sql, first look at the definition of the commodity inventory table (for ease of understanding, only the main fields are retained):
CREATE TABLE `store` (
`id` int(10) AUTO_INCREMENT COMMENT '主键',
`sku_code` varchar(45) COMMENT '商品编码',
`ws_code` varchar(32) COMMENT '仓库编码',
`store` int(10) COMMENT '库存量',
PRIMARY KEY (`id`),
KEY `idx_skucode` (`sku_code`),
KEY `idx_wscode` (`ws_code`)
) ENGINE=InnoDB COMMENT='商品库存表'Note that there are separate indexes defined for the two fields sku_code and ws_code: idx_skucode, idx_wscode . The main reason for this is that the business requires querying based on a single field.
Look at the inventory deduction update statement:
update store
set store = store-#{store}
where sku_code=#{skuCode} and ws_code = #{wsCode} and (store-#{store}) >= 0The business meaning of this sql is to deduct store inventory from a certain warehouse (wsCode) for a certain product (skuCode). At the same time, the sku_code and ws_code fields appear in the where condition above. The selection of sku_code in the pressure test data is required It is higher than ws_code. In theory, this sql should be indexed by idx_skucode . What is the real situation?
Okay, next, perform a pressure test on the inventory deduction interface card. The concurrency of 50, 5 items per order, deadlock occurs after less than half a minute of pressing, and then pressing again, the problem remains, indicating that it is a must. It must be resolved before proceeding. Execute the show engine innodb status command on the MySQL terminal to view the last deadlock log, focusing on the LATEST DETECTED DEADLOCK part of the log :
------------------------
LATEST DETECTED DEADLOCK
------------------------
2020-xx-xx 21:09:05 7f9b22008700
*** (1) TRANSACTION:
TRANSACTION 4219870943, ACTIVE 0 sec fetching rows
mysql tables in use 3, locked 3
LOCK WAIT 10 lock struct(s), heap size 2936, 3 row lock(s)
MySQL thread id 301903552, OS thread handle 0x7f9b21a7b700, query id 5373393954 10.101.22.135 root updating
update store
set update_time = now(), store = store-1
where sku_code='5468754' and ws_code = 'NO_001' and (store-1) >= 0
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3331 page no 16 n bits 904 index `idx_wscode` of table `store` trx id 4219870943 lock_mode X locks rec but not gap waiting
Record lock, heap no 415 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
0: len 5; hex 5730303735; asc NO_001;;
1: len 8; hex 00000000000025a7; asc % ;;
*** (2) TRANSACTION:
TRANSACTION 4219870941, ACTIVE 0 sec fetching rows, thread declared inside InnoDB 1
mysql tables in use 3, locked 3
9 lock struct(s), heap size 2936, 4 row lock(s)
MySQL thread id 301939956, OS thread handle 0x7f9b22008700, query id 5373393941 10.101.22.135 root updating
update store
set update_time = now(), store = store-1
where sku_code='5655620' and ws_code = 'NO_001' and (store-1) >= 0
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 3331 page no 16 n bits 904 index `idx_wscode` of table `store` trx id 4219870941 lock_mode X locks rec but not gap
Record lock, heap no 415 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
0: len 5; hex 5730303735; asc NO_001;;
1: len 8; hex 00000000000025a7; asc % ;;
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 3331 page no 7 n bits 328 index `PRIMARY` of table `store` trx id 4219870941 lock_mode X locks rec but not gap waiting
Record lock, heap no 72 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 8; hex 00000000000025a7; asc % ;;
1: len 6; hex 0000fb85fdf7; asc ;;
2: len 7; hex 1a00001d3b21d4; asc ;! ;;
3: len 7; hex 35343638373534; asc 5468754;;
4: len 5; hex 5730303735; asc NO_001;;
5: len 8; hex 8000000000018690; asc ;;
6: len 5; hex 99a76b2b97; asc k+ ;;
7: len 5; hex 99a7e35244; asc RD;;
8: len 1; hex 01; asc ;;As can be seen from the above log, there are two transactions, and deadlocks occurred during the execution of the two SQL:
update store set update_time = now(), store = store-1 where sku_code='5468754' and ws_code = 'NO_001' and (store-1) >= 0
update store set update_time = now(), store = store-1 where sku_code='5655620' and ws_code = 'NO_001' and (store-1) >= 0 Look at the actual data:
Figure 3-1 Inventory table data
In other words, the two transactions deadlock occurred when updating different rows of the same table. In our intuitive impression, InnoDB uses row locks. Should different row locks not interfere with each other? So what is going on?
Let's look at the update execution plan:
Figure 3-2 Update statement execution plan
Unlike what we imagined, InnoDB neither uses idx_skucode index nor idx_wscode index, but uses index_merge. What is the relationship between index_merge and these two indexes?
According to the query data, index_merge is an index merge optimization technology introduced after MySQL 5.1. It allows multiple indexes to be used to query the same table at the same time, and the query results of multiple indexes are merged (intersect, and (Union, etc.) and return.
Go back to the update statement above: where sku_code='5468754' and ws_code ='NO_001'; if there is no index_merge, either the idx_skucode index or the idx_wscode index will be used. There will be no situations where the two indexes are used together. After using the index_merge technology, two indexes will be executed at the same time, and the results will be checked separately before merging (where condition is and, so the intersection operation will be done). Combined with the understanding of the locking mechanism (step by record locking) in the second part, do you vaguely think that the simultaneous locking of two indexes is the cause of the deadlock?
Let's take a closer look at the deadlock log. The log is more complicated. The translation is as follows:
1) Transaction one 4219870943 is waiting for the row lock on the index idx_wscode (number space id 3331 page no 16 n bits 904) when executing the update statement .
2) the transaction two 4,219,870,941 during the update statement, already holding idx_wscode row lock on (numbering space id 3331 page no 16 n bits 904), from the lock number, is that a lock transaction needs.
3) Transaction 2 4219870941 is also waiting for a lock on the primary key index. Who is holding this lock? From this line of log (3: len 7; hex 35343638373534; asc 5468754 ;;), it can be seen that it is the line that needs to be updated in transaction one, indicating that this lock is occupied by transaction one.
Well, the deadlock condition is already clear: Transaction 1 is waiting for the row lock on index idx_wscode held by Transaction 2 (number space id 3331 page no 16 n bits 904 ), and Transaction 2 is also waiting for transaction 1 to hold The lock on the primary key index ( 5468754 ) of, everyone does not agree to each other, so they can only freeze there and lock themselves ^_^
Use a picture to illustrate this situation:
The above figure describes only one possible path for deadlock. In fact, if you carefully sort it out, there are other paths that can also lead to deadlock. If you are interested, you can explore it yourself. The above figure is explained as follows:
1) Transaction one ( where sku_code='5468754' and ws_code ='NO_001' ) first takes the idx_skucode index, and successfully locks the secondary index and primary key index (1-1 and 1-2).
2) At this time, transaction 2 starts to execute ( where sku_code='5655620' and ws_code ='NO_001' ), and firstly, idx_skucode (top left) index is taken. Because there is no conflict with the record locked by transaction 1, the lock is successfully locked. (2-1 and 2-2).
3) Transaction 2 continues to execute. At this time, the idx_wscode (upper right) index is taken, and the secondary index is successfully locked (2-3, at this time, transaction 1 has not started to lock on idx_wscode), but it is indexing the primary key When adding an index, it is found that the primary key index with id=9639 has been locked by the transaction, so it can only wait (2-4). At the same time, the locking of other records will be suspended before 2-4 completes the locking (2- 5 and 2-6, because InnoDB locks record by record, the previous one will not be executed if the previous one is not completed).
4) At this time, the transaction continues to execute, and the idx_wscode index is taken at this time, but when the lock is locked, it is found that the index item (NO_001, 9639) has been locked by the second transaction, so it can only wait. For the same reason, the following 1-4 cannot be executed.
So far, the deadlock phenomenon caused by "two transactions, reverse locking" has appeared.
Four, how to solve
The essential cause of the deadlock is still caused by the different locking sequence. In this example, it is caused by Index Merge using two index directions to lock at the same time. The solution is relatively simple, which is to eliminate multiple indexes caused by index merge at the same time. Implementation status.
1) Use force index (idx_skucode) to force an index, so that InnoDB will ignore index merge and avoid the situation where multiple indexes are locked at the same time.
Figure 4-1 Use Force Index to force an index
2) Disable Index Merge, so that InnoDB will only use one of idx_skucode and idx_wscode, all things are locked in the same order and will not cause deadlock.
用命令禁用Index Merge：SET GLOBAL optimizer_switch='index_merge=off,index_merge_union=off,index_merge_sort_union=off,index_merge_intersection=off';
Figure 4-2 Turn off the Index Merge feature
After logging in to the terminal again, look at the execution plan:
Figure 4-3 Index situation after closing Index Merge
3) Since Index Merge uses two independent indexes at the same time, we might as well create a new joint index containing all the fields of these two indexes, so that InnoDB will only use this single joint index, which is actually the same as disabling index merge.
New joint index:
alter table store add index
idx_skucode_wscode(sku_code,ws_code);
Look at the execution plan again, type=range means that index merge is not used, and key=idx_skucode_wscode means that the joint index just created is taken:
Figure 4-4 Use joint index to avoid Index Merge optimization
4) Finally, another way to bypass the index merge restriction is recommended. That is, to remove the deadlock conditions, the specific method is to first use idx_skucode and idx_wscode to query the primary key id, and then use the primary key id to perform the update operation. This method avoids the introduction of X locks by update. Since the final update conditions are uniquely fixed, there is no problem with the lock sequence and deadlocks are avoided.
V. Summary
This article describes the deadlock caused by Index Merge optimization through a practical case, describes in detail the causes and solutions of the deadlock, and introduces the MySQL index structure and locking mechanism by the way. Through this article, you can master the basic theories and general methods of deadlock analysis, hoping to provide ideas for quickly solving actual deadlock problems in your work.
Author: vivo official website mall development team