Lock issues in relational databases are sometimes difficult to deal with. This article from the technical community " Technology Sharing | Quickly Locate Troublesome Global Locks " helps to understand several ways to locate global locks in MySQL databases.
background
When using backup tools such as xtrabackup for backup, there will be a global lock. Normally, the lock occupies a very short time, but occasionally the lock will be occupied for a long time, causing system write blocking. The phenomenon is that show processlist shows that many sessions show wait global read lock. That could have a big impact on the business. Moreover, show processlist cannot see which session holds the global lock. If the backup process is killed directly, the process may be killed, but the lock is still not released, and the database still cannot be written. At this time, we need a method to quickly locate the session holding the global lock. Kill the corresponding session database and return to normal.
Usually when this kind of emergency occurs, the DBA needs to be able to quickly restore the business. If there is no reserve, it will definitely be too late to find a method now. The following methods can help quickly locate the lock session and restore the business in actual failures.
method
Method 1: Using the metadata_locks view
This method is only applicable to MySQL 5.7 and above. In this version, performance_schema adds metadata_locks. If the metadata lock probe is enabled before locking (the default is not enabled), the global lock session can be easily located. The process is as follows.
开启元数据锁对应的探针
mysql> UPDATE performance_schema.setup_instruments SET ENABLED = 'YES' WHERE NAME = 'wait/lock/metadata/sql/mdl';
Query OK, 1 row affected (0.04 sec)
Rows matched: 1 Changed: 1 Warnings: 0
模拟上锁
mysql> flush tables with read lock;
Query OK, 0 rows affected (0.06 sec)
mysql> select * from performance_schema.metadata_locks;
+-------------+--------------------+----------------+-----------------------+---------------------+---------------+-------------+-------------------+-----------------+----------------+
| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | OBJECT_INSTANCE_BEGIN | LOCK_TYPE | LOCK_DURATION | LOCK_STATUS | SOURCE | OWNER_THREAD_ID | OWNER_EVENT_ID |
+-------------+--------------------+----------------+-----------------------+---------------------+---------------+-------------+-------------------+-----------------+----------------+
| GLOBAL | NULL | NULL | 140613033070288 | SHARED | EXPLICIT | GRANTED | lock.cc:1110 | 268969 | 80 |
| COMMIT | NULL | NULL | 140612979226448 | SHARED | EXPLICIT | GRANTED | lock.cc:1194 | 268969 | 80 |
| GLOBAL | NULL | NULL | 140612981185856 | INTENTION_EXCLUSIVE | STATEMENT | PENDING | sql_base.cc:3189 | 303901 | 665 |
| TABLE | performance_schema | metadata_locks | 140612983552320 | SHARED_READ | TRANSACTION | GRANTED | sql_parse.cc:6030 | 268969 | 81 |
+-------------+--------------------+----------------+-----------------------+---------------------+---------------+-------------+-------------------+-----------------+----------------+
4 rows in set (0.01 sec)
OBJECT_TYPE=GLOBAL LOCK_TYPE=SHARED 表示全局锁
mysql> select t.processlist_id from performance_schema.threads t join performance_schema.metadata_locks ml on ml.owner_thread_id = t.thread_id where ml.object_type='GLOBAL' and ml.lock_type='SHARED';
+----------------+
| processlist_id |
+----------------+
| 268944 |
+----------------+
1 row in set (0.00 sec)Locate the lock session ID and kill the session directly.
Method 2: Using the events_statements_history view
This method is applicable to MySQL 5.6 and above. Enable performance_schema.eventsstatements_history (5.6 is not enabled by default, 5.7 is enabled by default). This table will be executed against SQL history. If there are too many requests, early information will be automatically cleared, and the session may be locked. information is cleared. The process is as follows.
mysql> update performance_schema.setup_consumers set enabled = 'YES' where NAME = 'events_statements_history'
Query OK, 0 rows affected (0.00 sec)
Rows matched: 1 Changed: 0 Warnings: 0
mysql> flush tables with read lock;
Query OK, 0 rows affected (0.00 sec)
mysql> select * from performance_schema.events_statements_history where sql_text like 'flush tables%'\G
*************************** 1. row ***************************
THREAD_ID: 39
EVENT_ID: 21
END_EVENT_ID: 21
EVENT_NAME: statement/sql/flush
SOURCE: socket_connection.cc:95
TIMER_START: 94449505549959000
TIMER_END: 94449505807116000
TIMER_WAIT: 257157000
LOCK_TIME: 0
SQL_TEXT: flush tables with read lock
DIGEST: 03682cc3e0eaed3d95d665c976628d02
DIGEST_TEXT: FLUSH TABLES WITH READ LOCK
...
NESTING_EVENT_LEVEL: 0
1 row in set (0.00 sec)
mysql> select t.processlist_id from performance_schema.threads t join performance_schema.events_statements_history h on h.thread_id = t.thread_id where h.digest_text like 'FLUSH TABLES%';
+----------------+
| processlist_id |
+----------------+
| 12 |
+----------------+
1 row in set (0.01 sec)Method 3: Use the gdb tool
If neither of the above two methods work or you haven’t had time to enable them, you can try the third method. Use gdb to find all thread information, check the global lock objects held in each thread, and output the corresponding session ID. In order to facilitate quick positioning, I wrote it in script form. You can also use gdb interactive mode, but after attaching the mysql process, mysql will hang completely, and read requests will also be affected. It is not recommended to use interactive mode.
#!/bin/bash
set -v
threads=$(gdb -p $1 -q -batch -ex 'info threads'| awk '/mysql/{print $1}'|grep -v '*'|sort -nk1)
for i in $threads; do
echo "######## thread $i ########"
lock=`gdb -p $1 -q -batch -ex "thread $i" -ex 'p do_command::thd->thread_id' -ex 'p do_command::thd->global_read_lock'|grep -B3 GRL_ACQUIRED_AND_BLOCKS_COMMIT`
if [[ $lock =~ 'GRL_ACQUIRED_AND_BLOCKS_COMMIT' ]]; then
echo "$lock"
break
fi
done
# thread_id变量，5.6和5.7版本有所不同，5.6版本是thd->thread_id，5.7版本是thd->m_thread_id，这里需要留意下script output,
######## thread 2 ########
[Switching to thread 2 (Thread 0x7f610812b700 (LWP 10702))]
#0 0x00007f6129685f0d in poll () from /lib64/libc.so.6
$1 = 9 此处就是mysql中的会话ID
$2 = {static m_active_requests = 1, m_state = Global_read_lock::GRL_ACQUIRED_AND_BLOCKS_COMMIT, m_mdl_global_shared_lock = 0x7f60e800cb10, m_mdl_blocks_commits_lock = 0x7f60e801c900}But the actual environment may be more complicated, and you may not be able to get the information you want using gdb. Is there no way to go about it?
Method 4: show processlist
If the specific user used by the backup program performs the backup, if it is a root user backup, the larger the time value, the greater the probability of a locked session. If the business is also accessed by root, the key point is that state and info are empty. Here is A little trick can be used to quickly filter. After filtering, try to kill the corresponding ID, and then observe whether there are still sessions in the wait global read lock state.
mysql>pager awk '/username/{if (length($7) == 4) {print $0}}'|sort -rk6
mysql>show processlistIf all the above methods fail, the ultimate move...
Method 5: Try restarting. . .
If you think this article is helpful, please feel free to click "Like" and "Reading" at the end of the article, or forward it directly to pyq,
Recently updated articles:
" The Three Cauldrons of Chinese Football and Basketball "
"" and 0 in MySQL "
" A Case Study of MySQL's Implicit Conversion Leading to Weird Phenomenon "
" A SQL scenario where indexing is still very slow in MySQL "
" What is "BRICS"? 》
Recent hot articles:
" Recommend a classic paper on Oracle RAC Cache Fusion "
" The shock that the open source code of the "Red Alert" game brings to us "
Article classification and indexing:
" Classification and Indexing of 1,200 Public Account Articles "