In daily operation and maintenance work, the execution of SQL statements on the MySQL database server causes a sudden increase in the CPU usage of the server. How to quickly locate which SQL statement to check through existing means and take emergency measures. This article introduces the CPU usage monitoring method based on the traditional operating system thread, and uses the corresponding relationship between the operating system thread ID and MySQL thread ID to gradually locate abnormal SQL and transactions.
1. Operating system process and thread ID
1.1 MySQL single process and multi-thread relationship
MySQL is a single-process, multi-threaded database. A process is an instance of a running program, and a thread is the smallest unit that the operating system can perform operation scheduling.
- Process: refers to an application program running in the memory, each process has an independent memory space; a process is also an execution process of the program, and is the basic unit of the system to run the program; running a program in the system is a process from creation, process until it dies.
- Thread: A thread is an execution unit in a process, responsible for the execution of programs in the current process, and there is at least one thread in a process. A thread is the smallest unit that an operating system can perform operation scheduling. It is included in a process and is the actual operating unit in the process. Multiple threads can run concurrently in a process, and each thread executes different tasks in parallel.
The MySQL database chooses single-process multi-threading because the process concurrently sends many threads to different CPUs at the same time, and the threads share the same memory unit/memory address space. Since the communication between threads is carried out on the same address space, there is no need Additional communication mechanisms, which make communication easier and information transfer faster.
The picture above is the background thread of the MySQL storage engine InnoDB, which is introduced in detail in "InnoDB Storage Engine Decryption". InnoDB background threads are mainly used to maintain the normal operation of the server and complete tasks submitted by users, mainly including: master thread, page cleaner thread, purge thread, read thread, write thread, redo log thread, insert buffer thread, monitor thread, error monitor thread, lock monitor thread, etc.
1.1.1 Master Thread
The master thread is the core background thread, which is mainly responsible for asynchronously refreshing the data in the buffer pool to the disk to ensure data consistency, including refreshing dirty pages, merging and inserting buffers, and recycling undo pages. The master thread is composed of multiple loops, including the main loop, background loop, refresh loop, and pause loop. The master thread will switch between different loops according to the state of the database. In the main loop before InnoDB 1.0.x, there are two major operations: the operation every second and the operation every 10 seconds.
1) Operations once per second include:
- The log buffer is flushed to disk, even if the transaction has not yet been committed (always), which explains why the large transaction commit is very fast
- Merge insert buffer (possibly), merge insert does not happen every second, InnoDB will judge whether the number of IOs occurring in the current one second is less than 5, if yes, the system thinks the current IO pressure is very small, you can execute merge insert buffer operate.
- Refresh at most 100 dirty pages of InnoDB's buffer pool to disk (possibly). This refresh of 100 dirty pages is not done every second.
Even if a transaction has not been committed, the InnoDB storage engine will still flush the contents of the redo log cache to the redo log every second.
2) Operations every 10 seconds include:
- Flush 100 dirty pages to disk (possibly);
- merge up to 5 insert buffers (always);
- Flush the log buffer to disk (always);
- delete useless undo pages (always);
- Generate a checkpoint (checkpoing);
In the above process, the InnoDB storage engine will first judge whether the disk IO operations in the past 10s are less than 200 times. If the InnoDB storage engine thinks that there is enough disk IO operation capability, it will flush 100 dirty pages to the disk. Next, the InnoDB storage engine merges the insert buffer, and then flushes the log buffer to disk. Finally, the InnoDB storage engine will perform a full purge operation to delete useless undo pages. It will first determine whether the rows that have been marked for deletion in the current system can be deleted, and if so, they can be deleted immediately.
Before the InnoDB 1.0.x version, the InnoDB storage engine had restrictions on IO, and the buffer pool was hard-coded for disk refresh. With the improvement of disk hardware performance, this method will limit the performance of InnoDB on disk IO. Therefore, after version 1.0.x, InnoDB provides the parameter innodb_io_capacity to indicate the throughput of disk IO, and the default value is 200. For the number of pages flushed to disk, it will be controlled according to the percentage of innodb_io_capacity:
- When merging insert buffers, the number of merged insert buffers is 5% of the innodb_io_capacity value
- When flushing dirty pages from the buffer, the number of dirty pages flushed is innodb_io_capacity
The command show engine innodb status; can view the master thread information:
=====================================
2021-08-22 20:38:37 140186194323200 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 12 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 1 srv_active, 0 srv_shutdown, 255 srv_idle
srv_master_thread log flush and writes: 0
1.1.2 Purge Thread
After the transaction is committed, the undo log it uses may no longer be needed, so Purge Thread is needed to reclaim the undo pages that have been used and allocated. Its function is to actually delete records and delete undo logs.
For example, the statement delete from tb1 where pk=1;
- Mark the record with pk=1 as delete (delete-mark, infobits), the record with pk=1 in the database still exists at this time, and the space has not been released. This operation is a synchronous operation (after the SQL is executed, the mark is completed up)
- Purge is an asynchronous operation of a background thread (purge thread), which will actually delete the record and release the space. The purge thread is automatic by the system and cannot be controlled manually.
There are two reasons why the page is marked as deleted: 1) The thing may need to be rolled back, so keep it first; 2) When thing 1 deletes pk=1 and does not submit it, thing 2 must be able to see pk=1 Record (isolation of things). According to different filter conditions, the processing of deletion marks is different, as shown in the following table:
Therefore, the records marked as delete-mark will be reclaimed by the purge thread at last, and Purge will detect whether there are other things referencing undo on the record, and if not, it can be deleted. Starting from InnoDB version 1.2, InnoDB supports multiple purge threads, which can speed up the recovery of undo pages. At the same time, reading undo pages discretely can further improve the random read performance of the disk. Currently, the default setting in MySQL 8.0 is 4.
mysql> show variables like 'innodb_purge_threads';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| innodb_purge_threads | 4 |
+----------------------+-------+
1 row in set (0.00 sec)
1.1.3 Page Cleaner Thread
Page Cleaner Thread is newly introduced in InnoDB 1.2.x version. Its function is to put the refresh operation of dirty pages in the previous version into a separate thread, which reduces the work of Master Thread and the blocking of user query threads. .
1.1.4 IO Thread
In the InnoDB storage engine, asynchronous IO is widely used to process write IO requests. The work of IO Thread is mainly responsible for the callback processing of these IO requests. There are four IO threads in InnoDB, namely write, read, insert buffer and log IO thread:
- write thread: Responsible for database write operations, multiple write threads can be configured
- read thread: Responsible for the read operation of the database, multiple read threads can be configured
- insert buffer thread: Mainly responsible for the merge operation of the insert buffer
- log thread: used to flush the redo log to the log file
The IO Thread in InnoDB can be observed by command SHOW ENGINE INNODB STATUS:
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
854 OS file reads, 207 OS file writes, 38 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
1.2 Get the MySQL process ID in the system
The method to obtain the MySQL process ID in the Linux system is very simple. Use the command ps -ef|grep mysqld to see its system process ID. As shown below, the running ID of the process MySQL is 1479:
[root@tango-GDB-DB01 ~]# ps -ef|grep mysqld
mysql 1479 1090 1 16:59 ? 00:00:03 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=tango-GDB-DB01.err --pid-file=/usr/local/mysql/data/tango-GDB-DB01.pid --socket=/tmp/mysql.sock
1.3 Thread ID information under the MySQL process
There are several ways to display a specific thread information in Linux:
1) TOP -H displays thread information
The top command can display the status of each thread in real time. Call the "-H" option of the top command, which will list all Linux threads. Adding -p will filter the thread information under the specific process, as follows:
[root@tango-GDB-DB01 ~]# top -H -p 1479
top - 17:10:44 up 11 min, 2 users, load average: 0.00, 0.05, 0.05
Threads: 39 total, 0 running, 39 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1867024 total, 1051452 free, 495804 used, 319768 buff/cache
KiB Swap: 2097148 total, 2097148 free, 0 used. 1185628 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1479 mysql 20 0 1318236 367380 16280 S 0.0 19.7 0:01.55 mysqld
1527 mysql 20 0 1318236 367380 16280 S 0.0 19.7 0:00.00 mysqld
1528 mysql 20 0 1318236 367380 16280 S 0.0 19.7 0:00.00 mysqld
2) In the ps command, the "-T" option can enable thread viewing
The following command lists all threads created by process id 1479.
ps -aT -p <pid>
[root@tango-GDB-DB01 ~]# ps -aT -p 1479
PID SPID TTY TIME CMD
1479 1479 ? 00:00:01 mysqld
1479 1527 ? 00:00:00 mysqld
1479 1528 ? 00:00:00 mysqld
3) Command ps -Lef to view threads
[root@tango-GDB-DB01 ~]# ps -Lef|grep 1479
UID PID PPID LWP C NLWP STIME TTY TIME CMD
mysql 1479 1090 1479 0 39 16:59 ? 00:00:01 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=tango-GDB-DB01.err --pid-file=/usr/local/mysql/data/tango-GDB-DB01.pid --socket=/tmp/mysql.sock
mysql 1479 1090 1527 0 39 16:59 ? 00:00:00 /usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --plugin-dir=/usr/local/mysql/lib/plugin --user=mysql --log-error=tango-GDB-DB01.err --pid-file=/usr/local/mysql/data/tango-GDB-DB01.pid --socket=/tmp/mysql.sock
In the above command, PID gives the process number, LWP shows the thread ID, C shows the CPU usage, and NLWP shows the number of threads in the thread group.
1.4 pstack view thread stack information
The command pstack can be used to view the stack information in the process. It should be noted that running pstack will temporarily block the mysqld process, so please do not execute it during peak business hours.
2. Correspondence between system thread ID and MySQL thread ID
2.1 ProcessID and ThreadID in MySQL
1) Connect to MySQL and execute the following statement
[root@tango-GDB-DB01 local]# mysql -h192.168.112.121 -P3306 -uroot -p –A
mysql> begin;
mysql> begin;select count(1),sleep(2000) from tango.t2 for update;
2) show processlist View processlist information, find processlist_id
mysql> show processlist;
+----+-----------------+----------------------+-------+---------+------+------------------------+-------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-----------------+----------------------+-------+---------+------+------------------------+-------------------------------------------+ |
| 10 | root | tango-GDB-DB01:50620 | tango | Query | 52 | User sleep | select count(1),sleep(2000) from tango.t2 for update |
| 11 | root | localhost | NULL | Query | 0 | init | show processlist |
+----+-----------------+----------------------+-------+---------+------+------------------------+-------------------------------------------+
3 rows in set (0.00 sec)
3) Find the system thread ID corresponding to the MySQL internal thread
Starting from MySQL 5.7, the performance_schema.threads table adds a THREAD_OS_ID column, which is used to record the system thread ID corresponding to the MySQL internal thread.
mysql> select * from performance_schema.threads where processlist_id=10\G
*************************** 1. row ***************************
THREAD_ID: 49
NAME: thread/sql/one_connection
TYPE: FOREGROUND
PROCESSLIST_ID: 10
PROCESSLIST_USER: root
PROCESSLIST_HOST: tango-GDB-DB01
PROCESSLIST_DB: tango
PROCESSLIST_COMMAND: Query
PROCESSLIST_TIME: 84
PROCESSLIST_STATE: User sleep
PROCESSLIST_INFO: select count(1),sleep(2000) from tango.t2 for update
PARENT_THREAD_ID: NULL
ROLE: NULL
INSTRUMENTED: YES
HISTORY: YES
CONNECTION_TYPE: SSL/TLS
THREAD_OS_ID: 1657
RESOURCE_GROUP: USR_default
1 row in set (0.00 sec)
The above finds the thread_id inside MySQL and the thread ID of the corresponding operating system: THREAD_OS_ID
4) Find the corresponding operating system thread information
[root@tango-GDB-DB01 ~]# top -H -p 1479
top - 17:41:22 up 41 min, 3 users, load average: 0.00, 0.01, 0.05
Threads: 39 total, 0 running, 39 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 1.5 sy, 0.0 ni, 98.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1867024 total, 483128 free, 513864 used, 870032 buff/cache
KiB Swap: 2097148 total, 2097148 free, 0 used. 1158844 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1657 mysql 20 0 1318236 377188 16536 S 0.0 20.2 0:00.04 mysqld
Through top -H, you can see the CPU usage of the corresponding thread, including memory and CPU usage
2.2 Query current statements and historical statements
1) Query the current statements table
mysql> select * from performance_schema.events_statements_current WHERE THREAD_ID = 49\G
*************************** 1. row ***************************
THREAD_ID: 49
EVENT_ID: 20
END_EVENT_ID: NULL
EVENT_NAME: statement/sql/select
SOURCE: init_net_server_extension.cc:96
TIMER_START: 10615744184481000
TIMER_END: 10743215769959000
TIMER_WAIT: 127471585478000
LOCK_TIME: 121000000
SQL_TEXT: select count(1),sleep(2000) from tango.t2 for update
DIGEST: 91558228446c9877c86805735096b85b8afe5a8148c4ea9c315a1948464ab27f
DIGEST_TEXT: SELECT COUNT (?) , `sleep` (?) FROM `tango` . `t2` FOR UPDATE
CURRENT_SCHEMA: tango
OBJECT_TYPE: NULL
OBJECT_SCHEMA: NULL
OBJECT_NAME: NULL
OBJECT_INSTANCE_BEGIN: NULL
MYSQL_ERRNO: 0
RETURNED_SQLSTATE: NULL
MESSAGE_TEXT: NULL
ERRORS: 0
WARNINGS: 0
ROWS_AFFECTED: 0
ROWS_SENT: 0
ROWS_EXAMINED: 0
CREATED_TMP_DISK_TABLES: 0
CREATED_TMP_TABLES: 0
SELECT_FULL_JOIN: 0
SELECT_FULL_RANGE_JOIN: 0
SELECT_RANGE: 0
SELECT_RANGE_CHECK: 0
SELECT_SCAN: 1
SORT_MERGE_PASSES: 0
SORT_RANGE: 0
SORT_ROWS: 0
SORT_SCAN: 0
NO_INDEX_USED: 1
NO_GOOD_INDEX_USED: 0
NESTING_EVENT_ID: 18
NESTING_EVENT_TYPE: TRANSACTION
NESTING_EVENT_LEVEL: 0
STATEMENT_ID: 37
1 row in set (0.00 sec)
2) Execute SHOW ENGINE INNODB STATUS\G to view the transaction status:
---TRANSACTION 2715147, ACTIVE 480 sec
mysql tables in use 1, locked 1
2 lock struct(s), heap size 1136, 3 row lock(s)
MySQL thread id 10, OS thread handle 140127942276864, query id 37 tango-GDB-DB01 192.168.112.121 root User sleep
select count(1),sleep(2000) from tango.t2 for update
Trx read view will not see trx with id >= 2715147, sees < 2715147
MySQL connection ID = 10, OS thread handle = 140127942276864
3) View historical thread_statement information
mysql> select * from performance_schema.events_statements_history WHERE THREAD_ID = 49 limit 1 \G
*************************** 1. row ***************************
THREAD_ID: 49
EVENT_ID: 17
END_EVENT_ID: 18
EVENT_NAME: statement/sql/begin
SOURCE: init_net_server_extension.cc:96
TIMER_START: 10288037810168000
TIMER_END: 10288037917673000
TIMER_WAIT: 107505000
LOCK_TIME: 0
SQL_TEXT: begin
DIGEST: 55fa5810fbb2760e86d578526176c1497b134d4ef3dd0863dd78b1c5e819848c
DIGEST_TEXT: BEGIN
CURRENT_SCHEMA: tango
OBJECT_TYPE: NULL
OBJECT_SCHEMA: NULL
OBJECT_NAME: NULL
OBJECT_INSTANCE_BEGIN: NULL
MYSQL_ERRNO: 0
RETURNED_SQLSTATE: 00000
MESSAGE_TEXT: NULL
ERRORS: 0
WARNINGS: 0
ROWS_AFFECTED: 0
ROWS_SENT: 0
ROWS_EXAMINED: 0
CREATED_TMP_DISK_TABLES: 0
CREATED_TMP_TABLES: 0
SELECT_FULL_JOIN: 0
SELECT_FULL_RANGE_JOIN: 0
SELECT_RANGE: 0
SELECT_RANGE_CHECK: 0
SELECT_SCAN: 0
SORT_MERGE_PASSES: 0
SORT_RANGE: 0
SORT_ROWS: 0
SORT_SCAN: 0
NO_INDEX_USED: 0
NO_GOOD_INDEX_USED: 0
NESTING_EVENT_ID: NULL
NESTING_EVENT_TYPE: NULL
NESTING_EVENT_LEVEL: 0
STATEMENT_ID: 28
1 row in set (0.00 sec)
2.3 kill long-running or high-CPU thread
1) show processlists to find long-running SQL
mysql> show processlist;
+----+-----------------+----------------------+-------+---------+-------+------------------------+------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+-----------------+----------------------+-------+---------+-------+------------------------+------------------------------------------------------+
| 5 | event_scheduler | localhost | NULL | Daemon | 11423 | Waiting on empty queue | NULL |
| 10 | root | tango-GDB-DB01:50620 | tango | Query | 816 | User sleep | select count(1),sleep(2000) from tango.t2 for update |
2) Find the thread with high CPU consumption through TOP -H
Reverse lookup to processlist id
mysql> select * from performance_schema.threads WHERE THREAD_OS_ID = 1657\G
*************************** 1. row ***************************
THREAD_ID: 49
NAME: thread/sql/one_connection
TYPE: FOREGROUND
PROCESSLIST_ID: 10
PROCESSLIST_USER: root
PROCESSLIST_HOST: tango-GDB-DB01
PROCESSLIST_DB: tango
PROCESSLIST_COMMAND: Query
PROCESSLIST_TIME: 1050
PROCESSLIST_STATE: User sleep
PROCESSLIST_INFO: select count(1),sleep(2000) from tango.t2 for update
PARENT_THREAD_ID: NULL
ROLE: NULL
INSTRUMENTED: YES
HISTORY: YES
CONNECTION_TYPE: SSL/TLS
THREAD_OS_ID: 1657
RESOURCE_GROUP: USR_default
1 row in set (0.00 sec)
3) kill processlist id on the mysql client
mysql> kill 10;
Query OK, 0 rows affected (0.00 sec)
Kill the current long transaction or thread
3. Correspondence between OS thread handle and operating system thread ID
The OS thread handle 140127942276864 found in step 2.2 (the OS thread handle is the internal ID used to identify each thread inside the process), here is a decimal value, which needs to be converted into hexadecimal first:
mysql> select lower(conv(140127942276864, 10, 16));
+--------------------------------------+
| lower(conv(140127942276864, 10, 16)) |
+--------------------------------------+
| 7f721438f700 |
+--------------------------------------+
1 row in set (0.00 sec)
2) Use pstack to query the association between the handle and the operating system thread ID:
[root@tango-GDB-DB01 ~]# pstack 1479 |grep 7f721438f700
Thread 3 (Thread 0x7f721438f700 (LWP 1657)):
You can see that LWP=1657, corresponding to the above THREAD_OS_ID value, LWP is the abbreviation of Light-Weight Processes.
- https://blog.csdn.net/n88Lpo/article/details/121964562
- InnoDB decryption of database series