What is the reason that there is a session that cannot be killed during the master-slave switching process?
Author: Qin Guangfei
A member of the Acson DBA team, responsible for the daily problem handling of the project and the troubleshooting of the company's platform. He is interested in databases and has ideas about technology. Once you enter IT, it is as deep as the sea, and since then you have been a passerby.
Source of this article: original contribution
- Produced by the Aikesheng open source community, the original content is not allowed to be used without authorization, please contact the editor and indicate the source for reprinting.
background
As usual, let’s talk about the background of this document first, but before introducing the background, let’s briefly talk about the process of MySQL master-slave switching.
Normally, when master-slave switching occurs, the main library will do the following actions:
- Disconnect traffic entry (unbind VIP)
- set read-only
- Kill the remaining connection to the database
From the library, you need to do the following actions:
- Completing the binlog log that is different from the main library
- close read only
- clear copy information
- Open traffic entry (bind VIP)
Our company's self-developed database cluster management platform Cloud Tree® DMP probably also undergoes such a switching process, and the point of connection between this switching process and this article lies in the residual connection of the main database Kill.
Occasionally found that when DMP kills residual connections during the switchover process, warninformation sometimes appears in the log:[warn] kill process warning：Error 1094：Unknown thread id：4
Later, it was observed that when the master-slave switch is performed in MySQL 5.7, this message will not appear warning, but it will reproduce stably in MySQL 8.0. After further testing and verification, I finally discovered Unknown thread idthe true face of this, which is the "connection" with USER as event_scheduler .
What is event_scheduler?
event_schedulerwhat is it? After all, as you can see from processlistthe message, it doesn't seem to be the same as a normal session.
In fact, it is a special thread in MySQL, which is mainly responsible for executing the events created by the MySQL event scheduler. We know that MySQL has events, which can crontabperform some tasks regularly like Linux.
The MySQL Event Scheduler manages the scheduling and execution of events, that is, tasks that run according to a schedule
When the MySQL event scheduler is enabled event_scheduler=ON, MySQL will start an event_scheduler thread in the background, and the event_scheduler thread will run until the MySQL service stops. This thread will be responsible for checking the current time and defined events, and if the event needs to be executed, the event_scheduler thread will start a new session to execute the event.
It should be noted that in MySQL 5.7, event_schedulerwarning is disabled by default, but it is enabled by default in MySQL 8.0, and this is why no information is found during the switching process of MySQL 5.7.
Why can't Kill be dropped?
After understanding what event_scheduler is probably, let's take a look at why it will be reported when it is Killed Unknown thread id.
Pay attention to processlistthe information, we found the COMMAND value of event_schedulerDaemon . Literally, Daemonit means to be a background guard. In fact, in MySQL, when some special functions are running in the background, the session COMMAND may be marked as (in actual work scenarios, only event_schedulerDaemon has been noticed ).
Because this type of session is not a connection directly initiated by the user, but a thread inside MySQL, it cannot be killed like a normal session.
In the official documentation, there is less information given, and if you are interested, you can read the code yourself.
How to use scheduled tasks?
How to use scheduled tasks, in fact, there are a lot of information on the Internet, if you really need to use it, it is recommended to refer to the official documents. Let's simply use it eventto see the effect.
enable/disable/disable
-- 修改变量 event_scheduler 来动态启用或者关闭 event
mysql> show variables like '%event_scheduler%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| event_scheduler | ON |
+-----------------+-------+
1 row in set (0.00 sec)
mysql>
-- 关闭
mysql> SET GLOBAL event_scheduler = 0;
-- 启用
mysql> SET GLOBAL event_scheduler = 1;
-- 禁用 event_scheduler，只能在配置文件中设置 event_scheduler 为 disable 并重启服务，而不能动态修改
[mysqld]
event_scheduler=DISABLEDcreate
-- 准备测试表和数据
mysql> CREATE TABLE logs(id INT(11) primary key AUTO_INCREMENT,log_message VARCHAR(255) NOT NULL,log_time TIMESTAMP NOT NULL);
Query OK, 0 rows affected, 1 warning (0.02 sec)
mysql> INSERT INTO logs (log_message, log_time) VALUES
-> ('君不见黄河之水天上来，奔流到海不复回', '2023-06-07 09:01:00'),
-> ('君不见高堂明镜悲白发，朝如青丝暮成雪', '2023-06-07 23:02:00'),
-> ('人生得意须尽欢，莫使金樽空对月', '2023-06-08 01:03:00'),
-> ('天生我材必有用，千金散尽还复来', '2023-06-08 18:04:00'),
-> ('烹羊宰牛且为乐，会须一饮三百杯', '2023-06-09 23:05:00'),
-> ('钟鼓馔玉不足贵，但愿长醉不复醒', '2023-06-09 11:06:00'),
-> ('古来圣贤皆寂寞，惟有饮者留其名', '2023-06-10 23:02:00'),
-> ('陈王昔时宴平乐，斗酒十千恣欢谑', '2023-06-11 01:03:00'),
-> ('主人何为言少钱，径须沽取对君酌', '2023-06-12 18:04:00'),
-> ('五花马、千金裘', '2023-06-13 23:05:00'),
-> ('呼儿将出换美酒，与尔同销万古愁', '2023-06-14 11:06:00');
Query OK, 11 rows affected (0.01 sec)
Records: 11 Duplicates: 0 Warnings: 0
mysql>
-- 创建event，实现定时将该日志表中 7 天之前的数据删除
-- 为了快速看到效果，我们每分钟执行一次，一次删除 1 行
mysql> CREATE EVENT delete_logs_event
-> ON SCHEDULE EVERY 1 MINUTE STARTS '2023-06-19 00:00:00'
-> DO
-> DELETE FROM logs
-> WHERE log_time < DATE_SUB(NOW(), INTERVAL 7 DAY) limit 1;
Query OK, 0 rows affected (0.01 sec)Check
-- 执行 show events 查看，需要先进到 event 所在的 schema
mysql> use universe
mysql> show events\G
*************************** 1. row ***************************
Db: universe
Name: delete_logs_event
Definer: root@localhost
Time zone: SYSTEM
Type: RECURRING
Execute at: NULL
Interval value: 1
Interval field: MINUTE
Starts: 2023-06-19 00:00:00
Ends: NULL
Status: ENABLED
Originator: 1862993913
character_set_client: utf8mb4
collation_connection: utf8mb4_0900_ai_ci
Database Collation: utf8mb4_bin
1 row in set (0.00 sec)
mysql>
-- 通过 information_schema.events 可以看到更详细的信息
mysql> select * from information_schema.events\G
*************************** 1. row ***************************
EVENT_CATALOG: def
EVENT_SCHEMA: universe
EVENT_NAME: delete_logs_event
DEFINER: root@localhost
TIME_ZONE: SYSTEM
EVENT_BODY: SQL
EVENT_DEFINITION: DELETE FROM logs
WHERE log_time < DATE_SUB(NOW(), INTERVAL 7 DAY) limit 1
EVENT_TYPE: RECURRING
EXECUTE_AT: NULL
INTERVAL_VALUE: 1
INTERVAL_FIELD: MINUTE
SQL_MODE: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
STARTS: 2023-06-19 00:00:00
ENDS: NULL
STATUS: ENABLED
ON_COMPLETION: NOT PRESERVE
CREATED: 2023-06-18 23:54:23
LAST_ALTERED: 2023-06-18 23:54:23
LAST_EXECUTED: 2023-06-19 00:00:00
EVENT_COMMENT:
ORIGINATOR: 1862993913
CHARACTER_SET_CLIENT: utf8mb4
COLLATION_CONNECTION: utf8mb4_0900_ai_ci
DATABASE_COLLATION: utf8mb4_bin
1 row in set (0.00 sec)
mysql>
-- 查看表中是否被定时删除
mysql> select * from logs;
+----+--------------------------------------------------------+---------------------+
| id | log_message | log_time |
+----+--------------------------------------------------------+---------------------+
| 2 | 君不见高堂明镜悲白发，朝如青丝暮成雪 | 2023-06-07 23:02:00 |
| 3 | 人生得意须尽欢，莫使金樽空对月 | 2023-06-08 01:03:00 |
| 4 | 天生我材必有用，千金散尽还复来 | 2023-06-08 18:04:00 |
| 5 | 烹羊宰牛且为乐，会须一饮三百杯 | 2023-06-09 23:05:00 |
| 6 | 钟鼓馔玉不足贵，但愿长醉不复醒 | 2023-06-09 11:06:00 |
| 7 | 古来圣贤皆寂寞，惟有饮者留其名 | 2023-06-10 23:02:00 |
| 8 | 陈王昔时宴平乐，斗酒十千恣欢谑 | 2023-06-11 01:03:00 |
| 9 | 主人何为言少钱，径须沽取对君酌 | 2023-06-12 18:04:00 |
| 10 | 五花马、千金裘 | 2023-06-13 23:05:00 |
| 11 | 呼儿将出换美酒，与尔同销万古愁 | 2023-06-14 11:06:00 |
+----+--------------------------------------------------------+---------------------+
10 rows in set (0.00 sec)
mysql>
-- 查看 show processlist 中 event_scheduler 的信息，可以看到 stats 为 Waiting for next activation
mysql> select * from information_schema.processlist where user='event_scheduler';
+-------+-----------------+-----------+------+---------+------+-----------------------------+------+
| ID | USER | HOST | DB | COMMAND | TIME | STATE | INFO |
+-------+-----------------+-----------+------+---------+------+-----------------------------+------+
| 12869 | event_scheduler | localhost | NULL | Daemon | 58 | Waiting for next activation | NULL |
+-------+-----------------+-----------+------+---------+------+-----------------------------+------+
1 row in set (0.00 sec)
mysql>
-- 我们在从库上看下 event 的信息，可以看到 STATUS 为 SLAVESIDE_DISABLED，因此不用担心从库重复执行 event
mysql> select * from information_schema.events\G
*************************** 1. row ***************************
EVENT_CATALOG: def
EVENT_SCHEMA: universe
EVENT_NAME: delete_logs_event
DEFINER: root@localhost
TIME_ZONE: SYSTEM
EVENT_BODY: SQL
EVENT_DEFINITION: DELETE FROM logs
WHERE log_time < DATE_SUB(NOW(), INTERVAL 7 DAY) limit 1
EVENT_TYPE: RECURRING
EXECUTE_AT: NULL
INTERVAL_VALUE: 1
INTERVAL_FIELD: MINUTE
SQL_MODE: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
STARTS: 2023-06-19 00:00:00
ENDS: NULL
STATUS: SLAVESIDE_DISABLED
ON_COMPLETION: NOT PRESERVE
CREATED: 2023-06-18 23:54:23
LAST_ALTERED: 2023-06-18 23:54:23
LAST_EXECUTED: NULL
EVENT_COMMENT:
ORIGINATOR: 1862993913
CHARACTER_SET_CLIENT: utf8mb4
COLLATION_CONNECTION: utf8mb4_0900_ai_ci
DATABASE_COLLATION: utf8mb4_bin
1 row in set (0.00 sec) Revise
-- 使用 ALTER 语句修改，其他高权限用户也可以执行，且 event 的用户会变成最后一个 ALTER 的用户
mysql> ALTER EVENT delete_logs_event
-> ON SCHEDULE EVERY 1 DAY STARTS '2023-06-19 00:00:00'
-> DO
-> DELETE FROM logs
-> WHERE log_time < DATE_SUB(NOW(), INTERVAL 7 DAY);
Query OK, 0 rows affected (0.01 sec)
mysql>
-- 可以看到 DEFINER 已经变成了修改的用户，且时间间隔也修改为了 1 天，DELETE 语句也去掉了 LIMIT
mysql> select * from information_schema.events\G
*************************** 1. row ***************************
EVENT_CATALOG: def
EVENT_SCHEMA: universe
EVENT_NAME: delete_logs_event
DEFINER: qin@%
TIME_ZONE: SYSTEM
EVENT_BODY: SQL
EVENT_DEFINITION: DELETE FROM logs
WHERE log_time < DATE_SUB(NOW(), INTERVAL 7 DAY)
EVENT_TYPE: RECURRING
EXECUTE_AT: NULL
INTERVAL_VALUE: 1
INTERVAL_FIELD: DAY
SQL_MODE: ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
STARTS: 2023-06-19 00:00:00
ENDS: NULL
STATUS: ENABLED
ON_COMPLETION: NOT PRESERVE
CREATED: 2023-06-18 23:54:23
LAST_ALTERED: 2023-06-19 00:23:03
LAST_EXECUTED: 2023-06-19 00:23:00
EVENT_COMMENT:
ORIGINATOR: 1862993913
CHARACTER_SET_CLIENT: utf8mb4
COLLATION_CONNECTION: utf8mb4_0900_ai_ci
DATABASE_COLLATION: utf8mb4_bin
1 row in set (0.00 sec)delete
mysql> drop event delete_logs_event;
Query OK, 0 rows affected (0.01 sec)
mysql> show events\G
Empty set (0.00 sec)Pay attention when switching
- When it was created on the master library
event, a master-slave switch occurred after that. At this timeevent, it will not become executed on the new master with the switchover, and the state will not change. eventThat is, the state of the original master is still thereENABLED, andeventthe state of the new master is still thereDISABLED.
Summarize
show processlistThe session whose User is event_scheduler is an internal MySQL thread and cannot be killed.- The SQL statements created on the master database
eventand executed regularly will be played back normally with the replication on the slave database, but will not be executed repeatedly. - After the master-slave switch, the original master
eventwill not be executed on the new master.About SQLE
The SQLE of the Akson open source community is a SQL auditing tool for database users and managers, which supports multi-scenario auditing, supports standardized online processes, natively supports MySQL auditing, and has scalable database types.
SQLE get
| type | address |
|---|---|
| Repository | https://github.com/actiontech/sqle |
| document | https://actiontech.github.io/sqle-docs/ |
| release news | https://github.com/actiontech/sqle/releases |
| Data audit plug-in development documentation | https://actiontech.github.io/sqle-docs-cn/3.modules/3.7_auditplugin/auditplugin_development.html |