The author uses a 1590 error in the master-slave replication process to illustrate the precautions in the process of creating user authorization for MySQL 8.0.
Author: Wang Xiang
A member of the Acson DBA team, mainly responsible for MySQL troubleshooting and performance optimization. Persevere in technology and be responsible for customers.
Source of this article: original contribution
- Produced by the Aikesheng open source community, the original content is not allowed to be used without authorization, please contact the editor and indicate the source for reprinting.
Fault description
DMP received an alarm: the SQL thread of the slave library stopped working, and the MySQL version was 5.7.32. When logging in to the slave library to view the replication information, the error was reported as follows:
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
……
Last_Errno: 1590
Last_Error: The incident LOST_EVENTS occured on the master. Message: REVOKE/GRANT failed while granting/revoking privileges in databases.
Skip_Counter: 0
Exec_Master_Log_Pos: 12531
Relay_Log_Space: 69304
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1590
Last_SQL_Error: The incident LOST_EVENTS occured on the master. Message: REVOKE/GRANT failed while granting/revoking privileges in databases.
……
The slave library error log information is as follows:
[ERROR] Slave SQL for channel '': The incident LOST_EVENTS occured on the master. Message: REVOKE/GRANT failed while granting/revoking privileges in databases. Error_code: 1590
[ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000003' position 12531.
Parse the corresponding Binlog information as follows:
# Incident: LOST_EVENTS
RELOAD DATABASE; # Shall generate syntax error
The main library error message is as follows:
[ERROR] REVOKE/GRANT failed while granting/revoking privileges in databases. An incident event has been written to the binary log which will stop the slaves.
The customer reported that after performing some authorization operations and then copying, an error occurred. The executed statement is as follows:
mysql> create user test@'%',app@'%' identified by 'Root@123';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql> grant all on test.* to test@'%',app@'%';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
Failure Analysis
According to the above error information, it can be seen that an error occurred when changing permissions, the main database writes an INCIDENT_EVENT in the binlog, and the standby database directly reports an error after parsing INCIDENT_EVENT.
Under what circumstances will the execution of the authorization statement cause the main library to write INCIDENT_EVENT in the binlog?
When only part of the permission change operation is processed and an error occurs, the main library will write an INCIDENT_EVENT in the binlog.
Then under what circumstances will the permission change only be processed partly but not partly processed?
The following examples illustrate two relevant scenarios.
Issues with MySQL 5.7
In MySQL 5.7, use the GRANT statement to create a new user, and some of the permissions have problems.
Use GRANT to create testusers (MySQL version 8.0 does not support using GRANT to create users):
mysql> grant select,insert,file on test.* to test@'%' identified by 'Q1w2e3E$';
ERROR 1221 (HY000): Incorrect usage of DB GRANT and GLOBAL PRIVILEGES
mysql> select user,host from mysql.user where user='test' and host='%';
+------+--------+
| user | host |
+------+--------+
| test | % |
+------+--------+
1 row in set (0.00 sec)
mysql> show grants for test@'%';
+--------------------------------------------+
| Grants for test@% |
+--------------------------------------------+
| GRANT USAGE ON *.* TO 'test'@'%' |
+---------------------------------------------+
1 row in set (0.00 sec)
When creating a user, testgrant SELECT, INSERT, and FILE permissions to the library. Because the FILE permission cannot be granted to a certain database, the execution of the statement fails. But the end result is: test@'%'creation succeeds, authorization part fails. From the above test, we can see that using GRANT to create a user is actually divided into two steps: creating a user and authorizing. A problem with permissions does not affect the creation of users. The above statement will cause the master library to write INCIDENT_EVENT in the binlog, which will cause an error in the master-slave replication.
GRANT authorizes two users at the same time
Use a GRANT statement to grant authorization test@'10.186.63.5'to test@'10.186.63.29'the user at the same time, where test@'10.186.63.5'the user exists but test@'10.186.63.29'does not exist.
mysql> create user test@'10.186.63.5' identified by '123';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on test.* to test@'10.186.63.5',test@'10.186.63.29';
ERROR 1133 (42000): Can't find any matching row in the user table
mysql> show grants for test@'10.186.63.5';
+----------------------------------------------------------+
| Grants for [email protected] |
+----------------------------------------------------------+
| GRANT USAGE ON *.* TO 'test'@'10.186.63.5' |
| GRANT ALL PRIVILEGES ON `test`.* TO 'test'@'10.186.63.5' |
+----------------------------------------------------------+
2 rows in set (0.00 sec)
According to the above experiments, it can be seen that test@'10.186.63.5'the authorization is successful because the user exists, but test@'10.186.63.29'the authorization fails if the user does not exist. The above statement will also cause the master library to write INCIDENT_EVENT in the binlog, which will cause an error in the master-slave replication.
However, the above two situations do not seem to be in line with the customer's execution statement. Judging from the error report, it is because the password complexity is not enough and the user creation failed. So what is the reason for the 1590 error from the database? Let's take a look at the problem of using the create statement to create two users at the same time after using the password complexity plug-in.
mysql> show global variables like '%validate%';
+--------------------------------------+--------+
| Variable_name | Value |
+--------------------------------------+--------+
| query_cache_wlock_invalidate | OFF |
| validate_password_check_user_name | OFF |
| validate_password_dictionary_file | |
| validate_password_length | 8 |
| validate_password_mixed_case_count | 1 |
| validate_password_number_count | 1 |
| validate_password_policy | MEDIUM |
| validate_password_special_char_count | 1 |
+--------------------------------------+--------+
mysql> select user,host from mysql.user;
+---------------+-----------+
| user | host |
+---------------+-----------+
| universe_op | % |
| root | 127.0.0.1 |
| mysql.session | localhost |
| mysql.sys | localhost |
| root | localhost |
+---------------+-----------+
5 rows in set (0.00 sec)
mysql> create user test@'%',app@'%' identified by 'Root@123';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
mysql> select user,host from mysql.user;（app@'%'创建成功，test@'%'创建失败）
+---------------+-----------+
| user | host |
+---------------+-----------+
| app | % |
| universe_op | % |
| root | 127.0.0.1 |
| mysql.session | localhost |
| mysql.sys | localhost |
| root | localhost |
+---------------+-----------+
6 rows in set (0.00 sec)
The above test uses CREATE USER to create both test@'%', app@'%'. But because the password complexity does not meet the requirements, it fails and reports an error (multiple tests found that the password complexity is not enough, as long as two users are created at the same time, the password complexity does not meet the requirements. It can be created at the same time when the password complexity plug-in is not used. Two users), normally both users should fail to be created. But actually app@'%'the user was created successfully.
At this point, we understand that the article begins to describe the cause of the failure: the database instance has enabled the password complexity plug-in, and used CREATE USER to create two users at the same time. An error was reported because the password complexity did not meet the requirements, but the user app@'%'was test@'%'. , and then execute the GRANT statement to authorize two users at the same time. At this time, because test@'%'the user does not exist, the partial execution of the GRANT statement results in a master-slave replication error.
troubleshooting
When a 1590 error occurs in master-slave replication, you can first parse the binlog to find INCIDENT_EVENT (search keyword LOST_EVENTS) and the corresponding GTID, and then resume master-slave replication by skipping this GTID. Before skipping the GTID, the data needs to be completed first, because the master database has a user who has been authorized successfully, and the authorization of the slave database is not executed. The specific operation is as follows (executed from the library):
mysql>set global super_read_only=0;
mysql>set sql_log_bin=0;
mysql>grant all on test.* to test@'10.186.63.5';
To skip the GTID corresponding to the error report, the specific operation is as follows:
Parse binlog to find INCIDENT_EVENT.
mysqlbinlog --no-defaults -vv --base64-output=decode-rows mysql-bin.000016 > /root/bin.log
Skip LOST_EVENTSthe corresponding GTID (executed from the library):
stop slave sql_thread;
set gtid_next='9d2e7089-2774-11ee-99d6-02000aba3f05:4198564';
begin;commit;
set gtid_next='automatic';
start slave sql_thread;
Summarize
- When only part of the permission change operation is processed and an error occurs, it will cause an INCIDENT_EVENT to be written in the binlog, which will cause an error in the master-slave replication.
- When using the password complexity plug-in, use the CREATE statement to create two users at the same time, and one user will be successfully created while the other user will fail.
suggestion
- When the password complexity plug-in is used, a CREATE statement only creates one user when creating a user.
- When authorizing, a GRANT statement authorizes only one user to prevent partial authorization success due to permission errors. For more technical articles, please visit: https://opensource.actionsky.com/
About SQLE
The SQLE of the Akson open source community is a SQL auditing tool for database users and managers, which supports multi-scenario auditing, supports standardized online processes, natively supports MySQL auditing, and has scalable database types.
SQLE get
| type | address |
|---|---|
| Repository | https://github.com/actiontech/sqle |
| document | https://actiontech.github.io/sqle-docs/ |
| release news | https://github.com/actiontech/sqle/releases |
| Data audit plug-in development documentation | https://actiontech.github.io/sqle-docs/docs/dev-manual/plugins/howtouse |