Author: Li Xichao
A database engineer of Jiangsu Suning Bank who loves to laugh, mainly responsible for the daily operation and maintenance of the database, automation construction, and DMP platform operation and maintenance. Good at MySQL, Python, Oracle, like cycling and research technology.
Source of this article: original contribution*Produced by the Aikesheng open source community, the original content is not allowed to be used without authorization, please contact the editor and indicate the source for reprinting.
The R&D students reported that the business system related to the MySQL database in a system performance test environment is running normally, but there are a large number of warning logs, and it is necessary to cooperate with the analysis of the reasons.
1. Abnormal phenomena
There are a large number of the following information in the mysql error log file:
2023-01-10T01:07:23.035479Z 13 [Warning] [MY-013360] [Server] Plugin sha256_password reported: ''sha256_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'
key environmental information
2. Preliminary analysis
When seeing the above warning log, according to empirical theory, the first reaction should be that the version of the client is too low, and its authentication plug-in is a version that the server will discard, so the above warning message is generated. Especially some common client tools may easily trigger this problem due to the update frequency.
try to reproduce
According to the preliminary analysis suggestions, after communicating the preliminary analysis suggestions with the R&D students, access the database through common database tools to see if the error can be reproduced. However, through common database users in the database and accessing the database through different tools, the exception was not triggered at the time of access.
Therefore, the first attempt to reproduce failed. Is it because of other reasons?
In the process of trying to access for the first time, observe the database error log in real time. In the process of trying to access with the client, the error was not reproduced. But still see the corresponding warning log is continuously output to the error log file. Moreover, the frequency is high and the interval time is fixed, which also proves that the error is not manually accessed by database tools.
The application system is running normally, and it is not caused by the client! As a DBA, how should you further analyze it?
Early tricks
Due to the test environment, for this error, you can perform the following operations to enable MySQL's general logs:
-- 开通一般日志：
show variables like 'general_log';
set global general_log=on;
show variables like 'general_log';
-- 查看一般日志路径：
show variables like 'general_log_file';
After enabling the log, observe the error log and find the following records in the general log:
Tip: After finding an exception, close the general log immediately to avoid generating too many logs and exhausting disk space:
-- 开通一般日志：
show variables like 'general_log';
set global general_log=off;
show variables like 'general_log';
That is, the user dbuser2 initiates a request to access the database from the 10.xy43 server at the time of the problem. After confirming the abnormal access user and server, check the database mysql.user table, skip-grant-tables and other configurations, and find that the user does not exist in the database, and the authorization table and other configurations are not skipped. Using this user will not be able to log in to the database.
Feedback the information to the research and development students, and soon confirmed that some applications were configured unreasonably, using non-existing database users, and regularly connecting to the database to perform tasks. So after the research and development students modified the configuration, the warning log was no longer generated.
So the analysis of this problem is here, can it be over?
After modifying the configuration, the warning log is no longer happening! But since it is a non-existent user, why is it prompted that the authentication plug-in will be discarded when accessing it?
3. Source code analysis
With questions, the first thing that comes to mind is: since the database user does not exist in the mysql.user table, login will also generate a warning. Is this user an internal user of mysql and is hard-coded! So get the source code of the corresponding version and confirm it with the following command:
cd mysql-8.0.27/
grep -rwi "dbuser2" *
The access result is empty, that is, there is no guessed "internal user".
Normal login authentication logic
Since there is no hard coding, it can only be caused by internal logic. So first of all, for the mysql user login process under normal circumstances, the source code analysis results are as follows:
|—> handle_connection
|—> thd_prepare_connection
|—> login_connection
|—> check_connection
// 判断客户端的主机名是否可以登录(mysql.user.host)，如果 mysql.user.host 有 '%' 那么将 allow_all_hosts，允许所有主机。
|—> acl_check_host
|—> acl_authenticate
|—> server_mpvio_initialize // 初始化mpvio对象，包括赋值 mpvio->ip / mpvio->host
|—> auth_plugin_name="caching_sha2_password"
|—> do_auth_once
|—> caching_sha2_password_authenticate // auth->authenticate_user
|—> server_mpvio_read_packet // vio->read_packet(vio, &pkt) // pkt=passwd
|—> parse_client_handshake_packet
|—> char *user = get_string(&end, &bytes_remaining_in_packet, &user_len);
|—> passwd = get_length_encoded_string(&end, &bytes_remaining_in_packet, &passwd_len);
|—> mpvio->auth_info.user_name = my_strndup(key_memory_MPVIO_EXT_auth_info, user, user_len, MYF(MY_WME))
// 根据 user 搜索 mysql.user.host ,并与客户端的 hostname/ip 进行比较：匹配记录后，赋值 mpvio->acl_user
|—> find_mpvio_user(thd, mpvio)
|—> list = cached_acl_users_for_name(mpvio->auth_info.user_name); // 根据 user 搜索 mysql.user.host
|—> acl_user_tmp->host.compare_hostname(mpvio->host, mpvio->ip) // 与客户端的 hostname/ip 进行比较
|—> mpvio->acl_user_plugin = mpvio->acl_user->plugin; // 赋值 acl_user_plugin 属性为用户的plugin名
|—> mpvio->auth_info.multi_factor_auth_info[0].auth_string = mpvio->acl_user->credentials[PRIMARY_CRED].m_auth_string.str;
|—> mpvio->auth_info.auth_string = mpvio->auth_info.multi_factor_auth_info[0].auth_string;
|—> if (my_strcasecmp(system_charset_info, mpvio->acl_user_plugin.str,plugin_name(mpvio->plugin)->str) != 0)
|—> my_strcasecmp(system_charset_info, client_plugin,user_client_plugin_name) //检查客户端的认证插件与用户插件是否相同
|—> make_hash_key(info->authenticated_as, hostname ? hostname : nullptr, authorization_id); // 生成 authorization_id = user1\000%
|—> g_caching_sha2_password->fast_authenticate(authorization_id,*scramble,20,pkt,false) // 进行快速授权操作
|—> m_cache.search(authorization_id, digest) // 根据 user、host 搜索密码，赋值到digest
|—> Validate_scramble validate_scramble_first(scramble, digest.digest_buffer[0], random, random_length);
|—> validate_scramble_first.validate(); // 校验 scramble
// 如验证成功
|—> vio->write_packet(vio, (uchar *)&fast_auth_success, 1)
|—> return CR_OK;
// 否则进行进行慢授权操作
|—> g_caching_sha2_password->authenticate( authorization_id, serialized_string, plaintext_password);
|—> server_mpvio_update_thd(thd, &mpvio);
|—> check_and_update_password_lock_state(mpvio, thd, res);
// 继续其它授权操作
That is, the core authentication operation is performed in the function caching_sha2_password_authenticate(), first call the function find_mpvio_user(), find the configured user through user and hostname, and then call the function fast_authenticate() to quickly verify the password.
Use nonexistent user authentication logic
When the user does not exist, the mysql user login process, the source code analysis results are as follows:
|—> handle_connection
|—> thd_prepare_connection
|—> login_connection
|—> check_connection
// 判断客户端的主机名是否可以登录(mysql.user.host)，如果 mysql.user.host 有 '%' 那么将 allow_all_hosts，允许所有主机。
|—> acl_check_host
|—> acl_authenticate
|—> server_mpvio_initialize // 初始化mpvio对象，包括赋值 mpvio->ip / mpvio->host
|—> auth_plugin_name="caching_sha2_password"
|—> do_auth_once
|—> caching_sha2_password_authenticate // auth->authenticate_user
|—> server_mpvio_read_packet // vio->read_packet(vio, &pkt) // pkt=passwd
|—> parse_client_handshake_packet
|—> char *user = get_string(&end, &bytes_remaining_in_packet, &user_len);
|—> passwd = get_length_encoded_string(&end, &bytes_remaining_in_packet, &passwd_len);
|—> mpvio->auth_info.user_name = my_strndup(key_memory_MPVIO_EXT_auth_info, user, user_len, MYF(MY_WME))
|—> find_mpvio_user(thd, mpvio)
|—> list = cached_acl_users_for_name(mpvio->auth_info.user_name); // 根据 user 搜索 mysql.user.host, 由于用户不存在，搜索不到记录
|—> mpvio->acl_user = decoy_user(usr, hst, mpvio->mem_root, mpvio->rand, initialized); //
|—> Auth_id key(user);
// 判断是否用户存在于 unknown_accounts
|—> unknown_accounts->find(key, value)
// 如存在：
|—> user->plugin = Cached_authentication_plugins::cached_plugins_names[value];
// 如不存在：
|—> const int DECIMAL_SHIFT = 1000;
|—> const int random_number = static_cast<int>(my_rnd(rand) * DECIMAL_SHIFT);
|—> uint plugin_num = (uint)(random_number % ((uint)PLUGIN_LAST));
|—> user->plugin = Cached_authentication_plugins::cached_plugins_names[plugin_num];
|—> unknown_accounts->insert(key, plugin_num)
|—> mpvio->acl_user_plugin = mpvio->acl_user->plugin; // 赋值 acl_user_plugin 属性为用户的plugin名
|—> mpvio->auth_info.multi_factor_auth_info[0].auth_string = mpvio->acl_user->credentials[PRIMARY_CRED].m_auth_string.str; // ""
|—> mpvio->auth_info.auth_string = mpvio->auth_info.multi_factor_auth_info[0].auth_string; // ""
|—> mpvio->auth_info.additional_auth_string_length = 0; // 0
|—> mpvio->auth_info.auth_string_length = mpvio->auth_info.multi_factor_auth_info[0].auth_string_length; // 0
|—> if (my_strcasecmp(system_charset_info, mpvio->acl_user_plugin.str,plugin_name(mpvio->plugin)->str) != 0)
|—> return packet_error;
|—> if (pkt_len == packet_error) goto err;
|—> return -1;
|—> auth_plugin_name = mpvio.acl_user->plugin;
|—> res = do_auth_once(thd, auth_plugin_name, &mpvio);
|—> sha256_password_authenticate() //auth->authenticate_user(mpvio, &mpvio->auth_info);
|—> LogPluginErr // Deprecate message for SHA-256 authentication plugin.
// 打印: 2023-01-10T01:07:23.035479Z 13 [Warning] [MY-013360] [Server] Plugin sha256_password reported: ''sha256_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'
|—> server_mpvio_read_packet() // vio->read_packet(vio, &pkt)
|—> if (info->auth_string_length == 0 && info->additional_auth_string_length == 0) // info -> auth_info
|—> return CR_ERROR;
|—> return res; // 0
|—> server_mpvio_update_thd(thd, &mpvio);
|—> check_and_update_password_lock_state(mpvio, thd, res); // 直接返回
...
|—> login_failed_error // 打印登录报错信息
// 2023-01-10T02:02:44.659796Z 19 [Note] [MY-010926] [Server] Access denied for user 'user2'@'localhost' (using password: YES)
|—> thd->send_statement_status(); // 客户端终止
That is, the acl_user object created by the function decoy_user() when a non-existing user is used to log in to the database. When this object is created, its plugin attribute is randomly selected from the cached_plugins_enum. Thus it is possible to select the PLUGIN_SHA256_PASSWORD plugin. However, at the entry of the function sha256_password_authenticate(), a Warning level prompt will be generated to indicate that the authentication PLUGIN_SHA256_PASSWORD will be discarded. Subsequently, since the length of the acl_user object auth_string_length created in decoy_user() is not 0, it will directly return CR_ERROR in the subsequent authentication logic, that is, the authentication fails.
root cause summary
According to the above certified analysis, the root cause of PLUGIN_SHA256_PASSWORD being discarded in the error log is: in the current version, when using a non-existent user to log in to the database, mysql will randomly select the user's password authentication plug-in, in the current version , there is a 1/3 probability that the PLUGIN_SHA256_PASSWORD plug-in will be selected. After selecting this plugin, the subsequent authentication logic will trigger the generation of warning logs.
4. Problem solving
Based on the above analysis process, the direct cause of this problem is that the application configures a non-existent database user, and the root cause is that there are certain defects in the database login authentication logic. So to solve this problem, you can refer to the following solutions:
1. Referring to the solution in the preliminary analysis, modify the connection configuration of the application to the correct user information;
2. You can filter the alarm through parameters in the mysql database to avoid inputting the alarm information into the error log file. The relevant configuration is as follows:
show variables like 'log_error_suppression_list';
set global log_error_suppression_list='MY-013360;
show variables like 'log_error_suppression_list';
Note that using this scheme will also result in an alert that exists and uses the SHA256_PASSWORD authentication plugin. Can be used as a temporary solution;
3. Modify the mysql code to avoid selecting the SHA256_PASSWORD authentication plug-in when logging in to the database with a non-existing user. Bug #109635 is currently filed for this solution.
Attachment: key function location
find_mpvio_user() (./sql/auth/sql_authentication.cc:2084)
parse_client_handshake_packet() (./sql/auth/sql_authentication.cc:2990)
server_mpvio_read_packet() (./sql/auth/sql_authentication.cc:3282)
caching_sha2_password_authenticate() (./sql/auth/sha2_password.cc:955)
do_auth_once() (./sql/auth/sql_authentication.cc:3327)
acl_authenticate() (./sql/auth/sql_authentication.cc:3799)
check_connection() (./sql/sql_connect.cc:651)
login_connection() (./sql/sql_connect.cc:716)
thd_prepare_connection() (./sql/sql_connect.cc:889)
handle_connection() (./sql/conn_handler/connection_handler_per_thread.cc:298)