question:
In a continuous integration environment, such as the continuous construction and destruction of PODs based on the kubersnets platform, in this environment, problems such as the failure to connect to the Mysql database and the excessive number of connections occur.
Cause Analysis
(1) First check the following two types of global variables of mysql, one is the timeout variable, as shown below
mysql> show variables like "%timeout%";
Another category is the connection number variable
mysql> show variables like "%connections%";
(2) Check the current number of connections again
mysql> SELECT substring_index(host, ':',1) AS host_name,state,count(*) FROM information_schema.processlist GROUP BY state,host_name;
(3) Through the comparison and analysis of the above three sets of data, it can be found that in the continuous integration environment, such as based on the k8s platform, because the POD of the application service will be continuously destroyed and newly created, and each time it is newly created, the service will establish several When the POD is destroyed, the mysql connection of the client is broken, but the connection of the server is not broken. After that, the connection of the mysql server will remain idle, waiting for the wait_timeout to time out, and the problem is also here. The default wait_timeout of mysql is set to 8 The hour is 28800 seconds, which causes the established connection to continue to survive for 8 hours. In this way, the new POD deployed in the continuous integration environment continuously establishes new connections, and then check the configuration of the maximum number of connections in the database. Mysql configures the maximum number of connections by default. It is 151. At this point in the analysis, the problem is obvious. First, the mysql timeout setting is too long, which is not suitable for mysql in the continuous integration environment. The other maximum number of connections is too small, only 151, which is easy to fill up.
solution
According to the analysis of the cause of the problem, the following solutions can be obtained.
(1) Set the maximum number of connections a little larger
mysql> set global max_connections=1000;
mysql> set global max_user_connections=1000;
mysql> set global mysqlx_max_connections=1000;
(2) Set the timeout to be smaller, such as 600 seconds or ten minutes
mysql> set global wait_timeout=600;
mysql> set global mysqlx_wait_timeout=1000;
mysql> set global interactive_timeout=1000;
mysql> set global mysqlx_interactive_timeout=1000;
At this point, the problem can be solved. Of course, this configuration method is a temporary solution. If you want to solve it completely, you need to modify the corresponding configuration parameters in the myslq configuration file, and then restart the database. The above method of modifying through the mysql command will become invalid after the database is restarted.