Some time ago, a tester contacted me and asked for help. Let me take a look. A mysql database in the test environment was constantly restarted, causing their testing to fail. The time delay will affect the progress of the launch.
Problem Description
In a test environment, mysql5.7 is deployed on a centos7.4 system. Testers use LoadRunner to perform a stress test of the application. The stress test concurrency is 128. After a few seconds after starting LoadRunner, the LoadRunner log reports an error, indicating that the mysql service is broken. It was turned on, recovered after a few seconds, and then disconnected again, which was very strange.
Troubleshoot and locate
So I logged in to the server and took a look at the memory of the database server. The memory was used up. Then I looked at the centos system log (/var/log/messages). In this log, I found that the mysql service was given by the system due to OOM. Kill is dropped, and then because the mysql service has a daemon process, it is automatically started again.
An experienced MySQL database operation and maintenance personnel should soon know what causes OOM.
First: The buffer pool memory parameter configuration of the
mysql database is unreasonable. Second: The session initialization memory parameter configuration of the mysql database is unreasonable.
Third: The configuration of the total number of database connections is unreasonable.
The database server memory of the test environment is 4G, so I won’t talk about it here. Why do testers do stress testing on this server? In fact, when doing application stress testing, you should use an approved stress testing environment instead of just finding an environment. Stress test, let's not talk about it.
Simulation review
Test environment
mysql5.7, centos7.7, memory 2G
In order to better simulate and reproduce the above failures, here is to close the swap of the simulation environment, the closing steps are as follows
[root@localhost data]# free -m
total used free shared buff/cache available
Mem: 2124 565 640 9 917 1398
Swap: 2047 0 2047
[root@localhost data]# swapon -s
文件名 类型 大小 已用 权限
/dev/dm-1 partition 2097148 264 -2
[root@localhost data]# swapoff /dev/dm-1
[root@localhost data]# free -m
total used free shared buff/cache available
Mem: 2124 254 944 9 925 1709
Swap: 0 0 0
From the result of free -m above, you can see that swap has become 0.
Release the memory occupied by the cache
[root@localhost data]# sync
[root@localhost data]# echo 1 > /proc/sys/vm/drop_caches ;
[root@localhost data]# free -m
total used free shared buff/cache available
Mem: 2124 254 1784 9 84 1748
Swap: 0 0 0
It can be seen that the current remaining memory is 1784M, and the memory occupied by the mysql database is mainly 2 large blocks.
First: the buffer pool occupies, and the
second: the memory occupied by the initial connection
Set the mysql buffer pool to 1500M here, and the session parameters are set as follows
read_buffer_size = 32M
read_rnd_buffer_size = 32M
sort_buffer_size = 32M
tmp_table_size = 32M
max_heap_table_size=32M
join_buffer_size=32M
Then 5 connections started to do big query operations, and it didn’t take long for the mysql process to be killed because of OOM
Aug 31 05:37:40 localhost kernel: Out of memory: Kill process 2534 (mysqld) score 658 or sacrifice child
Aug 31 05:37:40 localhost kernel: Killed process 2534 (mysqld), UID 1001, total-vm:1825792kB, anon-rss:654388kB, file-rss:0kB, shmem-rss:0kB
The mysql daemon starts to start the mysql service
/u02/mysql/bin/mysqld_safe: 行 198: 2534 已杀死 nohup /u02/mysql/bin/mysqld --defaults-file=/u02/conf/my3308.cnf --basedir=/u02/mysql --datadir=/u02/data/3308 --plugin-dir=/u02/mysql/lib/plugin --user=mysql --log-error=/u02/log/3308/error.log --open-files-limit=65535 --pid-file=/u02/run/3308/mysqld.pid --socket=/u02/run/3308/mysql.sock --port=3308 < /dev/null > /dev/null 2>&1
2020-08-30T21:37:40.375749Z mysqld_safe Number of processes running now: 0
2020-08-30T21:37:40.407781Z mysqld_safe mysqld restarted
2020-08-30T21:37:40.666886Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-08-30T21:37:40.667059Z 0 [Note] --secure-file-priv is set to NULL. Operations related to importing and exporting data are disabled
2020-08-30T21:37:40.667112Z 0 [Note] /u02/mysql/bin/mysqld (mysqld 5.7.26-log) starting as process 2954 ...
2020-08-30T21:37:40.782412Z 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
2020-08-30T21:37:40.782684Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-08-30T21:37:40.782729Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-08-30T21:37:40.782754Z 0 [Note] InnoDB: Uses event mutexes
2020-08-30T21:37:40.782772Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-08-30T21:37:40.782788Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-08-30T21:37:40.782841Z 0 [Note] InnoDB: Adjusting innodb_buffer_pool_instances from 8 to 1 since innodb_buffer_pool_size is less than 1024 MiB
2020-08-30T21:37:40.784518Z 0 [Note] InnoDB: Number of pools: 1
2020-08-30T21:37:40.784865Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-08-30T21:37:40.789314Z 0 [Note] InnoDB: Initializing buffer pool, total size = 512M, instances = 1, chunk size = 128M
2020-08-30T21:37:40.834948Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-30T21:37:40.843612Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-30T21:37:40.859028Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-08-30T21:37:40.863176Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 5707394229
2020-08-30T21:37:40.863221Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 5707394238
2020-08-30T21:37:40.863231Z 0 [Note] InnoDB: Database was not shutdown normally!
2020-08-30T21:37:40.863239Z 0 [Note] InnoDB: Starting crash recovery.
2020-08-30T21:37:40.901955Z 0 [Note] InnoDB: Last MySQL binlog file position 0 43848, file name binlog.000025
2020-08-30T21:37:41.075805Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-08-30T21:37:41.075860Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-08-30T21:37:41.075952Z 0 [Note] InnoDB: Setting file '/u02/log/3308/iblog/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-08-30T21:37:41.254016Z 0 [Note] InnoDB: File '/u02/log/3308/iblog/ibtmp1' size is now 12 MB.
2020-08-30T21:37:41.255390Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-08-30T21:37:41.255421Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-08-30T21:37:41.256171Z 0 [Note] InnoDB: Waiting for purge to start
2020-08-30T21:37:41.307237Z 0 [Note] InnoDB: 5.7.26 started; log sequence number 5707394238
2020-08-30T21:37:41.308291Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-08-30T21:37:41.310625Z 0 [Note] InnoDB: Loading buffer pool(s) from /u02/log/3308/iblog/ib_buffer_pool
2020-08-30T21:37:41.310785Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200831 5:37:41 (/u02/log/3308/iblog/ib_buffer_pool was empty)
2020-08-30T21:37:41.314568Z 0 [Note] Recovering after a crash using /u02/log/3308/binlog/binlog
2020-08-30T21:37:41.314730Z 0 [Note] Starting crash recovery...
2020-08-30T21:37:41.314842Z 0 [Note] Crash recovery finished.
2020-08-30T21:37:41.346280Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-08-30T21:37:41.346337Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-08-30T21:37:41.349079Z 0 [Warning] CA certificate ca.pem is self signed.
2020-08-30T21:37:41.349341Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-08-30T21:37:41.350297Z 0 [Note] Server hostname (bind-address): '0.0.0.0'; port: 3308
2020-08-30T21:37:41.350399Z 0 [Note] - '0.0.0.0' resolves to '0.0.0.0';
2020-08-30T21:37:41.350475Z 0 [Note] Server socket created on IP: '0.0.0.0'.
2020-08-30T21:37:41.376794Z 0 [Note] Failed to start slave threads for channel ''
2020-08-30T21:37:41.397237Z 0 [Note] Event Scheduler: Loaded 0 events
2020-08-30T21:37:41.397480Z 0 [Note] /u02/mysql/bin/mysqld: ready for connections.
Version: '5.7.26-log' socket: '/u02/run/3308/mysql.sock' port: 3308 Sour
The connected session is automatically disconnected
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql>
When configuring mysql parameters, you must consider the following 3 factors:
1. The expected total number of business connections
2. Session initialization memory
3. Buffer pool buffer size
Let's first use sql to query the total memory size of the session and the size of the database buffer pool
mysql> select (@@innodb_buffer_pool_size+@@innodb_log_buffer_size+@@key_buffer_size) / 1024 /1024 AS MEMORY_MB;
+--------------+
| MEMORY_MB |
+--------------+
| 584.00000000 |
+--------------+
1 row in set (0.00 sec)
mysql> select (@@read_buffer_size+@@read_rnd_buffer_size+@@sort_buffer_size+@@tmp_table_size+@@join_buffer_size+@@binlog_cache_size)/1024/1024 as MB;
+--------------+
| MB |
+--------------+
| 160.03125000 |
+--------------+
1 row in set (0.00 sec)
Suggest
The total size of mysql's buffer pool plus session memory does not exceed 80% of the total size of server memory.
Assuming that the server memory is 32G, the mysql database memory setting should not exceed 26G. If the buffer pool is set to 20G and each session memory is 160M, 6G can provide 6 *1024 /160 = 38 secure connections. If it exceeds, it may cause Insufficient memory, use swap partition.