Solve the returned OS error 124 caused by backing up mysql with innobackupex
xtrabackup reports an error Too many open files Troubleshooting
1. Background
The customer reported that the database backup failed.
2. Environmental description
[root@mes-node1 ~]# mysql -V mysql Ver 14.14 Distrib 5.7.24, for Linux (x86_64) using EditLine wrapper [root@mes-node1 ~]# innobackupex --version xtrabackup: recognized server arguments: --server-id=21 --log_bin=mysql-bin --datadir=/home/lib/mysql innobackupex version 2.4.20 Linux (x86_64) (revision id: c8b4056)
3. Troubleshooting
1. Check the backup script
[root@mes-node1 xtrabackup_full]# crontab -l 17 2 * * * /home/mysql_bak/backup.sh >>/home/mysql_bak/backup.log 0 4 * * * /home/xtra_backup.sh [root@mes-node1 ~]# cat /home/xtra_backup.sh /usr/local/xtrabackup/bin/innobackupex -uroot -phzjs1234 -S /home/lib/mysql/mysql.sock /home/xtrabackup_full find /home/xtrabackup_full -mtime +1 -exec rm -rf {} \;
2. Check the backup results
[root@mes-node1 xtrabackup_full]# ls -lrth total 0 drwxr-x--- 2 root root 32 Mar 21 04:00 2023-03-21_04-00-01 drwxr-x--- 2 root root 32 Mar 22 04:00 2023-03-22_04-00-02 [root@mes-node1 xtrabackup_full]# du -sh * 200K 2023-03-21_04-00-01 200K 2023-03-22_04-00-02 [root@mes-node1 xtrabackup_full]# ll 2023-03-21_04-00-01 total 200 -rw-r----- 1 root root 201216 Mar 21 04:00 xtrabackup_logfile
3. Manually perform the backup
[root@mes-node1 ~]# /usr/local/xtrabackup/bin/innobackupex -uroot -phzjs1234 -S /home/lib/mysql/mysql.sock /home/xtrabackup_full
InnoDB: Number of pools: 1 230322 13:38:14
>> log scanned up to (427884628939) xtrabackup: Generating a list of tablespaces InnoDB: Allocated tablespace ID 3463 for mysql/servers, old maximum was 0 InnoDB: Operating system error number 24 in a file operation. InnoDB: Error number 24 means 'Too many open files' InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html InnoDB: File ./quality@002dserver/spc_process_trace_data_T15.ibd: 'open' returned OS error 124. Cannot continue operation InnoDB: Cannot continue operation.
Analysis: The backup reports an error and throws an error message InnoDB: Error number 24 means 'Too many open files'.
4. Check the database open_files_limit parameter
mysql> show variables like 'open_files_limit';
+------------------+-------+ | Variable_name | Value | +------------------+-------+ | open_files_limit | 5000 |
+------------------+-------+ 1 row in set (0.01 sec)
5. Count how many files the current database needs to open
[root@mes-node1 ~]# find /home/lib/mysql -name "*.ibd" |wc -l 1114
Analysis: The operating parameter value of open_files_limit of the instance is 5000, which is greater than the number of files that need to be opened during backup, and there is no problem with the parameter setting.
6. View the current number of open files in the OS
[root@mes-node1 xtrabackup_full]# ulimit -a
Analysis: The number of open files is 1024, which is less than the number of files that need to be opened during backup. The number is not enough and needs to be modified.
7. Modify the number of OS open files
[root@mes-node1 ~]# vi /etc/security/limits.conf
* soft nproc 65536
* hard nproc 65536
* soft nofile 65536
* hard nofile 65536
-- 重新登录session
[root@mes-node1 ~]# ulimit -a|grep open open files (-n) 65536
8. Perform the backup manually again
[root@mes-node1 ~]# /usr/local/xtrabackup/bin/innobackupex -uroot -phzjs1234 -S /home/lib/mysql/mysql.sock /home/xtrabackup_full
Conclusion: The backup is successful and the problem is solved.