1. Description of the problem
An error is reported when the backup program executes the rollforward. (-apply-log)
InnoDB: Errornumber 24 means 'Too many open files'.
InnoDB: Some operatingsystem error numbers are described at
InnoDB: http://dev.mysql.com/doc/mysql/en/Operating_System_error_codes.html
InnoDB: File name/home/nice/backup/mysql/var.20121211021656/user_db_20121123/tuser_like_core.ibd
InnoDB: File operationcall: 'open'.
InnoDB: Cannot continueoperation.
innobackup:Error:
2. Problem analysis
openfiles is too small.
MySQL's open_file_limit is subject to the following parameters
max_connections, table_open_cache, open_files_limit, and configuration in /etc/security/limits.conf.
The relationship between max_connections and table_open_cache and open_files_limit:
max_1 = 10 + max_connections + table_cache * 2;
max_2 = max_connections * 5;
max_3 = max_os_open_files;//The maximum number of open file handles (file descriptors) allowed by a single process of the operating system.
open_files_limit= max( max_1, max_2 ) > max_3 ? max_3 : max ( max_1, max_2);
Comprehensive analysis found that open_files_limit | 10240
At present, there are already 10,800 ibd tables.
3. Solve the problem
1. my.cnf configure open_files_limit = 20480
Restart mysql and find no change, open_files_limit | 10240.
2.ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 137216
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
3. vi /etc/security/limits.conf, add
mysql soft nofile 8192
mysql hard nofile 20480
4. After the first three steps, restart mysql.
Restart mysql and find that open_files_limit | 20480.
5. Modify the OS user profile
After the first four steps, although I see open_files_limit | 20480 in mysql, ibbackup still reports an error when rolling forward. Report, open_file is not enough. Doubt it still doesn't work.
ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 137216
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
So, modify the OS user profile
Checked .bash_profile , .bashrc , /etc/bashrc respectively；
Found that .bash_profile calls .bashrc and .bashrc calls /etc/bashrc
Found that /etc/bashrc has
ulimit -S -c unlimited > /dev/null 2>&1
add after this line
ulimit -n 20480 >/dev/null 2>&1
6. Restart mysql, execute the backup program, in --apply-log, still report an error, report, open_file is not enough
InnoDB: Errornumber 24 means 'Too many open files'.
InnoDB: Some operatingsystem error numbers are described at
InnoDB: http://dev.mysql.com/doc/mysql/en/Operating_System_error_codes.html
7. Modify the innodb_open_files parameter
Previously, this parameter was set to 500. This time, it was changed to 1000.
Restart mysql to solve the problem
4. About Operating_System_error_codes.html
Linux System Error Codes
The following tableprovides a list of some common Linux system error codes. For a more completelist, see Linux source code.
Number |
Macro |
Description |
1 |
EPERM |
Operation not permitted |
2 |
ENOENT |
No such file or directory |
3 |
ESRCH |
No such process |
4 |
ENTRY |
Interrupted system call |
5 |
EIO |
I/O error |
6 |
ENXIO |
No such device or address |
7 |
E2BIG |
Arg list too long |
8 |
ENOEXEC |
Exec format error |
9 |
EBADF |
Bad file number |
10 |
ECHILD |
No child processes |
11 |
EAGAIN |
Try again |
12 |
ENAMEM |
Out of memory |
13 |
EACCES |
Permission denied |
14 |
EFAULT |
Bad address |
15 |
ENOTBLK |
Block device required |
16 |
EBUSY |
Device or resource busy |
17 |
EEXIST |
File exists |
18 |
EXDEV |
Cross-device link |
19 |
ENODEV |
No such device |
20 |
ENOTDIR |
Not a directory |
21 |
EISDIR |
Is a directory |
22 |
EINVAL |
Invalid argument |
23 |
PUT ON |
File table overflow |
24 |
HE IS DEAD |
Too many open files |
25 |
ENOTTY |
Inappropriate ioctl for device |
26 |
ETXTBSY |
Text file busy |
27 |
EFBIG |
File too large |
28 |
ENOSPC |
No space left on device |
29 |
ESPIPE |
Illegal seek |
30 |
EROFS |
Read-only file system |
31 |
EMLINK |
Too many links |