The system can't get in, and it is very slow to connect to the server using ssh. The load balancing shows that the back-end connection is abnormal. After restarting the mysql database, it is found that it is often restarted, or it is directly shut down, and the access page cannot be accessed.
Check the mysql log file and find no problem,
Check the log of nginx.error and find an error
018/04/23 16:44:42 [crit] 11182#0: accept4() failed (23: Too many open files in system)
2018/04/23 16:44:42 [crit] 11182#0: accept4( ) failed (23: Too many open files in system)
2018/04/23 16:44:47 [crit] 11185#0: accept4() failed (23: Too many open files in system)
2018/04/23 16: 44:47 [crit] 11185#0: accept4() failed (23: Too many open files in system )
because the system limits the open files
ulimit -n View the number of file descriptions for the current user
The value is displayed as 1024, which means that the number of open files is limited to 1024. You can make the number larger and make the website access more concurrent.
The method of modifying the ulimit limit number:
.First you have to modify the nginx.conf configuration file and add a line where the error.log log path is defined
vi nginx.conf
worker_rlimit_nofile 65535;
2. Add the following at the end of the /etc/bashrc file
ulimit -n 65535
3. Add the following at the end of the /etc/security/limits.conf file
* soft nofile 65535
* hard nofile 65535
* 代表所有用户,如果想代表某个用户的话，则 user soft/hard nofile 65535
soft代表软连接 hard代表硬限制
4. For the limits.conf file configuration to take effect, you must ensure that the pam_limits.so file is added to the startup file在/etc/bashrc后面加上ulimit -n 65535
If you view the mysql file and report an error:
101029 16:09:24 [ERROR] Error in accept: Too many open files
101029 16:17:20 [ERROR] Error in accept: Too many open files
101029 16:21:37 [ERROR] Error in accept: Too many open files
101029 16:25:53 [ERROR] Error in accept: Too many open files
101029 16:30:09 [ERROR] Error in accept: Too many open files
/proc/sys/fs/file-max
This is the maximum number of files allocated by system resources. The setting value is related to the memory size. In the early days when ram was very expensive, this value was usually not too large. Therefore, if the number of files opened by MySQL is too large, it may indeed be limited by this value. , but now there are several G of memory , this value is set to more than 200,000 by default on my linux system , so the problem is not this value. ( If you really want to modify this value, you can use sysctl -w fs.file-max=##### to modify it, but it will be automatically changed back after the system is restarted, and it can be written to /etc/rc.local to execute at boot )
cat /proc/sys/fs/file-max
203457
sysctl -w fs.file-max=406914
fs.file-max=406914
cat /proc/sys/fs/file-max
406914
vi /etcc/rc.local
sysctl -w fs.file-max=
Add this file to ensure that it takes effect when it is restarted
cat /etc/security/limits.conf | grep mysql
mysql soft nofile 24000
mysql hard nofile 32000
ulimit -n
ulimit can check the resource size of each shell . The -n parameter is written in the man page The maximum number of open file descriptors . In other words, in the shell where mysql is located , the files that can really be opened are only the value of ulimit -n , On my system, the value of ulimit -n is only 1024 , so even if there are hundreds of thousands of fs.file-max , the mysql shell can only open 1024 , which is the key.
The three parameters of table_open_cache, max_connections, open_files_limit in my.cnf
table_open_cache refers to the number of cache files for mysql to open a table. Generally, when mysql opens a table, it will open two files, *.MYI and *.MYD. For example, if we use phpMyAdmin to open a DB with 100 tables, mysql will cache 200 of them. files. (default: 64)
open_files_limit : The maximum number of files mysqld can open. (default: 0)
max_connections : The maximum number of connections. (default: 100)
mysqld will cache after opening a file. How many files does it have to open before releasing the cached files?
Then it depends on the value of table_cache, max_connections, open_files_limit in my.cnf ,
If: the value of open_files_limits is 0 , it depends on the value calculated by table_cache and max_connections through a function;
table_cache * 2 + max_connections
If the value of open_files_limits is not 0 , it should depend on the size setting of this value.
Whether we want to set our mysql open_files_limit or use table_cache * 2 + max_connections , we should pay attention to the value of ulimit -n is the correct solution, which is less related to fs.file-max .
To check the number of files opened by mysql , first use ps aux | grep mysql to see the mysql PID , and then use lsof -p PID# | wc -l for statistics.