Preface
It is well known that Linux does not support high concurrency well under the default parameters, which is mainly limited by the maximum number of open files in a single process, the kernel TCP parameters and the IO event allocation mechanism. The following adjustments will be made from several aspects to enable the Linux system to support a high concurrency environment.
1. Iptables related
If not necessary, turn off or uninstall the iptables firewall and prevent the kernel from loading the iptables module. These modules will affect concurrent performance.
2. Maximum number of open files in a single process
The general release version limits the maximum number of 1024 files that can be opened by a single process, which is far from meeting high concurrency requirements. The adjustment process is as follows:
Type in the # prompt:
# ulimit–n 65535
Set the maximum number of files that can be opened for a single process started by root to 65535. If the system echoes something like "Operationnotpermitted", it means that the above restriction modification failed, in fact, it is because the value specified in exceeds the soft or hard limit of the number of files opened by the Linux system for the user. Therefore, it is necessary to modify the Linux system's soft and hard limits on the number of open files.
The first step is to modify the limits.conf file and add:
# vim /etc/security/limits.conf
* softnofile 65536
* hard nofile65536
Among them, the'*' sign means to modify the limit of all users; soft or hard specifies whether to modify the soft limit or the hard limit; 65536 specifies the new limit value that you want to modify, that is, the maximum number of open files (please note that the soft limit value should be less than Or equal to the hard limit). Save the file after modification.
The second step is to modify the /etc/pam.d/login file and add the following line to the file:
vim /etc/pam.d/login
sessionrequired /lib/security/pam_limits.so
This tells Linux that after the user completes the system login, the pam_limits.so module should be called to set the system's maximum limit on the number of resources that the user can use (including the maximum number of files that the user can open), and the pam_limits.so module It will read the configuration from the /etc/security/limits.conf file to set these limits. Save this file after modification.
The third step is to check the maximum number of open files at the Linux system level, and use the following command:
cat/proc/sys/fs/file-max
32568
This shows that this Linux system allows up to 32568 files to be opened at the same time (that is, including the total number of open files by all users), which is a Linux system-level hard limit, and the limit on the number of open files at all user-level should not exceed this value. Usually this system-level hard limit is the optimal maximum number of open files at the same time calculated according to the system hardware resource status when the Linux system is started. If there is no special need, this limit should not be modified unless you want to limit the number of open files for the user level. Set a value that exceeds this limit. The way to modify this hard limit is to modify fs.file-max = 131072 in the /etc/sysctl.conf file
This is for Linux to forcibly set the system-level hard limit on the number of open files to 131072 after the boot is complete. Save this file after modification.
After completing the above steps, restart the system. Under normal circumstances, you can set the maximum number of files that the Linux system allows for a single process of a specified user to be opened at the same time to the specified value. If the ulimit-n command is used after restarting to see that the limit on the number of files that the user can open is still lower than the maximum value set in the above steps, this may be because the ulimit-n command in the user login script /etc/profile has already opened the user at the same time The number of files is limited. Because when the system limits the maximum number of files that users can open at the same time through ulimit-n, the newly modified value can only be less than or equal to the value set by ulimit-n last time, so it is impossible to increase this limit value with this command of. Therefore, if the above problems exist, you can only open the /etc/profile script file, find in the file whether ulimit-n is used to limit the maximum number of files that the user can open at the same time, if found, delete this command, Or change its set value to an appropriate value, then save the file, and the user can log out and log in to the system again.
Through the above steps, the system limit on the number of open files is lifted for the communication processing program that supports high concurrent TCP connection processing.
3. Kernel TCP parameters
Under Linux, after the TCP connection is disconnected, it will remain in the TIME_WAIT state for a certain period of time before releasing the port. When there are too many concurrent requests, a large number of connections in the TIME_WAIT state will be generated. If they cannot be disconnected in time, a large amount of port resources and server resources will be occupied. At this time, we can optimize the TCP kernel parameters to clear the ports in the TIME_WAIT state in time.
The method described below is only effective for system resource consumption caused by connections with a large number of TIME_WAIT states. If this is not the case, the effect may not be obvious. You can use the netstat command to check the connection status of the TIME_WAIT state, enter the following combined command,
View the status of the current TCP connection and the corresponding number of connections:
# netstat-n | awk ‘/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}’
#这个命令会输出类似下面的结果：
LAST_ACK16
SYN_RECV348
ESTABLISHED70
FIN_WAIT1229
FIN_WAIT230
CLOSING33
TIME_WAIT18098
We only need to care about the number of TIME_WAIT, as you can see here, there are more than 18,000 TIME_WAIT, so more than 18,000 ports are occupied. It is important to know that the number of ports is only 65,535, and occupying one less than one will seriously affect subsequent new connections. In this case, it is necessary to adjust the TCP kernel parameters of Linux to let the system release the TIME_WAIT connection faster.
Edit the configuration file: /etc/sysctl.conf, in this file, add the following lines:
vim /etc/sysctl.conf
net.ipv4.tcp_syncookies= 1#表示开启SYNCookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，默认为0，表示关闭；
net.ipv4.tcp_tw_reuse= 1#表示开启重用。允许将TIME-WAITsockets重新用于新的TCP连接，默认为0，表示关闭；
net.ipv4.tcp_tw_recycle= 1#表示开启TCP连接中TIME-WAITsockets的快速回收，默认为0，表示关闭；
net.ipv4.tcp_fin_timeout= 30#修改系統默认的TIMEOUT 时间。
#输入下面的命令，让内核参数生效：
sysctl-p
After such adjustments, in addition to further increasing the load capacity of the server, it can also defend against DoS, CC, and SYN attacks with low traffic levels.
In addition, if you have a lot of connections, we can optimize the TCP port range to further improve the server's concurrency. Still add the following configuration to the above parameter file:
net.ipv4.tcp_keepalive_time= 1200#表示当keepalive起用的时候，TCP发送keepalive消息的频度。缺省是2小时，改为20分钟。
net.ipv4.ip_local_port_range= 1024 65535#表示用于向外连接的端口范围。缺省情况下很小，改为1024到65535。
net.ipv4.tcp_max_syn_backlog= 8192#表示SYN队列的长度，默认为1024，加大队列长度为8192，可以容纳更多等待连接的网络连接数。
net.ipv4.tcp_max_tw_buckets= 5000#表示系统同时保持TIME_WAIT的最大数量，如果超过这个数字，TIME_WAIT将立刻被清除并打印警告信息。默认为180000，改为5000。此项参数可以控制TIME_WAIT的最大数量，只要超出了。
It is recommended to enable these parameters only on servers with very large traffic, which will have significant effects. Generally, there is no need to set these parameters on servers with low traffic.
4. Description of other TCP parameters of the kernel
net.ipv4.tcp_max_syn_backlog= 65536#记录的那些尚未收到客户端确认信息的连接请求的最大值。对于有128M内存的系统而言，缺省值是1024，小内存的系统则是128。
net.core.netdev_max_backlog= 32768#每个网络接口接收数据包的速率比内核处理这些包的速率快时，允许送到队列的数据包的最大数目。
net.core.somaxconn= 32768#例如web应用中listen函数的backlog默认会给我们内核参数的net.core.somaxconn限制到128，而nginx定义的NGX_LISTEN_BACKLOG默认为511，所以有必要调整这个值。
net.core.wmem_default= 8388608
net.core.rmem_default= 8388608
net.core.rmem_max= 16777216 #最大socket读buffer,可参考的优化值:873200
net.core.wmem_max= 16777216 #最大socket写buffer,可参考的优化值:873200
net.ipv4.tcp_timestsmps= 0#时间戳可以避免序列号的卷绕。一个1Gbps的链路肯定会遇到以前用过的序列号。时间戳能够让内核接受这种“异常”的数据包。这里需要将其关掉。
net.ipv4.tcp_synack_retries= 2#为了打开对端的连接，内核需要发送一个SYN并附带一个回应前面一个SYN的ACK。也就是所谓三次握手中的第二次握手。这个设置决定了内核放弃连接之前发送SYN+ACK包的数量。
net.ipv4.tcp_syn_retries= 2#在内核放弃建立连接之前发送SYN包的数量。
#net.ipv4.tcp_tw_len= 1
net.ipv4.tcp_tw_reuse= 1#开启重用。允许将TIME-WAITsockets重新用于新的TCP连接。
net.ipv4.tcp_wmem= 8192 436600 873200#TCP写buffer,可参考的优化值:8192 436600 873200
net.ipv4.tcp_rmem = 32768 436600 873200#TCP读buffer,可参考的优化值:32768 436600 873200
net.ipv4.tcp_mem= 94500000 91500000 92700000
#同样有3个值,意思是:
#net.ipv4.tcp_mem[0]:低于此值，TCP没有内存压力。
#net.ipv4.tcp_mem[1]:在此值下，进入内存压力阶段。
#net.ipv4.tcp_mem[2]:高于此值，TCP拒绝分配socket。
#上述内存单位是页，而不是字节。可参考的优化值是:7864321048576 1572864
net.ipv4.tcp_max_orphans= 3276800
#系统中最多有多少个TCP套接字不被关联到任何一个用户文件句柄上,如果超过这个数字，连接将即刻被复位并打印出警告信息,这个限制仅仅是为了防止简单的DoS攻击，不能过分依靠它或者人为地减小这个值，更应该增加这个值(如果增加了内存之后)。
net.ipv4.tcp_fin_timeout= 30
#如果套接字由本端要求关闭，这个参数决定了它保持在FIN-WAIT-2状态的时间。对端可以出错并永远不关闭连接，甚至意外当机。缺省值是60秒。2.2 内核的通常值是180秒，你可以按这个设置，但要记住的是，即使你的机器是一个轻载的WEB服务器，也有因为大量的死套接字而内存溢出的风险，FIN-WAIT-2的危险性比FIN-WAIT-1要小，因为它最多只能吃掉1.5K内存，但是它们的生存期长些。
At the same time, it also involves a TCP congestion algorithm. You can use the following command to view the congestion algorithm control module provided by this machine:
$ sysctl net.ipv4.tcp_available_congestion_control
#对于几种算法的分析，详情可以参考下：TCP拥塞控制算法的优缺点、适用环境、性能分析，比如高延时可以试用hybla，中等延时可以试用htcp算法等。
net.ipv4.tcp_congestion_control=hybla#如果想设置TCP 拥塞算法为hybla
net.ipv4.tcp_fastopen= 3#额外的，对于内核版高于于3.7.1的，我们可以开启tcp_fastopen
5. IO event distribution mechanism
To enable high-concurrency TCP connections in Linux, you must confirm whether the application uses the appropriate network I/O technology and I/O event dispatch mechanism. The available I/O technologies are synchronous I/O, non-blocking synchronous I/O, and asynchronous I/O. In the case of high TCP concurrency, if synchronous I/O is used, it will seriously block the operation of the program unless a thread is created for each TCP connection I/O. However, too many threads will cause huge overhead due to the system's scheduling of threads.
Therefore, it is not advisable to use synchronous I/O in the case of high TCP concurrency. At this time, consider using non-blocking synchronous I/O or asynchronous I/O. Non-blocking synchronous I/O techniques include the use of select(), poll(), epoll and other mechanisms. The technique of asynchronous I/O is to use AIO.
From the point of view of the I/O event dispatching mechanism, using select() is inappropriate, because the number of concurrent connections it supports is limited (usually within 1024). If you consider performance, poll() is also inappropriate, although it can support a higher TCP concurrency, but because of its "polling" mechanism, when the number of concurrency is high, its operating efficiency is quite low, and may exist The uneven distribution of I/O events results in "starvation" of I/O on some TCP connections.
If you use epoll or AIO, there is no such problem (the early implementation of AIO technology in the Linux kernel was achieved by creating a thread for each I/O request in the kernel. This implementation mechanism is in the case of high concurrent TCP connections. In fact, there are serious performance problems. But in the latest Linux kernel, the implementation of AIO has been improved).
In summary, when developing Linux applications that support high-concurrency TCP connections, epoll or AIO technology should be used as much as possible to achieve I/O control on concurrent TCP connections. This will improve the program’s support for high-concurrency TCP connections. Provide effective I/O guarantee.
After such optimized configuration, the TCP concurrent processing capability of the server will be significantly improved. The above configuration is for reference only. For production environment, please adjust, observe and then adjust according to your actual situation.
