The solution to the problem of excessive system connection time wait

In the process of using nginx and Tomcat, it is found that in the case of high concurrency, Tomcat will appear to increase the number of access threads even if the memory and CPU have not yet reached the bottleneck, and it will easily cause denial of service. There are a large number of resident in Tomcat's jvm Request threads, the entire link must be restarted to clear these resident threads.

In addition, Tomcat has more disconnected logs

You can check with this command:

#netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

Status: Description
CLOSED: No connection is active or in progress

LISTEN: the server is waiting for an incoming call SYN_RECV :
a connection request has arrived, waiting
for confirmation Packet dies CLOSING: Both sides try to close at the same time TIME_WAIT: The other side has initiated a release LAST_ACK: Waiting for all packets to die

If it is found that there are a large number of connections in the TIME_WAIT state in the system,

1 Adjust the parameters of nginx and Tomcat

Increase keepalived configuration to reduce disconnection

nginx:

# right before

fastcgi_intercept_errors on;

client_body_timeout 10;

client_header_timeout 10;

send_timeout 10;

keepalive_timeout 50;

client_body_buffer_size 4k;

client_header_buffer_size 1k;

# right after

proxy_connect_timeout 60;

proxy_send_timeout 60;

proxy_read_timeout 60;

proxy_buffer_size 512k;

proxy_buffers 6 512k;

proxy_busy_buffers_size 512k;

proxy_temp_file_write_size 512k;

#在upstream配置中设置

keepalive 120;

Tomcat:

protocol="org.apache.coyote.http11.Http11Nio2Protocol"

connectionTimeout="15000"

acceptCount="100"

acceptorThreadCount="2"

keepAliveTimeout="-1"

maxKeepAliveRequests="200"

maxThreads="300"

minSpareThreads="25"

maxPostSize="0"

redirectPort="8443"

2 通过调整内核参数解决，
vim /etc/sysctl.conf
编辑文件，加入以下内容：
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30

net.ipv4.tcp_timestamps = 1

net.ipv4.tcp_keepalive_time = 1200

#net.ipv4.ip_local_port_range = 65000

net.ipv4.tcp_max_syn_backlog = 8192

net.ipv4.tcp_max_tw_buckets = 5000

#net.bridge.bridge-nf-call-ip6tables = 0

#net.bridge.bridge-nf-call-iptables = 0

#net.bridge.bridge-nf-call-arptables = 0

然后执行 /sbin/sysctl -p 让参数生效。

net.ipv4.tcp_syncookies = 1 表示开启SYN Cookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，默认为0，表示关闭；
net.ipv4.tcp_tw_reuse = 1 表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭；
net.ipv4.tcp_tw_recycle = 1 表示开启TCP连接中TIME-WAIT sockets的快速回收，默认为0，表示关闭。
net.ipv4.tcp_fin_timeout = 30 表示如果套接字由本端要求关闭，这个参数决定了它保持在FIN-WAIT-2状态的时间。
net.ipv4.tcp_keepalive_time = 1200 表示当keepalive起用的时候，TCP发送keepalive消息的频度。缺省是2小时，改为20分钟。
net.ipv4.ip_local_port_range = 1024 65000 表示用于向外连接的端口范围。缺省情况下很小：32768到61000，改为1024到65000。
net.ipv4.tcp_max_syn_backlog = 8192 表示SYN队列的长度，默认为1024，加大队列长度为8192，可以容纳更多等待连接的网络连接数。
net.ipv4.tcp_max_tw_buckets = 5000 表示系统同时保持TIME_WAIT套接字的最大数量，如果超过这个数字，TIME_WAIT套接字将立刻被清除并打印警告信息。
默认为180000，改为5000。对于Apache、Nginx等服务器，上几行的参数可以很好地减少TIME_WAIT套接字数量，但是对于Squid，效果却不大。此项参数可以控制TIME_WAIT套接字的最大数量，避免Squid服务器被大量的TIME_WAIT套接字拖死。

net.ipv4.tcp_tw_reuse = 1 reuse是表示是否允许重新应用处于TIME-WAIT状态的socket用于新的TCP连接；

net.ipv4.tcp_tw_recycle = 1 recyse是加速TIME-WAIT sockets回收。

参考自：

http://kerry.blog.51cto.com/172631/105233/

http://blog.csdn.net/gloria_y/article/details/11733049

http://www.cnblogs.com/discuss/articles/1866851.html

The solution to the problem of excessive system connection time wait

Guess you like