Short connections lead to port exhaustion

Scene playback

The service of machine A requests the service of machine B
Short connection request, dynamically create connection port
A machine service will actively close the connection
High concurrent requests in a short period of time
A machine's tcpssports are exhausted
Most network connections are in the time_wait state

Kernel configuration

net.ipv4.ip_local_port_range = 1024     65000
net.ipv4.ip_local_reserved_ports = 5710-5739,15710-15739

port exhausted

Actively call close()/shutdown() to disconnect, and the status changes to TIME_WAIT after receiving confirmation from the other party.
The TCP protocol TIME_WAIT state will continue for 2MSL, TIME_WAIT has a time window, Linux defaults to 60 seconds
The resources occupied by the connection in the TIME_WAIT state will not be released by the kernel
The resource will not be actually reclaimed by the system until the TIME_WAIT state transitions to the CLOSE state.

Kernel parameter optimization

Client kernel parameter optimization

#Enable fast recycling of TIME-WAIT sockets in TCP connections
net.ipv4.tcp_tw_recycle=1   

#Enable reuse, indicating whether to allow the socket in the TIME-WAIT state (the port of TIME-WAIT) to be used for new TCP connections.
net.ipv4.tcp_tw_reuse=1     

#For the socket connection disconnected by the local end, the time (seconds) that TCP keeps in the FIN-WAIT-2 state.
#The other party may disconnect or never end the connection or die unexpectedly.
net.ipv4.tcp_fin_timeout=5    

#TCP timestamp (adds 12 bytes to the TCP header), in a more accurate way than sending timeouts (refer to RFC 1323)
# to enable calculation of RTT, this option should be enabled for better performance.
net.ipv4.tcp_timestamps=1  

#Shrink the recovery time window of the TIME_WAIT state socket
net.ipv4.tcp_tw_timeout=3

Server kernel parameter optimization

#Defines the length of the maximum listening queue of each port in the system, which is a global parameter.
sysctl net.core.somaxconn=1024  
        
#Transfer buffer length size
ifconfig eth0 txqueuelen 5000           
echo "/sbin/ifconfig eth0 txqueuelen 5000" >> /etc/rc.local

#Maximum number of packets allowed to be sent to the queue when each network interface is receiving packets faster than the kernel can process them
sysctl net.core.netdev_max_backlog=2000

#For connection requests that have not been confirmed by the other party, the maximum number that can be stored in the queue. If the server is overloaded, try increasing this number.
sysctl net.ipv4.tcp_max_syn_backlog=2048

References:

https://en.wikipedia.org/wiki/Transmission_Control_Protocol?spm=5176.100239.blogcont52884.5.JrKkjE

http://www.medianet.kent.edu/techreports/TR2005-07-22-tcp-EFSM.pdf

http://stackoverflow.com/questions/410616/increasing-the-maximum-number-of-tcp-ip-connections-in-linux/3923785#3923785

http://www.cnblogs.com/fczjuever/archive/2013/04/05/3000680.html

http://www.cnblogs.com/fczjuever/archive/2013/04/17/3026694.html