During the stress test today, there were many exceptions at the beginning, all of which were java.net.NoRouteToHostException: Cannot assign requested address.
After checking the information on the Internet, it was because the client connection ports allocated by Linux were exhausted and the socket connection could not be established. Although The socket is closed normally, but the port is not released immediately, but is in the TIME_WAIT state, which waits for 60s by default before releasing.
Check the range of client connection ports supported by linux, that is, 28232 ports:
cat /proc/sys/net/ipv4/ip_local_port_range
32768 - 61000
Solution:
1. Reduce the waiting time after port release, the default is 60s, and change it to 15~30s
echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
2. Modify the tcp/ip protocol configuration, by configuring /proc/sys/net/ipv4/tcp_tw_resue, the default is 0, modify it to 1, release the TIME_WAIT port to The new connection is used.
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
3. Modify the tcp/ip protocol configuration to quickly recycle socket resources, the default is 0, modify it to 1.
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
With the above 3 adjustments, the stress test runs normally.