The production environment of the recently responsible project will report an abnormal response error after a long time. Check the corresponding NGINX log that has a connection timeout that lasts for several minutes, as follows:
upstream timed out (110: Connection timed out) while reading response header from upstream, client
Check the corresponding access log, the request at the corresponding time has no response code, and then look at the request log before there is no response, and find that there are several requests that continue to exceed the set response time of 5 seconds. Check the TCP request status of the application server and find that many of them are in the CLOSE_WAIT state. The app automatically resumes after a few minutes without processing.
Solution to the problem:
1. The processing of individual interfaces takes a long time;
by checking the processing time of the interfaces in the corresponding time period, find out the time-consuming and time-out interfaces for optimization. The optimization idea is to look at the code logic and SQL execution, whether there is too much data Or the query does not use the index.
2. Optimize the TCP request of the application server through the following configuration adjustments;
sudo sysctl -a | grep conntrack
sudo sysctl -w net.netfilter.nf_conntrack_max=262144
sudo sysctl -w net.nf_conntrack_max=262144
sudo sysctl -w net.ipv4.tcp_tw_reuse=1
sudo sysctl -w net.ipv4.tcp_tw_recycle=1
sudo sysctl -w net.ipv4.tcp_fin_timeout=30
Execute the following command to take effect.
sudo sysctl -p
Later, this problem is solved by adding a configuration standby node, the configuration is as follows:
upstream gin_server{
#主节点
server 127.0.0.1:8000;
server 127.0.0.1:8001 backup;#备用节点
}