KeepAlive is both familiar and unfamiliar, and students who have stepped on the pit know the pain. You should not miss the summary of KeepAlive after the front-line operation and maintenance engineers stepped on the pit!
I recently encountered a problem at work and wanted to record it. The scene is as follows:
As can be seen from the above figure, the user accesses the LVS VIP through the Client, and the RealServer mounted on the VIP backend is the Nginx server. Client can be a browser or a client program. Under normal circumstances, there will be no problems with this architecture, but if the client sends the request to Nginx, it will take a while for the Nginx backend to return the result. If it exceeds 1 minute and 30 seconds, there will be problems. Use LVS as a load balancing device to see The phenomenon is that after 1 minute and 30 seconds, the connection between Client and Nginx is disconnected, and no data is returned.
The reason is that LVS keeps the TCP session as 90s by default. If no TCP packets are transmitted on the link for more than 90s, LVS will send a RESET packet to both ends to disconnect the link. I believe everyone knows the reason why LVS does this. There are two main reasons I know:
1. Save the resources of the load balancing device. Each TCP/UDP link will create a Session structure on the load balancing device. If the link is not disconnected, this session structure information will eventually consume all resources, so it must be released. .
2. In addition, release the resources that can protect the backend. If an attacker connects to Nginx through an empty link, if Nginx does not do proper protection, Nginx will be unable to provide services due to too many links.
This kind of problem is not only on LVS. The same problem has been encountered on the commercial load balancing device F5 before. The session disconnection method of F5 is a bit different from that of LVS. F5 will not actively send RESET to both ends of the link, and the session will disappear. After that, when one party in the link sends the message again, it will receive the RESET of F5. The phenomenon after that is that the TCP link state of the end that sends the message again has been disconnected, while the other end is still in the ESTABLISH state.
知道是负载均衡设备原因之后，第一反应就是通过开启KeepAlive来解决。到此这个问题应该是结束了，但是我发现过一段时间总又有人提起KeepAlive的问题，甚至发现由于KeepAlive的理解不正确浪费了很多资源，原本能使用LVS的应用放在了公网下沉区，或者换成了商用F5设备(F5设备的Session断开时间要长一点，默认应该是5分钟)。
所以我决定把我知道的KeepAlive知识点写篇博客分享出来。
为什么要有KeepAlive？
在谈KeepAlive之前，我们先来了解下简单TCP知识(知识很简单，高手直接忽略)。首先要明确的是在TCP层是没有“请求”一说的，经常听到在TCP层发送一个请求，这种说法是错误的。
TCP是一种通信的方式，“请求”一词是事务上的概念，HTTP协议是一种事务协议，如果说发送一个HTTP请求，这种说法就没有问题。也经常听到面试官反馈有些面试运维的同学，基本的TCP三次握手的概念不清楚，面试官问TCP是如何建立链接，面试者上来就说，假如我是客户端我发送一个请求给服务端，服务端发送一个请求给我。。。
这种一听就知道对TCP基本概念不清楚。下面是我通过wireshark抓取的一个TCP建立握手的过程。（命令行基本上用TCPdump,后面我们还会用这张图说明问题）:
现在我看只要看前3行，这就是TCP三次握手的完整建立过程，第一个报文SYN从发起方发出，第二个报文SYN,ACK是从被连接方发出，第三个报文ACK确认对方的SYN，ACK已经收到，如下图：
但是数据实际上并没有传输，请求是有数据的，第四个报文才是数据传输开始的过程，细心的读者应该能够发现wireshark把第四个报文解析成HTTP协议，HTTP协议的GET方法和URI也解析出来，所以说TCP层是没有请求的概念，HTTP协议是事务性协议才有请求的概念，TCP报文承载HTTP协议的请求(Request)和响应(Response)。
现在才是开始说明为什么要有KeepAlive。链接建立之后，如果应用程序或者上层协议一直不发送数据，或者隔很长时间才发送一次数据，当链接很久没有数据报文传输时如何去确定对方还在线，到底是掉线了还是确实没有数据传输，链接还需不需要保持，这种情况在TCP协议设计中是需要考虑到的。
TCP协议通过一种巧妙的方式去解决这个问题，当超过一段时间之后，TCP自动发送一个数据为空的报文给对方，如果对方回应了这个报文，说明对方还在线，链接可以继续保持，如果对方没有报文返回，并且重试了多次之后则认为链接丢失，没有必要保持链接。
如何开启KeepAlive?
KeepAlive并不是默认开启的，在Linux系统上没有一个全局的选项去开启TCP的KeepAlive。需要开启KeepAlive的应用必须在TCP的socket中单独开启。Linux Kernel有三个选项影响到KeepAlive的行为：
1.net.ipv4.tcpkeepaliveintvl = 75
2.net.ipv4.tcpkeepaliveprobes = 9
3.net.ipv4.tcpkeepalivetime = 7200
tcpkeepalivetime的单位是秒，表示TCP链接在多少秒之后没有数据报文传输启动探测报文; tcpkeepaliveintvl单位是也秒,表示前一个探测报文和后一个探测报文之间的时间间隔，tcpkeepaliveprobes表示探测的次数。
TCP socket也有三个选项和内核对应，通过setsockopt系统调用针对单独的socket进行设置：
TCPKEEPCNT: 覆盖 tcpkeepaliveprobes
TCPKEEPIDLE: 覆盖 tcpkeepalivetime
TCPKEEPINTVL: 覆盖 tcpkeepalive_intvl
举个例子，以我的系统默认设置为例，kernel默认设置的tcpkeepalivetime是7200s, 如果我在应用程序中针对socket开启了KeepAlive,然后设置的TCP_KEEPIDLE为60，那么TCP协议栈在发现TCP链接空闲了60s没有数据传输的时候就会发送第一个探测报文。
TCP KeepAlive和HTTP的Keep-Alive是一样的吗？
估计很多人乍看下这个问题才发现其实经常说的KeepAlive不是这么回事，实际上在没有特指是TCP还是HTTP层的KeepAlive，不能混为一谈。TCP的KeepAlive和HTTP的Keep-Alive是完全不同的概念。
TCP层的KeepAlive上面已经解释过了。 HTTP层的Keep-Alive是什么概念呢？ 在讲述TCP链接建立的时候，我画了一张三次握手的示意图，TCP在建立链接之后， HTTP协议使用TCP传输HTTP协议的请求(Request)和响应(Response)数据，一次完整的HTTP事务如下图：
各位看官请注意，这张图我简化了HTTP(Req)和HTTP(Resp)，实际上的请求和响应需要多个TCP报文。
从图中可以发现一个完整的HTTP事务，有链接的建立，请求的发送，响应接收，断开链接这四个过程,早期通过HTTP协议传输的数据以文本为主，一个请求可能就把所有要返回的数据取到，但是，现在要展现一张完整的页面需要很多个请求才能完成，如图片,JS,CSS等，如果每一个HTTP请求都需要新建并断开一个TCP，这个开销是完全没有必要的。
开启HTTP Keep-Alive之后，能复用已有的TCP链接，当前一个请求已经响应完毕，服务器端没有立即关闭TCP链接，而是等待一段时间接收浏览器端可能发送过来的第二个请求，通常浏览器在第一个请求返回之后会立即发送第二个请求，如果某一时刻只能有一个链接，同一个TCP链接处理的请求越多，开启KeepAlive能节省的TCP建立和关闭的消耗就越多。
当然通常会启用多个链接去从服务器器上请求资源，但是开启了Keep-Alive之后，仍然能加快资源的加载速度。HTTP/1.1之后默认开启Keep-Alive, 在HTTP的头域中增加Connection选项。当设置为Connection:keep-alive表示开启，设置为Connection:close表示关闭。实际上HTTP的KeepAlive写法是Keep-Alive，跟TCP的KeepAlive写法上也有不同。所以TCP KeepAlive和HTTP的Keep-Alive不是同一回事情。
Nginx的TCP KeepAlive如何设置?
开篇提到我最近遇到的问题，Client发送一个请求到Nginx服务端，服务端需要经过一段时间的计算才会返回， 时间超过了LVS Session保持的90s,在服务端使用Tcpdump抓包,本地通过wireshark分析显示的结果如第二副图所示，第5条报文和最后一条报文之间的时间戳大概差了90s。
在确定是LVS的Session保持时间到期的问题之后，我开始在寻找Nginx的TCP KeepAlive如何设置，最先找到的选项是keepalivetimeout,从同事那里得知keepalivetimeout的用法是当keepalivetimeout的值为0时表示关闭keepalive,当keepalivetimeout的值为一个正整数值时表示链接保持多少秒，于是把keepalivetimeout设置成75s,但是实际的测试结果表明并不生效。
显然keepalivetimeout不能解决TCP层面的KeepAlive问题，实际上Nginx涉及到keepalive的选项还不少，Nginx通常的使用方式如下：
From the TCP level, Nginx not only needs to care about KeepAlive with Client, but also with Upstream. At the same time, from the HTTP protocol level, Nginx needs to care about Keep-Alive with Client. If Upstream uses HTTP protocol, it also needs to care about Keep-Alive with Upstream. , all in all, it is more complicated.
Therefore, after figuring out the KeepAlive of the TCP layer and the Keep-Alive of HTTP, the KeepAlive of Nginx will not be set incorrectly. When I solved this problem, I was not sure that Nginx had the option to configure TCP keepAlive, so I opened the source code of Ngnix and searched for TCP_KEEPIDLE in the source code. The relevant code is as follows:
From the context of the code, I found that TCP KeepAlive can be configured, so I then looked for which option to configure, and finally found that the so_keepalive option of the listen command can configure the TCP socket with KeepAlive.
Only one of the above three parameters can be used, and cannot be used at the same time, such as sokeepalive=on, sokeepalive=off or sokeepalive=30s:: (represents waiting for 30s without data packets to send probe packets). By setting listen 80, sokeepalive=60s::, the problem of Nginx maintaining long links in LVS is successfully solved, avoiding the use of other high-cost solutions. If a similar problem is encountered on commercial load equipment, it can also be solved in this way.