Too much TIMEWAIT in TCP

Article directory

1. What is TIMEWAIT?

Time_wait is a frequently asked question. The most difficult thing to understand in tcp network programming is its time_wait state, which also shows the importance of time_wait state in tcp/ip four-way wave.
This issue has already been discussed by many people on the Internet. Let’s talk about this issue again, just to share my views based on the related businesses I have handled. As for what TIMEWAIT is, we will not discuss it in depth again. If you are interested, you can read my article:

The TIMEWAIT state itself has nothing to do with the client or server at the application layer . It is only the party that actively closes , and this TIMEWAIT will appear when the TCP connection is normally closed using the FIN|ACK|FIN|ACK four-packet. When the server is processing client requests, if your program is designed so that the server is actively shut down, then you may need to pay attention to the problem of too many TIMEWAIT states. If your server is designed to close passively, then the first thing you should be concerned about is CLOSE_WAIT .

2. Principles

TIMEWAIT is not redundant. After the TCP protocol was created and experienced a lot of actual scene practice, TIMEWAIT appeared, because the party that TCP actively closes the connection needs the TIMEWAIT state, which is our friend. This is the author of "UNIX Network Programming" - Steven's attitude towards TIMEWAIT.

3. TCP state transition diagram

The publication of the TCP protocol: In December 1974, Kahn and Cerf's first detailed description of the TCP protocol was officially published. At that time, the U.S. Department of Defense signed an agreement to complete TCP/IP with three groups of scientists. As a result, the group led by Cerf took the lead and first formulated the TCP/IP protocol standard defined in detail. At that time, an experiment was conducted, in which information packets were passed through a point-to-point satellite network, then a terrestrial cable, then a satellite network, and then ground transmission, passing through various computer systems in Europe and the United States, and no data was lost during the 94,000 kilometers. The long-distance reliable data transmission proves the success of the TCP/IP protocol.

This picture is from "TCP IP Detailed Explanation Volume 1: Protocol Original Book 2nd Edition Chinese" 13.5 TCP State Transition Diagram

This picture is from "UNIX Network Programming, Volume 1: Socket Networking API" 2.6.4 TCP State Transition Diagram

The TCP state machine contains a total of 11 states , and the states are migrated under the drive of various socket apis. Although this picture looks complicated, it is relatively easy for students with certain experience in TCP network programming to understand. Due to space limitations, this article is not going to elaborate. Novice students who want to know the specific migration process are recommended to read Section 2.6 of "Linux Network Programming Volume1".

It can be seen from the TCP state transition diagram that only the party that first calls close() to initiate an active shutdown will enter the TIME_WAIT state, and must enter (the three state transition lines shown in the lower left corner of the figure must eventually enter this state before returning to initial CLOSED state).
It can also be seen from the figure that the TCP connection entering the TIME_WAIT state needs to go through 2MSL to return to the initial state, where MSL refers to the Max
Segment Lifetime, that is, the maximum lifetime of the data packet in the network. Each implementation method of the TCP protocol must specify an appropriate MSL value. For example, the suggested value given by RFC1122 is 2 minutes, and the TCP implementation of the Berkeley system usually chooses 30 seconds as the MSL value. This means that the typical duration of TIME_WAIT is 1-4 minutes.

4. The scenario of sending ACK and RST

4.1 Scenario of TCP sending ACK

In the following situations, TCP will send the ACK packet:

After receiving 1 packet, start the 200ms timer, wait until the 200ms timer is up (the second packet did not come), then the confirmation ack for this packet is sent. This is called "Delayed Sending";
After receiving 1 packet, start the 200ms timer, the 200ms timer has not yet arrived, and the second packet comes again (two packets and one ack);
After receiving 1 packet, start the 200ms timer, and it has not timed out yet, just to send some content to the other party. So the confirmation ack of this package will follow along. This is called "piggybacking";
Whenever TCP receives an out-of-sequence data that exceeds the expected sequence number, it always sends an ACK confirming that the sequence number is its expected sequence number;
Window update or also called open window (when the receiving end window reaches the maximum, all the data in the receiving buffer is pushed to the process and the receiving buffer is empty), and the sending end is notified to continue sending;
Under normal circumstances, the response to the other party's keep-alive probe, see TCP keepalive for details

4.2 Scenario of TCP sending RST

RST packets are sent in the following situations:

connect a port that does not exist;
send data to a closed connection;
send data to a crashed peer (connection was established before);
When close(sockfd), directly discard the unread data in the receiving buffer, and send an RST to the other party. This is controlled by the SO_LINGER option;
A restarts, receives b's keep-alive probe, a sends rst, and notifies b.

In any state of the TCP socket , as long as it receives the RST packet, it can enter the initial state of CLOSED . It is worth noting that the RST segment will not cause any response from the other end, and the other end will not confirm at all. The party receiving the RST will terminate the connection. The program behaves as follows:

Under the blocking model, the kernel cannot actively notify the application layer of an error. Only when the application layer actively calls an IO system call such as read() or write(), the kernel will use the error to notify the application layer of the peer RST.
In the non-blocking model, select or epoll will return sockfd readable, and when the application layer reads it, read() will report an error RST.

If you want to know more about the RST message, you can refer to the article TCP Abnormal Termination (reset message)

5. The upper limit of the number of TCP connections

Considering objective limitations such as memory and file descriptors, it is impossible for almost all servers to reach the theoretical maximum number of tcp connections. Considering these, we only consider the maximum number of tcp connections in the case of unlimited performance in this article.

5.1 Upper limit of TCP port number

The tcp port type is unsigned short, so the upper limit of the port number is 65536, which is 16 times of 2. Since port 0 has a special purpose, the upper limit of the port number is actually 65535, which is 2^16-1.

5.2 The maximum number of TCP connections between client and server

A connection is determined by a 5-tuple : protocol type, local ip, local port, remote ip, remote port.

The client can only choose the local port when using tcp (the protocol type is tcp, the server ip, the port and its own ip are fixed), so only up to 2^16-1 connections can be established.
When the server uses tcp, the protocol type, local ip and local port are fixed, so the remote ip and remote port are variable. There are 2^(8*4) ipv4, that is, 2^32 kinds of ip; ipv6 needs more. So if it is an ipv4 address, then the server can receive 2^32 * (2^16-1) connections.

The maximum number of connections on the server side is very large, but in fact, the performance of the server is basically unable to support such a huge number of connections. Moreover, there are almost no server tasks that require such a huge number of concurrent connections, so this maximum number of connections generally only exists in theory.

6. 11 states of TCP connection

Establish a connection protocol (three-way handshake)
　(1) The client sends a TCP message with the SYN flag to the server. This is message 1 in the three-way handshake process.
　(2) The server responds to the client. This is the second message in the three-way handshake. This message carries both the ACK flag and the SYN flag. Therefore, it represents the response to the client's SYN message just now; at the same time, it marks the SYN to the client and asks whether the client is ready for data communication.
　(3) The client must respond with an ACK message in the service segment again, which is segment 3.
Connection termination protocol (four-way handshake)
Since the TCP connection is full-duplex, each direction must be closed separately. The principle is that when one party completes its data sending task, it can send a FIN to terminate the connection in this direction. Receiving a FIN only means that there is no data flow in this direction, and a TCP connection can still send data after receiving a FIN. The party that shuts down first will perform an active close, while the other party will perform a passive close.
　(1) The TCP client sends a FIN to close the data transmission from the client to the server (segment 4).
　(2) When the server receives the FIN, it sends back an ACK, confirming that the serial number is the received serial number plus 1 (segment 5). Like SYN, a FIN will occupy a sequence number.
　(3) The server closes the client connection and sends a FIN to the client (segment 6).
　(4) The client segment sends back an ACK message for confirmation, and sets the confirmation sequence number to the received sequence number plus 1 (segment 7).

CLOSED : There is nothing to say about this, it means the initial state.
LISTEN : This is also a state that is very easy to understand, indicating that a certain SOCKET on the server side is in the listening state and can accept connections.
SYN_RCVD : This state indicates that a SYN message has been received. Under normal circumstances, this state is an intermediate state during the three-way handshake session of SOCKET on the server side when establishing a TCP connection. It is very short. Basically, it is difficult for you to see it with netstat In this state, unless you specially write a client test program, deliberately not send the last ACK message during the three-way TCP handshake. Therefore, in this state, after receiving the ACK message from the client, it will enter the ESTABLISHED state.
SYN_SENT : This state echoes SYN_RCVD. When the client SOCKET executes the CONNECT connection, it first sends a SYN message, so it will enter the SYN_SENT state immediately, and wait for the server to send the second message in the three-way handshake. The SYN_SENT state indicates that the client has sent a SYN message.
ESTABLISHED : This is easy to understand, indicating that the connection has been established.
FIN_WAIT_1 : This state needs to be explained carefully. In fact, the true meaning of the FIN_WAIT_1 and FIN_WAIT_2 states is to wait for the FIN message from the other party. The difference between these two states is that the FIN_WAIT_1 state actually means that when the SOCKET is in the ESTABLISHED state, it wants to actively close the connection and sends a FIN message to the other party. At this time, the SOCKET enters the FIN_WAIT_1 state. And when the other party responds to the ACK message, it enters the FIN_WAIT_2 state. Of course, under actual normal circumstances, no matter what the other party’s situation is, it should immediately respond to the ACK message, so the FIN_WAIT_1 state is generally difficult to see, and The FIN_WAIT_2 state can often be seen with netstat.
FIN_WAIT_2 : This state has been explained in detail above. In fact, SOCKET in the FIN_WAIT_2 state means a semi-connection, that is, one party requests a close connection, but also tells the other party that I still have some data to send to you for the time being, and I will send it to you later. Close the connection.
TIME_WAIT : Indicates that the FIN message from the other party has been received and an ACK message has been sent, and it will return to the CLOSED available state after waiting for 2MSL. If in the FIN_WAIT_1 state, when receiving a message with both the FIN flag and the ACK flag from the other party, it can directly enter the TIME_WAIT state without going through the FIN_WAIT_2 state.
CLOSING : This state is quite special, it should be rare in actual situations, and belongs to a relatively rare exception state. Under normal circumstances, when you send a FIN message, you should receive (or receive at the same time) the other party's ACK message first, and then receive the other party's FIN message. But the CLOSING state means that after you sent the FIN message, you did not receive the other party's ACK message, but instead received the other party's FIN message. Under what circumstances will this happen? In fact, if you think about it carefully, it is not difficult to draw a conclusion: that is, if both parties close a SOCKET almost at the same time, then there will be a situation where both parties send FIN messages at the same time, that is, the CLOSING state will appear, indicating that both parties are closing the SOCKET connection .
CLOSE_WAIT : The meaning of this state is actually waiting to be closed. How do you understand it? When the other party closes a SOCKET and sends a FIN message to itself, your system will undoubtedly respond with an ACK message to the other party, and then enter the CLOSE_WAIT state. Next, what you really need to consider is to check whether you still have data to send to the other party. If not, then you can close the SOCKET and send a FIN message to the other party, that is, close the connection. So what you need to do in the CLOSE_WAIT state is to wait for you to close the connection.
LAST_ACK : This state is relatively easy to understand. It is passively closed after one party sends a FIN message, and finally waits for the other party's ACK message. After receiving the ACK message, it can enter the CLOSED available state.

7. The role of TIMEWAIT state (reason)

There are two main reasons for the existence of the TIME_WAIT state:

7.1 In order to realize the reliable release of TCP full-duplex (full-duplex) connection

Refer to another article for the three-way handshake and four-way handshake detailed explanation of the TCP release connection and 4-way handshake diagram, assuming that the party that initiates the active close (the client in the figure) sends the ACK (the last packet of the 4 interactions) in the network Lost, then due to the TCP retransmission mechanism, the party that executes passive close (the server in the figure) needs to resend its FIN. Before the FIN reaches the client (the client is the initiator of the active close), the client must maintain the state of this connection (Although it has called close), specifically, the (local_ip, local_port) resources corresponding to this TCP connection cannot be released or reallocated immediately. The TCP connection cannot restore the initial CLOSED state until the FIN resent by the romete peer arrives and the client also resends the ACK. If the active close party does not enter TIME_WAIT to maintain its connection state, when the FIN resent by the passive close party arrives, the TCP transport layer of the active close party will respond to the other party with an RST packet, which will be considered by the other party to have an error (in fact, This is the normal process of closing a connection, not an exception).

7.2 In order to make old data packets disappear due to expiration in the network

To illustrate this problem, we first assume that there is no TIME_WAIT state limitation in the TCP protocol, and then assume that there is currently a TCP connection: (local_ip, local_port, remote_ip, remote_port), for some reason, we close it first, and then quickly use the same The four-tuple of the set up a new connection. As mentioned earlier in this article, a TCP connection is uniquely identified by a quadruple. Therefore, in our hypothetical situation, the TCP protocol stack cannot distinguish the difference between the two TCP connections before and after. In its view, this is the same connection at all. The process of first releasing and then establishing in the middle is not "perceived" for it. In this way, it may happen that after the data sent by the local peer in the previous TCP connection arrives at the remote peer, it will be received by the TCP transport layer of the remote peer as the normal data of the current TCP connection and passed up to the application layer (in fact, , in our hypothetical scenario, before the old data reaches the remote peer, the old connection has been disconnected and a new TCP connection consisting of the same quadruple has been established. Therefore, these old data should not be passed up to the application layer ), which will cause data confusion and lead to various unpredictable and strange phenomena. As a reliable transport protocol, TCP must consider and avoid this situation at the protocol level, which is the second reason why the TIME_WAIT state exists.
Specifically, after the local peer actively calls close, the TCP connection at this time enters the TIME_WAIT state, and the TCP connection in this state cannot immediately establish a new connection with the same quadruple, that is, the local port occupied by the party that initiated the active close Can no longer be reallocated during TIME_WAIT. Since the duration of the TIME_WAIT state is 2MSL, this ensures that the old data packets in the duplex link of the old TCP connection will disappear due to expiration (more than MSL). After that, a new connection can be established with the same quadruple without The situation where the connection data was confused twice before and after occurred.

7.3 Another more in-depth statement

There are two reasons for the existence of the TIME_WAIT state: (1) To make the 4-way handshake closing process more reliable . The last ACK of the 4-way handshake is sent by the active closing party. If this ACK is lost, the passive closing party will send a FIN again. If the active closing party can maintain a 2MSL TIME_WAIT state, there is a greater chance that the lost ACK will be sent out again. (2) Prevent the lost duplicate from causing damage to the transmission of subsequent new normal links . Lost duplicates are very common in actual networks. It is often due to router failures that prevent path convergence, causing a packet to jump between routers A, B, and C like an endless loop. There is a TTL in the IP header, which limits the maximum hops of a packet in the network, so this packet has two fates, either the last TTL becomes 0 and disappears in the network; or the router path converges before the TTL becomes 0, It finally reaches its destination with the remaining TTL hops. But it is a pity that TCP sent a packet exactly like it earlier through the timeout retransmission mechanism, and reached the destination before it, so its fate is destined to be abandoned by the TCP protocol stack. Another concept is called incarnation connection , which refers to a new connection that is exactly the same as the last socket pair, called incarnation of previous connection. Lost duplicate plus incarnation connection will cause fatal errors to our transmission. Everyone knows that TCP is streaming, and the order in which all packets arrive is inconsistent. The sequence number is spliced by the TCP protocol stack; suppose an incarnation connection receives seq=1000 at this time, and a lost duplicate is seq =1000, len=1000, then tcp thinks that the lost duplicate is legal and stores it in the receive buffer, resulting in an error in transmission.Through a 2MSL TIME_WAIT state , ensure that all lost duplicates will disappear, avoiding errors for new connections .

8. Why does TIMEWAIT need 2MSL

8.1 MSL (Maximum Segment Lifetime) maximum packet lifetime

Every TCP implementation must choose an MSL. It is the maximum time any segment can be in the network before it is discarded. This time is limited because TCP segments are transmitted within the network as IP datagrams, and IP datagrams have a TTL time that limits their lifetime. RFC 793 states that the MSL is 2 minutes, and 30 seconds or 1 minute is commonly used in reality.

8.2 2MSL

When TCP performs an active close and sends the last ACK, the link must stay in the TIME_WAIT state for 2MSL. This can (1) allow TCP to send the last ACK again to prevent this ACK from being lost (the passively closed party times out and resends the last FIN); to ensure the termination of TCP's reliable full-duplex connection. (2) Allow old duplicate sections to disappear in the network. Refer to the article "Unix Network Programming" (3) When the TCP connection is established and terminated in the TIME_WAIT state, the ports at both ends cannot be used, and they cannot be used until the end of the 2MSL time. When the connection is in the 2MSL waiting phase, any late segment will be discarded. However, in practical applications, you can use the port without waiting for the 2MSL time to end by setting the SO_REUSEADDR option. In order to facilitate everyone's understanding, the following uses two easy-to-understand ways to explain this problem.

8.3 Why does TCP wait for 2MSL when it waves 4 times? explain one

The primary problem to be solved in communication is to maintain the information symmetry between the two parties in communication, so that the two parties in communication are in a synchronized state .

Let's start with an example:

Romeo wrote a letter to his middle school classmate Juliet during college. The content of the letter is as follows:

Little Leaf, I like you!

After the letter is sent, Romeo has no way of knowing whether Juliet can receive it. Only when he receives Xiao Yezi's reply, can he know that his letter has reached the other party.

Three days later, Xiao Yezi wrote back, and the content of the letter was as follows:

Xiao Ou, I have read your letter, I like you too...

At this time, the state of both parties in Xiao Yezi's eyes is: mutual admiration!

If Xiao Ou receives the reply, the state of both parties in Xiao Ou's eyes is also: love each other!

If Xiao Ou didn't receive a reply, Xiao Ou saw the state of both parties: unrequited love!

In order to put an end to Xiaoou's ambiguous state and make him reach a consensus of " mutual love " with herself, Xiaoye needs to do the following work:

1) Wait patiently for Xiaoou's third letter

2) If you do not receive a reply within a few days, you need to send your second letter again

If Xiaoou's reply is received, the status of the two parties will finally be synchronized: "Mutual love"!

Even if 2) happens, a synchronized state can be reached after N days.

After that, the two parties can talk sweetly and fall in love.

The TCP four-wave wave also follows a similar pattern.

The side that is actively disconnected is A, and the side that is passively disconnected is B.

The first message: A sends FIN

Second message: B replies with ACK

Third message: B sends FIN

At this moment: B unilaterally believes that he has reached a consensus with A, that is, both parties agree to close the connection.

At this time, can B release the memory resources occupied by this TCP connection? No, B must ensure that A receives his ACK and FIN.

So B needs to wait quietly for the arrival of A's fourth message:

The fourth message: A sends an ACK to confirm receipt of B's FIN

When B receives this message, it thinks that the two parties have reached synchronization: both parties know that the connection can be released, and at this time B can safely release the memory resources and port numbers occupied by the TCP connection.

Therefore, B that is passively closed directly releases resources without any wait time.

However, A does not know whether B has received his ACK, A thinks so:

1) If B does not receive its own ACK, it will timeout and retransmit FiN

Then A receives the retransmitted FIN again and sends ACK again

2) If B receives its own ACK, it will not send any more messages, including ACK

Regardless of whether it is 1 or 2, A needs to wait, and the maximum value of the waiting time in these two cases should be taken to deal with the worst case . The worst case is:

The maximum lifetime (MSL) of the outgoing ACK message + the maximum lifetime (MSL) of the incoming FIN message.

This is exactly 2MSL (Maximum Segment Life) .

Waiting for 2MSL time, A can safely release the resources and port numbers occupied by TCP, and can use this port number to connect to any server at this time .

Why do we have to wait for 2MSL?

If not, the released port may reconnect to the server port just disconnected, so that the old TCP packets still surviving in the network may conflict with the new TCP connection packets, resulting in data conflicts. To avoid this situation, you need Wait patiently for all the active packets of the old TCP connections on the network to die, 2MSL time can meet this requirement (although it is very conservative)!

8.4 Why does TCP wait for 2MSL when it waves 4 times? Explanation two

MSL is the Maximum Segment Lifetime, the maximum lifetime of a packet . It is the longest time that any packet exists on the network. After this time, the packet will be discarded. Because the TCP message is based on the IP protocol , and there is a TTL field in the IP header , which is the maximum number of routes that the IP datagram can pass through . This value will be reduced by 1 every time it passes through a router that processes it. When this value is 0, the data The message will be discarded, and an ICMP message will be sent to notify the source host at the same time.

The difference between MSL and TTL: The unit of MSL is time, while TTL is the number of route hops . Therefore, the MSL should be greater than or equal to the time when the TTL is 0 to ensure that the packet has been destroyed naturally.

The time of 2MSL starts from the time when the client sends ACK after receiving FIN . If within the TIME-WAIT time, because the client's ACK is not transmitted to the server, and the client receives the FIN message resent by the server, then the 2MSL time will be reset .

Next, we start to analyze why it is necessary to wait for 2MSL time after sending the last ACK message to ensure that no message belonging to the current connection is still alive in the network (provided that no FIN message from the other party is received within this 2MSL time. text, but even if the FIN message from the opposite end is received, it will not affect our discussion, because if FIN is received, ACK will be replied and the timing will be restarted).

For ease of description, we imagine that there is a TCP connection in the process of being disconnected. The two ends of this connection are A and B, where A is the end that actively closes the connection, because it has just sent a message sent to the other end to the other end. The ACK of the FIN message is in the TIME_WAIT state at this time; while B is the passively closed end, it is in the LAST_ACK state at this time, and it will retransmit the FIN message until it times out before receiving the last ACK. As time goes by, the ACK message sent by A to B will have two endings:

The ACK message is lost in the network; as mentioned earlier, we don't need to consider this situation, because unless multiple retransmissions fail, the status of both ends of AB will not change until a certain ACK is no longer lost.
The ACK message is received by B. We assume that B receives the ACK after a period of time t after A sends the ACK message, then 0 < t <= MSL. Because A does not know how long it will take for the other party to receive the ACK it sends out, so A must maintain the TIME_WAIT state for at least the MSL duration to ensure that its ACK disappears from the network. At the same time, B in the LAST_ACK state directly enters the CLOSED state because it has received the ACK, without sending any message to the network. So at a glance, A only needs to wait for 1 MSL, but if you think about it carefully, 1 MSL is not enough, because at the moment before B receives the ACK, B may retransmit a FIN because it did not receive the ACK message, this FIN message needs at most one MSL to disappear from the network, so A still needs to wait for one more MSL.

To sum up, TIME_WAIT needs to last at least 2MSL. The first MSL of the two MSLs is to wait for the last ACK sent by itself to disappear from the network, while the second MSL is to wait for the ACK to be received by the peer The FIN message that may be retransmitted in the previous instant disappears from the network.

It can be seen that the 2MSL duration is actually equivalent to allowing at least one packet loss . For example, if the ACK is lost within one MSL, the FIN resent by the passive party will arrive within the second MSL, and the connection in the TIME_WAIT state can handle it.

Why not 4 or 8 MSL? You can imagine a bad network with a packet loss rate of 1%. The probability of two consecutive packet loss is only 1/10,000. This probability is too small. Ignoring it is more cost-effective than solving it.

In the Linux system, 2MSL is 60 seconds by default, so one MSL is 30 seconds. The Linux system stays in TIME_WAIT for a fixed 60 seconds .

Its name defined in the Linux kernel code is TCP_TIMEWAIT_LEN:

#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT 
                                    state, about 60 seconds  */

If you want to modify the time length of TIME_WAIT, you can only modify the value of TCP_TIMEWAIT_LEN in the Linux kernel code and recompile the Linux kernel.

“ 为什么需要 TIME_WAIT 状态？”

Only the party that actively initiates the closing of the connection will have the TIME-WAIT state.

The TIME-WAIT state is required for two main reasons:

Prevent "old" data packets with the same "quadruple" from being received; ensure that the party that "passively closes the connection" can be closed correctly, that is, ensures that the last ACK can be received by the passively closed party, thereby helping it to close normally;

Reason 1: Prevent packets from old connections
Assuming TIME-WAIT has no waiting time or is too short, what happens after the delayed packet arrives?

As shown in the figure above, the SEQ = 301 message sent by the server in the yellow box before closing the connection was delayed by the network.
At this time, after the TCP connection with the same port is multiplexed, the delayed SEQ = 301 arrives at the client, then the client may receive the expired message normally, which will cause serious problems such as data confusion.

Therefore, TCP has designed such a mechanism. After 2MSL, the data packets in both directions are enough to be discarded, so that the original connected data packets will naturally disappear in the network, and the reappearing data packets must be Generated by newly established connections .

Reason 2: To ensure that the connection is closed correctly
In RFC 793, it is pointed out that another important role of TIME-WAIT is:

TIME-WAIT - represents waiting for enough time to pass to be sure the remote TCP received the acknowledgment of its connection termination request.

In other words, the role of TIME-WAIT is to wait for enough time to ensure that the last ACK can be received by the passive closing party, thereby helping it to close normally.

Assuming that TIME-WAIT has no waiting time or the time is too short, what problems will disconnection cause?

If the last ACK message of the client waved four times in the red box in the above figure is lost in the network, if the client TIME-WAIT is too short or does not exist, it will directly enter the CLOSED state, and the server will Always in LASE_ACK state.
When the client initiates a SYN request message to establish a connection, the server will send a RST message to the client, and the connection establishment process will be terminated.

If TIME-WAIT waits long enough, you will encounter two situations:

If the server normally receives the last ACK packet of four waved hands, the server closes the connection normally.
When the server does not receive the last ACK message of the four waved hands, it will resend the FIN close connection message and wait for a new ACK message.

Therefore, after the client waits for 2MSL in the TIME-WAIT state, it can ensure that the connections of both parties can be closed normally.

9. The problem of a large number of TIME_WAIT

9.1 TIMEWAIT is friendly

TCP must ensure that all data can be delivered correctly under all possible circumstances. When you close a socket, the socket that is actively closed will enter the TIME_WAIT state, while the passively closed one will enter the CLOSED state , which can indeed ensure that all data is transmitted. When a socket is closed, it is completed through a four-way handshake at both ends. When one end calls close(), it means that there is no data to send at this end. It seems that after the handshake is completed, the socket can be in the initial CLOSED state, but it is not. The reason is that there are two problems in arranging the state in this way. First, we do not have any mechanism to ensure that the last ACK can be transmitted normally . Second, there may still be residual data packets (wandering duplicates) on the network, and we must be able to process them normally.
TIMEWAIT was born to solve these two problems.

Assuming the last ACK is lost, the passive close party will resend its FIN. The active closing party must maintain a valid state information (maintained in the TIMEWAIT state) so that the ACK can be resent. If the actively closed socket does not maintain this state and enters the CLOSED state, then when the actively closed socket is in the CLOSED state, it will respond with a RST after receiving FIN . Passive closing party will think that something went wrong after receiving RST. If the TCP protocol wants to complete the necessary operations normally and terminate the data stream transmission between the two parties, it must completely and correctly transmit the four sections of the four-way handshake without any loss. This is the first reason why the socket is still in the TIME_WAIT state after it is closed, because he has to wait to resend the ACK.
Assume that the currently connected communication parties have already called close(), and both parties enter the CLOSED final state at the same time, instead of going to the TIME_WAIT state. The following problem will occur, now a new connection is established, the IP address and port used are exactly the same as the previous one, and the connection established later is a complete reuse of the original connection. It is also assumed that there are datagrams remaining in the network in the original connection, so that the datagrams received by the new connection may be the datagrams of the previous connection. To prevent this, TCP does not allow new connections to reuse sockets in the TIME_WAIT state . The socket in the TIME_WAIT state waits for twice the MSL time (the reason why it is twice the MSL is that the MSL is the time when a datagram is sent one-way in the network until it is considered lost. A datagram may be sent on the way or It becomes a residual datagram during the response process, and it takes twice the MSL to confirm the discarding of a datagram and its response), and it will change to the CLOSED state. This means that a successfully established connection must cause the remaining datagrams in the previous network to be lost.

9.2 Troublesome business problems caused by a large number of TIMEWAIT in some scenarios

A large number of TIMEWAITs appear and need to be solved:
On a TCP server with high concurrent short connections, when the server finishes processing the request, it immediately closes the connection automatically and normally~~~ In this scenario, a large number of sockets will be in the TIMEWAIT state. If the client's concurrency continues to be high, some clients will fail to connect at this time.
Let me explain the scene. Actively close the TCP connection normally, and TIMEWAIT will appear. Why should we pay attention to this high concurrent short connection? There are two aspects to be aware of:

High concurrency allows the server to occupy a large number of ports at the same time in a short period of time , and the ports range from 0 to 65535, which is not many. Excluding those used by the system and other services, there are even fewer ports left.
In this scenario, a short connection means a connection in which " the time for business processing + data transmission is much shorter than the time for TIMEWAIT timeout ". Here is a relatively long and short concept. For example, take a web page, and a 1-second http short connection processes the business. After the connection is closed, the port used by the business will stay in the TIMEWAIT state for a few minutes, and these few minutes, This port cannot be occupied when other HTTP requests come. If you use this business alone to calculate the server utilization rate, you will find that the ratio of the time the server is doing serious business to the time when the port (resource) is hung up and cannot be used is 1: hundreds, which is a serious waste of server resources. (To make a digression, if you consider server performance tuning in this sense, the service of long connection business does not need to consider the TIMEWAIT state. At the same time, if you are very familiar with server business scenarios, you will find that in actual business scenarios , generally the concurrency of the business corresponding to the long connection is not very high)
Combining these two aspects, the continuous arrival of a certain amount of high-concurrency short connections will cause the server to refuse to serve some customers due to insufficient port resources . At the same time, these ports are temporarily allocated by the server, and the SO_REUSEADDR option cannot be used to solve this problem. (The server can set the SO_REUSEADDR socket option to notify the kernel if the port is busy, but the port can be reused when the TCP connection is in the TIME_WAIT state. In a very useful scenario, if your server program stops and you want to restart immediately, and the new The socket still wants to use the same port, so the SO_REUSEADDR option can avoid the TIME_WAIT state.)

9.3 A pair of contradictions

TIMEWAIT is as friendly as it is headache-inducing.
But we still have to look at it with a friendly attitude, because it does its best to ensure the robustness of the server .

9.4 Feasible and must exist, but unprincipled solutions

Linux does not expose the interface for modifying the TIMEWAIT timeout in the sysctl or proc file system. You can modify the timeout parameter of the TIMEWAIT in the kernel protocol stack code and reprogram the kernel to shorten the timeout time and speed up recycling;
Use the forced closing method of the SO_LINGER option to send RST instead of FIN to bypass the TIMEWAIT state and directly enter the CLOSED state. See my blog post "TCP Option SO_LINGER" for details.

9.5 How do I see this problem

Why do I think the above two solutions are feasible, but they are not in line with the principle?
I first think that I have to rely on the TIMEWAIT state to ensure the robustness of my server program. There are too many messy problems on the network, and I need the service to function normally.
Does that mean there is no need for performance? Not really. If the short connection traffic running on the server reaches the point where I really have to deal with the problem of too many TIMEWAIT states, my principle is to deal with it as much as possible instead of doing it with TIMEWAIT. If you try to deal with it, but still can't solve the problem, and still refuse to serve some requests, then I will adopt the method of dividing the machine, and let multiple machines resist these high-concurrency short requests. Sustaining 100,000 concurrent short connection requests, two machines, 50,000 each, should be enough. The general business volume and most of the domestic websites do not actually need to pay attention to this issue. In a word, the number of visits that need to pay attention to this issue cannot be reached.
Is there a scenario where you really have to solve this problem in the way I find unreasonable above? The answer is yes.
Sites like Taobao, Baidu, Sina, and JD.com have a lot of static small picture businesses. If the servers are excessively distributed, a large number of machines will need to be launched. Buying more machines will cost more, and more computer rooms will be allocated. To guard these machines, the cost growth is very serious. . . At this time, we must do everything possible to optimize.
As an aside, there are no absolute technical issues on the server, everything is for business needs.

9.6 How to deal with too much TIMEWAIT as much as possible

Just change two kernel parameters in /etc/sysctl.conf, as follows:

net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

After modification, execute /sbin/sysctl -pto make the parameters take effect.
To put it simply, it is to enable the TIMEWAIT reuse and fast recycling of the system . As for how to reuse and fast recycling, I have not delved into this issue. In actual scenarios, it is indeed effective. Observations with netstat or ss can draw conclusions.
Some friends also open the function of syncookies at the same time, as follows:

net.ipv4.tcp_syncookies = 1

The purpose of opening this syncookies is actually: "In the case of insufficient server resources (not just port resources, denial of service has many resource shortages), try not to reject TCP syn (connection) requests, try to send syn requests Cache it, and save it to process these TCP connection requests when you have the ability later."
If the concurrency is really very, very high, it is not very useful to turn this on.

10. Configure /etc/sysctl.conf

10.1 /etc/sysctl.conf

# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
#关闭ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

# 避免放大攻击
net.ipv4.icmp_echo_ignore_broadcasts = 1

# 开启恶意icmp错误消息保护
net.ipv4.icmp_ignore_bogus_error_responses = 1

# 关闭路由转发
#net.ipv4.ip_forward = 1
#net.ipv4.conf.all.send_redirects = 0
#net.ipv4.conf.default.send_redirects = 0

 #开启反向路径过滤
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

#处理无源路由的包
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

 #关闭sysrq功能
kernel.sysrq = 0

#core文件名中添加pid作为扩展名
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1

#修改消息队列长度
kernel.msgmnb = 65536
kernel.msgmax = 65536

#设置最大内存共享段大小bytes
kernel.shmmax = 68719476736
kernel.shmall = 4294967296

#timewait的数量，默认180000
net.ipv4.tcp_max_tw_buckets = 6000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096        87380   4194304
net.ipv4.tcp_wmem = 4096        16384   4194304
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144

#限制仅仅是为了防止简单的DoS 攻击
net.ipv4.tcp_max_orphans = 3276800

#未收到客户端确认信息的连接请求的最大值
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_timestamps = 0

#内核放弃建立连接之前发送SYNACK 包的数量
net.ipv4.tcp_synack_retries = 1

#内核放弃建立连接之前发送SYN 包的数量
net.ipv4.tcp_syn_retries = 1

#启用timewait 快速回收
net.ipv4.tcp_tw_recycle = 1

#开启重用。允许将TIME-WAIT sockets 重新用于新的TCP 连接
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_fin_timeout = 1

#当keepalive 起用的时候，TCP 发送keepalive 消息的频度。缺省是2 小时
net.ipv4.tcp_keepalive_time = 30

#允许系统打开的端口范围
net.ipv4.ip_local_port_range = 1024    65000

# 确保无人能修改路由表
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

10.2 Configure sysctl.conf to quickly recycle and reuse TIME_WAIT resources

Optimize kernel parameters so that the server can quickly recycle and reuse those TIME_WAIT resources

编辑内核文件/etc/sysctl.conf，加入以下内容：

net.ipv4.tcp_syncookies = 1    %表示开启SYN Cookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，默认为0，表示关闭；
net.ipv4.tcp_tw_reuse = 1      %表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭；
net.ipv4.tcp_tw_recycle = 1    %表示开启TCP连接中TIME-WAIT sockets的快速回收，默认为0，表示关闭。
net.ipv4.tcp_fin_timeout      %表示如果套接字由本端要求关闭，这个参数决定了它保持在FIN-WAIT-2状态的时间

执行 /sbin/sysctl -p 让参数生效：
/etc/sysctl.conf是一个允许改变正在运行中的Linux系统的接口，它包含一些TCP/IP堆栈和虚拟内存系统的高级选项，修改内核参数永久生效。

10.3 Misunderstanding of tcp_fin_timeout

A common mistake is to think that tcp_fin_timeout is the time of TIME_WAIT. In fact, tcp_fin_timeout is the time to stay in the FIN-WAIT-2 state.
The 2MSL of TIME_WAIT is written in the Linux kernel code, see 6.4 of this article

#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT 
                                    state, about 60 seconds  */

10.4 Common parameters

TCP parameter description:

net.ipv4.tcp_syncookies = 1 表示开启SYN Cookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，默认为0，表示关闭；

net.ipv4.tcp_tw_reuse = 1 表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭；

net.ipv4.tcp_tw_recycle = 1 表示开启TCP连接中TIME-WAIT sockets的快速回收，默认为0，表示关闭。

net.ipv4.tcp_fin_timeout = 30 表示如果套接字由本端要求关闭，这个参数决定了它保持在FIN-WAIT-2状态的时间。

net.ipv4.tcp_keepalive_time = 1200 表示当keepalive起用的时候，TCP发送keepalive消息的频度。缺省是2小时，改为20分钟。

net.ipv4.ip_local_port_range = 1024 65000 表示用于向外连接的端口范围。缺省情况下很小：32768到61000，改为1024到65000。

net.ipv4.tcp_max_syn_backlog = 8192 表示SYN队列的长度，默认为1024，加大队列长度为8192，可以容纳更多等待连接的网络连接数。

net.ipv4.tcp_max_tw_buckets = 5000表示系统同时保持TIME_WAIT套接字的最大数量，如果超过这个数字，TIME_WAIT套接字将立刻被清除并打印警告信息。默认为180000，改为5000。

For servers such as Apache and Nginx, the parameters in the above lines can reduce the number of TIME_WAIT sockets very well, but for Squid, the effect is not great. This parameter can control the maximum number of TIME_WAIT sockets to prevent the Squid server from being dragged to death by a large number of TIME_WAIT sockets.

10.5 Common commands

In the case of a large number of short connections, the TCP stack of Linux generally generates a large number of sockets in the TIME_WAIT state. You can see it with the following command:

netstat -ant| grep -i time_wait

Sometimes, this number is staggering:
tcp

netstat -ant|grep -i time_wait |wc -l

netstat
It may be more than 30,000 to 40,000. At this time, we need to modify the tcp time wait time of the linux kernel. To shorten it, there is a sysctl parameter that seems to be available. It is /proc/sys/net/ipv4/tcp_fin_timeout, and the default value is 60, which is 60 seconds. A lot of information on the Internet say that setting this value lower can reduce the TIME_WAIT status in netstat, but this statement is wrong.

After carefully reading the Linux kernel source code , we found that this value is actually for output. After modification, it is not actually read back into the kernel for use, but what really works in the kernel is a macro definition, in $KERNEL/include/ net Inside /tcp.h , there are the following lines:

#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
 state, about 60 seconds */

And this macro really controls the timeout period of TCP TIME_WAIT state. If we want to reduce the number of TIME_WAIT states (thus saving a little kernel operation time), then we can set this value lower. According to our test, it is more appropriate to set it to 10 seconds!

View the number of each state of TCP

netstat -ant|awk '/^tcp/ {++S[$NF]} END {for(a in S) print (a,S[a])}'

output:

LAST_ACK 14
SYN_RECV 348
ESTABLISHED 70
FIN_WAIT1 229
FIN_WAIT2 30
CLOSING 33
TIME_WAIT 18122

command explanation

先来看看netstat：
netstat -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 123.123.123.123:80 234.234.234.234:12345 TIME_WAIT
你实际执行这条命令的时候，可能会得到成千上万条类似上面的记录，不过我们就拿其中的一条就足够了。
 
再来看看awk：
/^tcp/
滤出tcp开头的记录，屏蔽udp, socket等无关记录。
state[]相当于定义了一个名叫state的数组
NF
表示记录的字段数，如上所示的记录，NF等于6
$NF
表示某个字段的值，如上所示的记录，$NF也就是$6，表示第6个字段的值，也就是TIME_WAIT
state[$NF]表示数组元素的值，如上所示的记录，就是state[TIME_WAIT]状态的连接数
++state[$NF]表示把某个数加一，如上所示的记录，就是把state[TIME_WAIT]状态的连接数加一
END
表示在最后阶段要执行的命令
for(key in state)
遍历数组

11. Some questions about TCP three-way handshake and four-way handshake

第一次握手，如果客户端发送的SYN一直都传不到被服务器，那么客户端是一直重发SYN到永久吗？客户端停止重发SYN的时机是什么？
第三次握手，如果服务器永远不会收到ACK，服务器就永远都留在 Syn-Recv 状态了吗？退出此状态的时机是什么？
第三次挥手，如果客户端永远收不到 FIN,ACK，客户端永远停留在 Fin-Wait-2状态了吗？退出此状态时机是什么时候呢？
第四次挥手，如果服务器永远收不到 ACK，服务器永远停留在 Last-Ack 状态了吗？退出此状态的时机是什么呢？
如果客户端在 2SML内依旧没收到 FIN,ACK，会关闭链接吗？服务器那边怎么办呢，是怎么关闭链接的呢？
可以看到，这些问题都是关于 TCP 是如何处理这些异常场景的，我们在学 TCP 连接建立和断开的时候，总是以为这些过程能如期完成。
可惜理想很丰满，现实很骨感，事实预料呀。
TCP 当然不傻，对以上这些异常场景都是有做处理的。
这些异常场景共分为两大类，第一类是 TCP 三次握手期间的异常，第二类是 TCP 四次挥手期间的异常。

12. TCP 三次握手期间的异常

我们先来看看 TCP 三次握手是怎样的。
three handshake

12.1 第一次握手丢失了，会发生什么？

当客户端想和服务端建立 TCP 连接的时候，首先第一个发的就是 SYN 报文，然后进入到SYN_SENT状态。
在这之后，如果客户端迟迟收不到服务端的 SYN-ACK 报文（第二次握手），就会触发超时重传机制。
不同版本的操作系统可能超时时间不同，有的 1 秒的，也有 3 秒的，这个超时时间是写死在内核里的，如果想要更改则需要重新编译内核，比较麻烦。
当客户端在 1 秒后没收到服务端的 SYN-ACK 报文后，客户端就会重发 SYN 报文，那到底重发几次呢？
在 Linux 里，客户端的 SYN 报文最大重传次数由tcp_syn_retries内核参数控制，这个参数是可以自定义的，默认值一般是 5。
通常，第一次超时重传是在 1 秒后，第二次超时重传是在 2 秒，第三次超时重传是在 4 秒后，第四次超时重传是在 8 秒后，第五次是在超时重传 16 秒后。没错，每次超时的时间是上一次的 2 倍。
当第五次超时重传后，会继续等待 32 秒，如果服务端仍然没有回应 ACK，客户端就不再发送 SYN 包，然后断开 TCP 连接。
所以，总耗时是 1+2+4+8+16+32=63 秒，大约 1 分钟左右。

12.2 第二次握手丢失了，会发生什么？

当服务端收到客户端的第一次握手后，就会回 SYN-ACK 报文给客户端，这个就是第二次握手，此时服务端会进入SYN_RCVD状态。
第二次握手的SYN-ACK报文其实有两个目的：

第二次握手里的 ACK，是对第一次握手的确认报文；
第二次握手里的 SYN，是服务端发起建立 TCP 连接的报文；

所以，如果第二次握手丢了，就会发送比较有意思的事情，具体会怎么样呢？
因为第二次握手报文里是包含对客户端的第一次握手的 ACK 确认报文，所以，如果客户端迟迟没有收到第二次握手，那么客户端就觉得可能自己的 SYN 报文（第一次握手）丢失了，于是客户端就会触发超时重传机制，重传 SYN 报文。
然后，因为第二次握手中包含服务端的 SYN 报文，所以当客户端收到后，需要给服务端发送 ACK 确认报文（第三次握手），服务端才会认为该 SYN 报文被客户端收到了。
那么，如果第二次握手丢失了，服务端就收不到第三次握手，于是服务端这边会触发超时重传机制，重传 SYN-ACK 报文。
在 Linux 下，SYN-ACK 报文的最大重传次数由tcp_synack_retries内核参数决定，默认值是 5。
因此，当第二次握手丢失了，客户端和服务端都会重传：

客户端会重传 SYN 报文，也就是第一次握手，最大重传次数由 tcp_syn_retries内核参数决定。；
服务端会重传 SYN-AKC 报文，也就是第二次握手，最大重传次数由 tcp_synack_retries 内核参数决定。

12.3 第三次握手丢失了，会发生什么？

客户端收到服务端的 SYN-ACK 报文后，就会给服务端回一个 ACK 报文，也就是第三次握手，此时客户端状态进入到ESTABLISH状态。
因为这个第三次握手的 ACK 是对第二次握手的 SYN 的确认报文，所以当第三次握手丢失了，如果服务端那一方迟迟收不到这个确认报文，就会触发超时重传机制，重传 SYN-ACK 报文，直到收到第三次握手，或者达到最大重传次数。
注意，ACK 报文是不会有重传的，当 ACK 丢失了，就由对方重传对应的报文。

13. TCP 四次挥手期间的异常

我们再来看看 TCP 四次挥手的过程。 waved four times

13.1 第一次挥手丢失了，会发生什么？

当客户端（主动关闭方）调用 close 函数后，就会向服务端发送 FIN 报文，试图与服务端断开连接，此时客户端的连接进入到FIN_WAIT_1状态。
正常情况下，如果能及时收到服务端（被动关闭方）的 ACK，则会很快变为FIN_WAIT2状态。
如果第一次挥手丢失了，那么客户端迟迟收不到被动方的 ACK 的话，也就会触发超时重传机制，重传 FIN 报文，重发次数由tcp_orphan_retries参数控制。
当客户端重传 FIN 报文的次数超过tcp_orphan_retries后，就不再发送 FIN 报文，直接进入到close状态。

13.2 第二次挥手丢失了，会发生什么？

当服务端收到客户端的第一次挥手后，就会先回一个 ACK 确认报文，此时服务端的连接进入到CLOSE_WAIT状态。
在前面我们也提了，ACK 报文是不会重传的，所以如果服务端的第二次挥手丢失了，客户端就会触发超时重传机制，重传 FIN 报文，直到收到服务端的第二次挥手，或者达到最大的重传次数。
这里提一下，当客户端收到第二次挥手，也就是收到服务端发送的 ACK 报文后，客户端就会处于FIN_WAIT2状态，在这个状态需要等服务端发送第三次挥手，也就是服务端的 FIN 报文。
对于 close 函数关闭的连接，由于无法再发送和接收数据，所以FIN_WAIT2状态不可以持续太久，而
tcp_fin_timeout控制了这个状态下连接的持续时长，默认值是 60 秒。
这意味着对于调用 close 关闭的连接，如果在 60 秒后还没有收到 FIN 报文，客户端（主动关闭方）的连接就会直接关闭。

13.3 第三次挥手丢失了，会发生什么？

当服务端（被动关闭方）收到客户端（主动关闭方）的 FIN 报文后，内核会自动回复 ACK，同时连接处于CLOSE_WAIT状态，顾名思义，它表示等待应用进程调用 close 函数关闭连接。
此时，内核是没有权利替代进程关闭连接，必须由进程主动调用 close 函数来触发服务端发送 FIN 报文。
服务端处于 CLOSE_WAIT 状态时，调用了 close 函数，内核就会发出 FIN 报文，同时连接进入LAST_ACK 状态，等待客户端返回 ACK 来确认连接关闭。
如果迟迟收不到这个 ACK，服务端就会重发 FIN 报文，重发次数仍然由tcp_orphan_retries 参数控制，这与客户端重发 FIN 报文的重传次数控制方式是一样的。

13.4 The fourth wave is lost, what happens?

When the client receives the FIN message of the third waving from the server, it will return an ACK message, which is the fourth waving, and the client connection enters the TIME_WAIT state.
In the Linux system, the TIME_WAIT state will last for 60 seconds before entering the shutdown state.
Then, before the server (passive closing party) does not receive the ACK message, it is still in the LAST_ACK state.
If the ACK message of the fourth wave does not reach the server, the server will resend the FIN message, and the number of retransmissions is still controlled by the tcp_orphan_retries parameter introduced earlier.
Yes, TCP is very smart!

14. Related Articles

Three-way handshake and four-way wave
Detailed explanation of "three-way handshake" and "four-way wave" of TCP connection
Abnormal TCP termination (reset packet)
Too much TIMEWAIT in TCP
Three communication methods - simplex, half-duplex and duplex communication
Why does HTTP 3.0 completely abandon TCP
DoS and DDoS attacks
Solve the problem caused by too much Linux TIME_WAIT https://blog.csdn.net/zhangjunli/article/details/89321202
tcp connection memory
https://zhuanlan.zhihu.com/p/530809991
http://t.zoukankan.com/xiangsikai-p-9560569.html

15. Future plans

The relationship between TCP connection and memory will be further explored later