Hello everyone, this article combines specific problem examples to explain in detail the heartbeat mechanism, packet loss retransmission mechanism, etc. of the TCP/IP protocol stack to provide you with a reference.
1. Problem overview
Although the underlying module of the software can automatically reconnect to the server after the network is restored, the conference has exited due to network problems and needs to be rejoined.
Due to the customer's special network operating environment, network jitter and instability will occur frequently. The customer requires that the meeting must be maintained after the network is restored within 60 seconds to ensure that the meeting process is not interrupted.
The customer insists on realizing this special function point. The project is nearing completion and is currently in the customer trial stage. If this function is not implemented, the project will not pass acceptance and the customer will not pay.
Colleagues at the front reported the current problems and project progress to the R&D department leaders, and the R&D department held an emergency discussion meeting to discuss a plan to ensure that the meeting would not be missed for 60 seconds.
This involves two major types of network connections, one is a TCP connection that transmits control signaling, and the other is a UDP connection that transmits audio and video streams. The problem of UDP connection is not big, the main problem is the disconnection and reconnection of TCP connection. The following mainly discusses the problems related to TCP connection.
When the network is unstable and the session is lost, it may be that the system TCPIP protocol stack has detected a network abnormality and the system protocol layer has disconnected the network; it may also be that the heartbeat mechanism of the software application layer has detected a network failure and disconnected the connection with the server. Link.
For network abnormalities detected by the system TCPIP protocol stack itself, there may be two situations. One is detected by the heartbeat mechanism of the TCPIP protocol stack itself; the other is detected by the packet loss and retransmission mechanism of the TCP connection.
For the heartbeat detection mechanism of the application layer, we can enlarge the timeout detection time. In this article, we mainly discuss the heartbeat, packet loss retransmission, connection timeout and other mechanisms of the TCP connection of the TCPIP protocol stack.
After detecting a network abnormality, our bottom layer can automatically initiate reconnection or trigger automatic reconnection by sending signaling. The business module will save the conference-related resources and not release them. After the network is restored, it can continue to be in the conference and can continue to receive conference information. audio and video code stream, you can continue to perform some operations in the meeting!
2. Heartbeat mechanism of TCPIP protocol stack
2.1. ACK mechanism in TCP
The three-way handshake process during TCP link establishment is as follows:
The reason why TCP connections are reliable is that the connection must be established before sending data, and then after receiving the data, an ACK packet will be restored to the other party, indicating that I have received your data packet. For the data sending end, if no ACK packet is received after the data is sent, the packet loss retransmission mechanism will be triggered.
Whether it is when establishing a link or when sending and receiving data after establishing a link, there are ACK packets, and the heartbeat packet of the TCP/IP protocol stack is no exception.
2.2. Description of heartbeat mechanism of TCPIP protocol stack
The TCP/IP protocol stack has a default TCP heartbeat mechanism. This heartbeat mechanism is bound to the socket (TCP socket). The heartbeat detection mechanism of the protocol stack can be turned on for the specified socket. By default, the heartbeat mechanism of the protocol stack is closed for sockets. If you want to use it, you need to turn it on manually.
In Windows, the default is to send a heartbeat packet every 2 hours. After the client program sends the heartbeat packet to the server, there will be two situations:
1) When the network is normal: When the server receives the heartbeat packet, it will immediately reply with an ACK packet. After the client receives the ACK packet, it will wait another 2 hours to send the next heartbeat packet. Among them, the heartbeat packet sending interval keepalivetime is 2 hours by default in Windows systems and can be configured. If there is data interaction between the client and the server within the 2-hour interval, the client will receive an ACK packet from the server, which is also counted as a heartbeat packet for the heartbeat mechanism, and the 2-hour interval will be re-timed.
2) When the network is abnormal: the server cannot receive the heartbeat packet sent by the client and cannot reply ACK. The default timeout in Windows system is 1 second, and the heartbeat packet will be resent after 1 second. If the ACK of the heartbeat packet is not received, the heartbeat packet will be resent after 1 second. If the heartbeat packet is still not received, the upper limit of the system will be reached after sending 10 heartbeat packets, and the network is considered to be faulty. The protocol stack The connection will be disconnected directly. Among them, the timeout period for sending a heartbeat packet and failing to receive an ACK is called keepaliveinterval. The default in Windows systems is 1 second and is configurable; the number of retransmissions of ACK packets that cannot be received corresponding to the heartbeat packet probe is fixed in Windows systems. 10 times, non-configurable.
Therefore, the heartbeat mechanism of the TCP/IP protocol stack can also detect network anomalies, but it may take a long time to detect it under the default configuration, unless the network anomaly occurs while waiting for a response from the peer after sending a heartbeat packet. In this case, if there are multiple If no ACK response is received after resending the heartbeat packet, the protocol stack will determine that the network is faulty and actively close the connection.
2.3. Modify the default heartbeat parameters of the TCP/IP protocol stack
The default heartbeat mechanism of the TCP/IP protocol stack does not enable heartbeat monitoring for the entire protocol stack of the system, but for a certain socket.
After turning on the heartbeat mechanism, you can also modify the time parameters of the heartbeat. From the code point of view, first call setsockopt to enable the heartbeat monitoring mechanism for the target socket, and then call WSAIoctl to modify the default time parameters of heartbeat detection. The relevant code is as follows:
SOCKET socket;
// ......(中间代码省略)
int optval = 1;
int nRet = setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, (const char *)&optval,
sizeof(optval));
if (nRet != 0)
return;
tcp_keepalive alive;
alive.onoff = TRUE;
alive.keepalivetime = 10*1000;
alive.keepaliveinterval = 2*1000;
DWORD dwBytesRet = 0;
nRet = WSAIoctl(socket, SIO_KEEPALIVE_VALS, &alive, sizeof(alive), NULL, 0,
&dwBytesRet, NULL, NULL);
if (nRet != 0)
return;
As you can see from the above code, the setsockopt function is called first, the SO_KEEPALIVE parameter is passed in, and the heartbeat switch of the TCP connection is turned on. At this time, the heartbeat parameter uses the system's default heartbeat parameter value.
Then, call the WSAIoCtrl function, pass in the SIO_KEEPALIVE_VALS parameter, and pass in the heartbeat parameter structure with the set time value.
The following is a detailed explanation of the heartbeat parameter structure tcp_keepalive: (taking Windows system as an example)
1) keepalivetime: By default, a heartbeat keepalive packet is sent every 2 hours. For example, after sending the first keepalive packet, the next keepalive packet will be sent after an interval of 2 hours. If there is data interaction during this period, it is considered a valid keep-alive packet. No keep-alive packet will be sent during this period. The time interval for sending the next keep-alive packet will start from 0 again from the time of the last piece of data sent and received.
2) keepaliveinterval: After sending a keep-alive packet, the timeout for not receiving an ack from the peer is 1 second by default. Suppose there is a problem with the network with the peer. Send the first keep-alive packet to the peer. If no ack is received from the peer within 1 second, then send the second keep-alive packet. No keep-alive packet is received from the peer within 1 second. , and then send the next keep-alive packet,..., until after sending the 10th keep-alive packet, if no ack response is received within 1 second, the upper limit of detection times of sending 10 keep-alive packets is reached, and the network is considered there is a problem.
3) Number of probe detections: The number of probes on Windows systems is fixed at 10 times and cannot be modified.
The description on MSDN of network anomalies detected by the heartbeat mechanism is as follows:
If a connection is dropped as the result of keep-alives the error code WSAENETRESET is returned to any calls in progress on the socket, and any subsequent calls will fail with WSAENOTCONN.
Because the keep-alive count reaches the upper limit and the connection is discarded, all socket interfaces being called will return the WSAENETRESET error code, and subsequent calls to the socket api function will return WSAENOTCONN.
3. The heartbeat mechanism in the libwebsockets open source library uses the heartbeat mechanism of the TCPIP protocol stack.
When our products used websocket before, we encountered the problem that the heartbeat mechanism was not set, causing long TCP connections to be released by the network device for no reason.
When our client program logs in, it will connect to the registration server of a certain business and establish a long websocket connection. This long connection is always maintained. This connection is only used when using the business of this business module, and data is exchanged on this connection. After logging in to the software, if the business module has not been operated, this long connection will remain idle, that is, there is no data interaction on this connection.
As a result, a problem occurred during a certain test. After investigation, it was found that this long connection was closed by the intermediate network device because there was no data interaction for a long time. Later, in order to solve this problem, we set the heartbeat parameters when initializing the websocket library, so that the above-mentioned long websocket connection can run a heartbeat packet when it is idle, thus ensuring that the long connection will not be closed for no reason because there is no data running for a long time. problem.
When we call the lws_create_context interface to create a websockets session context, the structure parameter lws_context_creation_info of the interface has a field for setting heartbeat parameters:
/**
* struct lws_context_creation_info - parameters to create context with
*
* This is also used to create vhosts.... if LWS_SERVER_OPTION_EXPLICIT_VHOSTS
* is not given, then for backwards compatibility one vhost is created at
* context-creation time using the info from this struct.
*
* If LWS_SERVER_OPTION_EXPLICIT_VHOSTS is given, then no vhosts are created
* at the same time as the context, they are expected to be created afterwards.
*
* @port: VHOST: Port to listen on... you can use CONTEXT_PORT_NO_LISTEN to
* suppress listening on any port, that's what you want if you are
* not running a websocket server at all but just using it as a
* client
* @iface: VHOST: NULL to bind the listen socket to all interfaces, or the
* interface name, eg, "eth2"
* If options specifies LWS_SERVER_OPTION_UNIX_SOCK, this member is
* the pathname of a UNIX domain socket. you can use the UNIX domain
* sockets in abstract namespace, by prepending an @ symbole to the
* socket name.
* @protocols: VHOST: Array of structures listing supported protocols and a protocol-
* specific callback for each one. The list is ended with an
* entry that has a NULL callback pointer.
* It's not const because we write the owning_server member
* @extensions: VHOST: NULL or array of lws_extension structs listing the
* extensions this context supports. If you configured with
* --without-extensions, you should give NULL here.
* @token_limits: CONTEXT: NULL or struct lws_token_limits pointer which is initialized
* with a token length limit for each possible WSI_TOKEN_***
* @ssl_cert_filepath: VHOST: If libwebsockets was compiled to use ssl, and you want
* to listen using SSL, set to the filepath to fetch the
* server cert from, otherwise NULL for unencrypted
* @ssl_private_key_filepath: VHOST: filepath to private key if wanting SSL mode;
* if this is set to NULL but sll_cert_filepath is set, the
* OPENSSL_CONTEXT_REQUIRES_PRIVATE_KEY callback is called
* to allow setting of the private key directly via openSSL
* library calls
* @ssl_ca_filepath: VHOST: CA certificate filepath or NULL
* @ssl_cipher_list: VHOST: List of valid ciphers to use (eg,
* "RC4-MD5:RC4-SHA:AES128-SHA:AES256-SHA:HIGH:!DSS:!aNULL"
* or you can leave it as NULL to get "DEFAULT"
* @http_proxy_address: VHOST: If non-NULL, attempts to proxy via the given address.
* If proxy auth is required, use format
* "username:password@server:port"
* @http_proxy_port: VHOST: If http_proxy_address was non-NULL, uses this port at
* the address
* @gid: CONTEXT: group id to change to after setting listen socket, or -1.
* @uid: CONTEXT: user id to change to after setting listen socket, or -1.
* @options: VHOST + CONTEXT: 0, or LWS_SERVER_OPTION_... bitfields
* @user: CONTEXT: optional user pointer that can be recovered via the context
* pointer using lws_context_user
* @ka_time: CONTEXT: 0 for no keepalive, otherwise apply this keepalive timeout to
* all libwebsocket sockets, client or server
* @ka_probes: CONTEXT: if ka_time was nonzero, after the timeout expires how many
* times to try to get a response from the peer before giving up
* and killing the connection
* @ka_interval: CONTEXT: if ka_time was nonzero, how long to wait before each ka_probes
* attempt
* @provided_client_ssl_ctx: CONTEXT: If non-null, swap out libwebsockets ssl
* implementation for the one provided by provided_ssl_ctx.
* Libwebsockets no longer is responsible for freeing the context
* if this option is selected.
* @max_http_header_data: CONTEXT: The max amount of header payload that can be handled
* in an http request (unrecognized header payload is dropped)
* @max_http_header_pool: CONTEXT: The max number of connections with http headers that
* can be processed simultaneously (the corresponding memory is
* allocated for the lifetime of the context). If the pool is
* busy new incoming connections must wait for accept until one
* becomes free.
* @count_threads: CONTEXT: how many contexts to create in an array, 0 = 1
* @fd_limit_per_thread: CONTEXT: nonzero means restrict each service thread to this
* many fds, 0 means the default which is divide the process fd
* limit by the number of threads.
* @timeout_secs: VHOST: various processes involving network roundtrips in the
* library are protected from hanging forever by timeouts. If
* nonzero, this member lets you set the timeout used in seconds.
* Otherwise a default timeout is used.
* @ecdh_curve: VHOST: if NULL, defaults to initializing server with "prime256v1"
* @vhost_name: VHOST: name of vhost, must match external DNS name used to
* access the site, like "warmcat.com" as it's used to match
* Host: header and / or SNI name for SSL.
* @plugin_dirs: CONTEXT: NULL, or NULL-terminated array of directories to
* scan for lws protocol plugins at context creation time
* @pvo: VHOST: pointer to optional linked list of per-vhost
* options made accessible to protocols
* @keepalive_timeout: VHOST: (default = 0 = 60s) seconds to allow remote
* client to hold on to an idle HTTP/1.1 connection
* @log_filepath: VHOST: filepath to append logs to... this is opened before
* any dropping of initial privileges
* @mounts: VHOST: optional linked list of mounts for this vhost
* @server_string: CONTEXT: string used in HTTP headers to identify server
* software, if NULL, "libwebsockets".
*/
struct lws_context_creation_info {
int port; /* VH */
const char *iface; /* VH */
const struct lws_protocols *protocols; /* VH */
const struct lws_extension *extensions; /* VH */
const struct lws_token_limits *token_limits; /* context */
const char *ssl_private_key_password; /* VH */
const char *ssl_cert_filepath; /* VH */
const char *ssl_private_key_filepath; /* VH */
const char *ssl_ca_filepath; /* VH */
const char *ssl_cipher_list; /* VH */
const char *http_proxy_address; /* VH */
unsigned int http_proxy_port; /* VH */
int gid; /* context */
int uid; /* context */
unsigned int options; /* VH + context */
void *user; /* context */
int ka_time; /* context */
int ka_probes; /* context */
int ka_interval; /* context */
#ifdef LWS_OPENSSL_SUPPORT
SSL_CTX *provided_client_ssl_ctx; /* context */
#else /* maintain structure layout either way */
void *provided_client_ssl_ctx;
#endif
short max_http_header_data; /* context */
short max_http_header_pool; /* context */
unsigned int count_threads; /* context */
unsigned int fd_limit_per_thread; /* context */
unsigned int timeout_secs; /* VH */
const char *ecdh_curve; /* VH */
const char *vhost_name; /* VH */
const char * const *plugin_dirs; /* context */
const struct lws_protocol_vhost_options *pvo; /* VH */
int keepalive_timeout; /* VH */
const char *log_filepath; /* VH */
const struct lws_http_mount *mounts; /* VH */
const char *server_string; /* context */
/* Add new things just above here ---^
* This is part of the ABI, don't needlessly break compatibility
*
* The below is to ensure later library versions with new
* members added above will see 0 (default) even if the app
* was not built against the newer headers.
*/
void *_unused[8];
};
The three fields ka_time, ka_probes and ka_interval are the heartbeat-related setting parameters. Our code to initialize the websockets context is as follows:
static lws_context* CreateContext()
{
lws_set_log_level( 0xFF, NULL );
lws_context* plcContext = NULL;
lws_context_creation_info tCreateinfo;
memset(&tCreateinfo, 0, sizeof tCreateinfo);
tCreateinfo.port = CONTEXT_PORT_NO_LISTEN;
tCreateinfo.protocols = protocols;
tCreateinfo.ka_time = LWS_TCP_KEEPALIVE_TIME;
tCreateinfo.ka_interval = LWS_TCP_KEEPALIVE_INTERVAL;
tCreateinfo.ka_probes = LWS_TCP_KEEPALIVE_PROBES;
tCreateinfo.options = LWS_SERVER_OPTION_DISABLE_IPV6;
plcContext = lws_create_context(&tCreateinfo);
return plcContext;
}
By consulting the libwebsockets open source library code, we know that the heartbeat set here uses the heartbeat mechanism of the TCPIP protocol stack, as shown below:
LWS_VISIBLE int
lws_plat_set_socket_options(struct lws_vhost *vhost, lws_sockfd_type fd)
{
int optval = 1;
int optlen = sizeof(optval);
u_long optl = 1;
DWORD dwBytesRet;
struct tcp_keepalive alive;
int protonbr;
#ifndef _WIN32_WCE
struct protoent *tcp_proto;
#endif
if (vhost->ka_time) {
/* enable keepalive on this socket */
// 先调用setsockopt打开发送心跳包（设置）选项
optval = 1;
if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE,
(const char *)&optval, optlen) < 0)
return 1;
alive.onoff = TRUE;
alive.keepalivetime = vhost->ka_time*1000;
alive.keepaliveinterval = vhost->ka_interval*1000;
if (WSAIoctl(fd, SIO_KEEPALIVE_VALS, &alive, sizeof(alive),
NULL, 0, &dwBytesRet, NULL, NULL))
return 1;
}
/* Disable Nagle */
optval = 1;
#ifndef _WIN32_WCE
tcp_proto = getprotobyname("TCP");
if (!tcp_proto) {
lwsl_err("getprotobyname() failed with error %d\n", LWS_ERRNO);
return 1;
}
protonbr = tcp_proto->p_proto;
#else
protonbr = 6;
#endif
setsockopt(fd, protonbr, TCP_NODELAY, (const char *)&optval, optlen);
/* We are nonblocking... */
ioctlsocket(fd, FIONBIO, &optl);
return 0;
}
4. TCPIP packet loss retransmission mechanism
If there is a network failure and TCP data interaction is in progress between the client and the server, and the client cannot receive the ACK packet from the server due to the network failure after sending the data packet to the server, the client's TCP packet loss and retransmission will be triggered, and the packet loss will occur. The retransmission mechanism can also determine if there is an abnormality in the network.
For TCP connections, if the client does not receive an ACK packet from the server after sending data to the server, packet loss and retransmission will be triggered. The time interval for each retransmission will be doubled. When the number of retransmissions reaches the system upper limit (the default upper limit for Windows is 5 times, and the default upper limit for Linux is 15 times), the protocol stack will think that the network has failed and will directly transfer the corresponding data. The connection is closed.
Therefore, when there is data interaction when the network fails, the protocol stack will detect an abnormality in the network within tens of seconds and close the connection directly. The detailed description of the packet loss retransmission mechanism is as follows:
For the packet loss retransmission mechanism, you can check it by plugging and unplugging the network cable to the PC, or you can use wireshark to capture the packet. When quickly plugging and unplugging the network cable (unplug the network cable first, wait a few seconds and then plug it in), the operation instructions sent to the server will receive data due to packet loss and retransmission.
5. Use non-blocking socket and select interface to implement timeout control of connect connection
5.1. Description of connect and select interfaces on MSDN
For tcp sockets, we need to call the socket function connect to establish a TCP connection. Let’s first take a look at the description of the socket interface connect on Microsoft MSDN:
On a blocking socket, the return value indicates success or failure of the connection attempt.
For blocking sockets, the return value of connect can be used to determine whether the connection is successful. Returning 0 indicates that the connection is successful.
With a nonblocking socket, the connection attempt cannot be completed immediately. In this case, connect will return SOCKET_ERROR, and WSAGetLastError will return WSAEWOULDBLOCK. In this case, there are three possible scenarios: Use the select function to determine the completion of the connection request by checking to see if the socket is writeable.
For non-group sockets, the connect call will return immediately, but the connection operation has not yet been completed. connect returns SOCKET_ERROR. For non-blocking sockets, returning SOCKET_ERROR does not mean failure. You need to call WSAGetLastError to obtain the LastError value after the connect function is executed. Generally, WSAGetLastError will return WSAEWOULDBLOCK at this time:
Indicates that the connection is in progress. You can use the select interface to check whether the socket is writable (whether the socket is in the writefds collection). If it is writable, the connection is successful. If the socket is in the exceptfds collection, it means that an exception occurred in the connection, as shown below:
5.2. Use non-blocking sockets and select to control connection timeout
For blocking sockets, under Windows, if the remote IP and Port are unreachable, SOCKET_ERROR will be returned after blocking for 75s, indicating that the connection failed. So when we test whether the remote IP and Port can be connected, we do not use a blocking socket, but a non-blocking socket, and then call select to add the connection timeout through select to control the connection timeout.
The select function will return 0 because it times out; if an error occurs, it will return SOCKET_ERROR, so the select return value must be judged when making a judgment. If it is less than or equal to 0, the connection fails and the socket is closed immediately.
If the select return value is greater than 0, the return value is the number of ready sockets, such as the socket with a successful connection. We determine whether the socket is in the writable collection writefds. If it is in this collection, the connection is successful.
According to the relevant description on MSDN, we can roughly know how to implement the timeout control of connect. The relevant code is as follows:
bool ConnectDevice( char* pszIP, int nPort )
{
// Create TCP socket
SOCKET connSock = socket(AF_INET, SOCK_STREAM, 0);
if (connSock == INVALID_SOCKET)
{
return false;
}
// Fill in IP and port
SOCKADDR_IN devAddr ;
memset(&devAddr, 0, sizeof(SOCKADDR_IN));
devAddr.sin_family = AF_INET;
devAddr.sin_port = htons(nPort);
devAddr.sin_addr.s_addr = inet_addr(pszIP);
// Set the socket to non-blocking , prepare for the following selection
unsigned long ulnoblock = 1;
ioctlsocket(connSock, FIONBIO, &ulnoblock);
// Initiate connnect, the interface immediately returns
connect(connSock, (sockaddr*)&devAddr, sizeof(devAddr));
FD_SET writefds ;
FD_ZERO(&writefds);
FD_SET(connSock, &writefds);
//Set the connection timeout to 1 second
timeval tv;
tv.tv_sec = 1; //Timeout 1s
tv.tv_usec = 0;
// The select function returns the total number of socket handles that are ready and contained
// in the fd_set structures, zero if the time limit expired, or SOCKET_ERROR(-1) if an error occurred.
if (select(0, NULL, &writefds, NULL, &tv) <= 0)
{
closesocket(connSock) ;
return false; //Exit if timeout fails
}
ulnoblock = 0;
ioctlsocket(connSock, FIONBIO, &ulnoblock);
closesocket(connSock);
return true;
}