Overview of sticky bag issues
Describe background
In the software design that uses TCP protocol for network data transmission, there is a common problem of sticky packets. This is mainly due to the network transmission mechanism of modern operating systems.
We know that the socket technology used in network communication is actually implemented by the system kernel providing a continuous cache (stream buffer) to implement the transfer function between the application layer program and the network card interface.
Multiple data packets are continuously stored in continuous buffers. When reading data packets, since the sending boundary of the sender cannot be determined, a certain estimated value size is used to read the data. If the sizes of both parties are inconsistent, This will cause the boundaries of the data packets to be misaligned, leading to the reading of erroneous data packets, thus misinterpreting the meaning of the original data.
The concept of sticky bags
The essence of the sticky problem is caused by data reading boundary errors. The following figure can visually understand the phenomenon.
As shown in Figure 1, 6 data packets have arrived in the current socket cache, and their sizes are as shown in the figure. When the application collects data (as shown in Figure 2), it adopts the requirement of 300 bytes to read, and will mistakenly collect pkg1 and pkg2 together as one package.
In fact, it is very likely that pkg1 is the content of a text file, while pkg2 may be an audio content. These two unrelated data packets are lumped into one packet for processing, which is obviously inappropriate. In serious cases, the loss of pkg2 may cause the software to fall into an abnormal branch and cause an own incident.
Therefore, the problem of sticky packages must attract the great attention of all software designers (project managers)!
Then, some readers may ask, why not let the receiving program read according to 100 bytes? I think if you know some TCP programming you won't have such a problem.
In network communication programs, the size of data packets usually cannot be determined, especially in the software design stage, which cannot really be determined as a fixed value. For example, if the chat software client uses TCP to transmit a username and password to the server for verification and login, I think the data packet is only a few tens of bytes, or at most a few hundred bytes, and it can be sent. Sometimes a very large packet needs to be transmitted. Even if the video file is sent in packages, it should be several kilobytes per package. (It is said that the MW of a certain country's telecommunications platform has seen 15,000 bytes of telephone data being sent at one time).
In this case, the packet size of the sent data cannot be fixed, and the receiving end cannot be fixed. Therefore, a more reasonable estimate is generally used for polling reception. (The MTU of the network card is 1500 bytes, so this estimated value is generally 1 to 3 times the MTU).
I believe readers should have a preliminary understanding of the problem of sticky bags.
Sticky bag avoidance design
1: Send at fixed length
A fixed-length design is used when sending data. That is, no matter how much data is sent, it is packetized into a fixed length (for the convenience of description, the fixed length is recorded as LEN here), that is, when the sender sends data, it is packetized with LEN as the length.
In this way, the receiver receives with a fixed LEN, so that sending and receiving can correspond one to one. When sub-packaging, it may not be completely divided into multiple complete LEN packets. The last packet will generally be smaller than LEN. At this time, the last packet can fill the missing part with blank bytes.
Of course, this approach has drawbacks:
1. The insufficient length of the last packet is filled with blank parts, which is invalid byte order. Then the receiver may have difficulty identifying this invalid part. It is just for filling in and has no actual meaning. This creates trouble for the receiving end to process its meaning. Of course, there are solutions, which can be compensated by adding flag bits, that is, adding a fixed-length header to the front of each data packet, and then sending the end mark of the data packet together. The receiver confirms the invalid byte sequence according to this mark, thereby achieving complete reception of the data.
2. When the length of sent packets is randomly distributed, bandwidth will be wasted. For example, the sending length may be 1,100, 1000, 4000 bytes, etc., and they all need to be sent according to the maximum fixed length, which is 4000. Other packets with data packets smaller than 4000 bytes will also be filled to 4000, causing ineffective waste of network load. .
In summary, this solution is suitable for situations where the length of the sent data packet is relatively stable (tends to a certain fixed value) and has better results.
2: Tail mark sequence
Set a special byte sequence at the end of each data packet to be sent. This sequence has a special meaning, which is the same as the end character identifier "\0" of the string. It is used to mark the end of the data packet. Only then can the received data be analyzed and the boundaries of the data packets confirmed through the tail sequence.
The shortcomings of this method are more obvious:
1. The receiver needs to analyze the data and identify tail sequences.
2. The determination of the tail sequence itself is a problem. What sequence can be used as a terminator like "\0"? This sequence must be a data sequence that does not have any meaning that is generally recognized by humans or programs, just like "\0" is an invalid string content and can therefore be used as the end mark of the string. So what is this sequence in ordinary network communication? I think it's hard to find the right answer for a while.
Three: Step-by-step reception of header marks
This method is the best method based on the author's limited knowledge. It does not lose efficiency and perfectly solves the boundary problem of packets of any size.
The implementation of this method is as follows:
1. Define a user header and indicate the size of each data packet sent in the header.
2. Each time the receiver receives, it first reads the data with the size of the header. This will inevitably only read the data of one header, and obtain the data size of the data packet from the header.
3. Read again according to this size, and you can read the content of the data.
In this way, each data packet is encapsulated with a header when it is sent, and then the receiver receives a packet in two times, the first time to receive the header, and the second time to receive the data content based on the size of the header.
(The essence of data[0] here is a pointer, pointing to the text part of the data, or it can be the starting position of a continuous data area. Therefore, it can be designed as data[user_size], in this case.)
The following is a diagram to show the design idea.
As can be seen from the figure, the data is sent with the action of encapsulating the header; the receiver splits the reception of each packet into two times.
This plan seems to be elegant, but in fact it also has flaws:
1. Although the header is small, each packet needs to encapsulate more sizeof(_data_head) data, and the cumulative effect cannot be completely ignored.
2. The receiving action of the receiver is divided into two times, that is, the data reading operation is doubled, and the recv or read of the data reading operation are both system calls, which is an unacceptable overhead for the kernel. The impact is completely ignored and the performance impact on the program is negligible (system calls are very fast).
Advantages: It avoids the complexity of program design, its effectiveness is easy to verify, and it is easier to meet the stability requirements of software design.
Information Direct: Linux kernel source code technology learning route + video tutorial kernel source code
Learning Express: Linux Kernel Source Code Memory Tuning File System Process Management Device Driver/Network Protocol Stack
Replenish
When do you need to consider sticking bags?
1. If you use tcp to send data every time, you will establish a connection with the other party, and then both parties will close the connection after sending a piece of data, so that there will be no sticky packet problem (because there is only one packet structure, similar to the http protocol).
To close the connection, both parties must send a close connection (refer to the tcp closing protocol). For example: A needs to send a string to B, then A establishes a connection with B, and then sends the protocol characters that both parties have defaulted to, such as "hello give me sth abour yourself", and then after B receives the message, the buffer data Receive, and then close the connection, so that the problem of sticky packets does not need to be considered, because everyone knows that it is sending a paragraph of characters.
2. If the sent data has no structure, such as file transfer, then the sender only needs to send, and the receiver only needs to receive and store, it will be ok, and there is no need to consider packet sticking.
3. If both parties establish a connection, they need to send data of different structures within a period of time after the connection. For example, after the connection, there are several structures:
1)"hello give me sth abour yourself"
2)"Don't give me sth abour yourself"
In this case, if the sender sends these two packets continuously, the receiver may receive "hello give me sth abour yourselfDon't give me sth abour yourself".
This will make the receiving party stupid. What on earth is going on? I don’t know, because the protocol does not stipulate such a weird string, so it needs to be divided into packets. How to divide it also requires both parties to organize a better packet structure, so generally a packet such as data length may be added to the header. Make sure to receive.
Reasons for sticky packets: Occurrence in streaming transmission, UDP will not have sticky packets because it has message boundaries.
1 The sending end needs to wait until the buffer is full before sending out, causing sticky packets;
2 The receiver does not receive the packets in the buffer in time, causing multiple packets to be received;
Solution:
In order to avoid the sticking phenomenon, the following measures can be taken.
First, users can avoid the packet sticking phenomenon caused by the sender through programming settings. TCP provides the push operation command that forces data to be transmitted immediately. After receiving the operation command, the TCP software immediately sends this data out, and No need to wait for the send buffer to be full;
Second, for sticky packets caused by the receiver, measures such as optimizing program design, streamlining the workload of the receiving process, and improving the priority of the receiving process can be used to receive data in a timely manner, thereby minimizing the sticky phenomenon;
The third is controlled by the receiver, which manually controls a packet of data to be received multiple times according to the structure field, and then merged. This method can avoid packet sticking.
The three measures mentioned above all have their shortcomings.
Although the first programming setting method can avoid sticky packets caused by the sender, it turns off the optimization algorithm, reduces network sending efficiency, affects the performance of the application, and is generally not recommended.
The second method can only reduce the possibility of sticky packets, but it cannot completely avoid sticky packets. When the sending frequency is high, or due to network bursts, data packets in a certain period of time may arrive at the receiver faster, and the receiver It may still be too late to receive it, resulting in sticky packets.
Although the third method avoids sticky packets, the application efficiency is low and is not suitable for real-time applications.
Why TCP-based communication programs need to pack and unpack
TCP is a "stream" protocol. The so-called stream is a string of data without boundaries. You can think of the flowing water in a river, which is connected into one piece with no dividing lines. But in general communication program development, you need to define each Independent data packets, such as data packets for login and logout. Due to the characteristics of TCP "flow" and network conditions, the following situations will occur during data transmission.
Suppose we call send twice in a row to send two pieces of data data1 and data2 respectively. There are the following reception situations on the receiving end (of course there are more than these situations, only representative situations are listed here).
A. Data1 is received first, and then data2 is received.
B. First receive part of the data of data1, then receive the remaining part of data1 and all of data2.
C. First received all the data of data1 and part of the data of data2, and then received the remaining data of data2.
D. All the data of data1 and data2 are received at one time.
For situation A, this is exactly what we need and will not be discussed further. For B, C, and D, the situation is what everyone often calls "sticky packets", which requires us to unpack the received data into independent data packets. In order to unpack, the packet must be encapsulated at the sending end.
Another: For UDP, there is no problem of unpacking, because UDP is a "data packet" protocol, that is, there is a boundary between two pieces of data. The receiving end either cannot receive the data, or it receives a complete piece. Data will not be received less or more.
Why does BCD occur?
"Sticking packets" can occur on the sending end or on the receiving end.
1. Sticky packets at the sender caused by Nagle algorithm:
Nagle's algorithm is an algorithm that improves network transmission efficiency. Simply put, when we submit a piece of data to TCP for sending, TCP does not send the data immediately, but waits for a short period of time to see if there is still data to be sent during the waiting period. If so, it will send the data at once. Two pieces of data are sent.
This is a simple explanation of Nagle's algorithm. Please read related books for details. Situations like C and D may be caused by Nagle's algorithm.
2. The receiving end does not receive packets in time due to the receiving end’s sticky packets:
TCP will store the received data in its own buffer, and then notify the application layer to retrieve the data. When the application layer cannot take out TCP data in time due to some reasons, several pieces of data will be stored in the TCP buffer.
How to pack and unpack
When I first encountered the problem of "sticky packets", I solved it by calling sleep for a short period of time between two sends.
The disadvantages of this solution are obvious, which greatly reduces the transmission efficiency and is unreliable. Later, it was solved through the response method. Although it is feasible most of the time, it cannot solve the situation like B. Moreover, the response method increases the communication volume and increases the network load. The next step is to pack and unpack the data packets.
Packaging is to add a header to a piece of data, so that the data packet is divided into two parts: the header and the packet body (later, when filtering illegal packets, the packet will add "packet tail" content).
The header is actually a structure with a fixed size. There is a structure member variable that represents the length of the package. This is a very important variable. Other structure members can be defined as needed. According to the fixed length of the packet header and the variable containing the packet body length in the packet header, a complete data packet can be correctly split.
For unpacking , I currently use the following two methods most often.
1. Dynamic buffer temporary storage method. The reason why the buffer is dynamic is that when the length of the data to be buffered exceeds the length of the buffer, the buffer length will be increased. The approximate process is described as follows:
A. Dynamically allocate a buffer for each connection, and associate the buffer with SOCKET, usually through a structure. B. When receiving data, first store this data in the buffer. C. Judgment Whether the data length in the buffer area is enough for the length of a packet header, if not, no unpacking operation will be performed. D. Parse the variables representing the length of the packet body based on the packet header data. E. Determine the length of the data in the buffer area except for the packet header. Is the length of a packet body sufficient? If not, the unpacking operation will not be performed. F, take out the entire data packet. "Get" here means not only copying the data packet from the buffer, but also deleting the data packet from the buffer. The method of deletion is to move the data behind the packet to the starting address of the buffer.
This method has two disadvantages: 1. Dynamically allocating a buffer for each connection increases memory usage. 2. There are three places where data needs to be copied, one is to store the data in the buffer, one is to take out the complete data packet from the buffer, and one is to delete the data packet from the buffer.
The disadvantages of this approach were mentioned earlier. An improvement method is given below, that is, using a ring buffer. However, this improvement method still cannot solve the first shortcoming and the first data copy. It can only solve the data copy in the third place (this place is the place where the most data is copied). ). The second unpacking method will solve these two problems.
The ring buffer implementation is to define two pointers, pointing to the head and tail of valid data respectively. When storing and deleting data, only the head and tail pointers are moved.
2. Use the underlying buffer to unpack
Since TCP also maintains a buffer, we can completely use the TCP buffer to cache our data, so that we do not need to allocate a buffer for each connection. On the other hand, we know that recv or wsarecv has a parameter to indicate the length of data we want to receive. Using these two conditions we can optimize the first method.
For blocking SOCKET, we can use a loop to receive the data of the packet header length, then parse out the variable representing the packet body length, and then use a loop to receive the data of the packet body length. The relevant code is as follows:
char PackageHead[1024];
char PackageContext[1024*20];
int len;
PACKAGE_HEAD *pPackageHead;
while( m_bClose == false )
{
memset(PackageHead,0,sizeof(PACKAGE_HEAD));
len = m_TcpSock.ReceiveSize((char*)PackageHead,sizeof(PACKAGE_HEAD));
if( len == SOCKET_ERROR )
{
break;
}
if(len == 0)
{
break;
}
pPackageHead = (PACKAGE_HEAD *)PackageHead;
memset(PackageContext,0,sizeof(PackageContext));
if(pPackageHead->nDataLen>0)
{
len = m_TcpSock.ReceiveSize((char*)PackageContext,pPackageHead->nDataLen);
}
}m_TcpSock is a variable of a class that encapsulates SOCKET. The ReceiveSize is used to receive data of a certain length and will not return until a certain length of data is received or a network error occurs.
int winSocket::ReceiveSize( char* strData, int iLen )
{
if( strData == NULL )
return ERR_BADPARAM;
char *p = strData;
int len = iLen;
int ret = 0;
int returnlen = 0;
while( len > 0)
{
ret = recv( m_hSocket, p+(iLen-len), iLen-returnlen, 0 );
if ( ret == SOCKET_ERROR || ret == 0 )
{
return ret;
}
len -= ret;
returnlen += ret;
}
return returnlen;
}For non-blocking SOCKETs, such as completion ports, we can submit a request to receive data of the header length. When GetQueuedCompletionStatus returns, we determine whether the length of the received data is equal to the length of the packet header. If it is equal, a request to receive the data of the packet body length is submitted; if it is not equal, a request to receive the remaining data is submitted. A similar approach is used when receiving the package body.
Original author: Learn embedded together
