"Network programming" transport layer protocol _ TCP protocol learning _ and in-depth understanding of principles (1) [Thousand words detailed explanation]

The content of the "Preface" article is roughly the transport layer protocol, the explanation of the TCP protocol, and the continuation of the UDP protocol .

"Belonging column" network programming

"Homepage link" personal homepage

"Author" Mr. Maple Leaf (fy)

1. Introduction to TCP protocol

TCP ( Transmission Control Protocol) is a connection-oriented and reliable transmission protocol. The full name of TCP is
"Transmission Control Protocol". As the name suggests, TCP needs to control the transmission of data in detail.

It is located at the transport layer and is used to transmit data in a computer network

TCP overview:

• transport layer protocol
• connected
• reliable transmission
• stream-oriented

Two, TCP protocol

The format of the TCP protocol is as follows:

The meaning of each field in the TCP header is as follows:

1. 16-bit source port number: indicates which process the data comes from.
2. 16-bit destination port number: Indicates which process the data will go to (the process of the peer host).
3. 32-bit serial number: used to identify the serial number of the byte stream in the TCP data packet for orderly transmission. Follow-up
4. 32-bit confirmation sequence number: used to confirm the sequence number of the byte stream that the receiver has successfully received, and to achieve reliable transmission. Follow-up
5. 4-bit header length: Indicates the length of the TCP header, in units of 4 bytes.
6. Reserved 6 bits: Reserved field, which is not used by TCP temporarily. don't talk
7. 6-bit control flags: including URG, ACK, PSH, RST, SYN, FIN, only six are learned here, and the details will be discussed later
8. 16-bit window size: Used for flow control, indicating how much data the sender can send. Follow-up
9. 16-bit checksum: used to detect errors in TCP headers and data. don't talk
10. 16-bit urgent pointer: identifies which part of the data is urgent data.
11. Options: optional field, used to extend the functionality of the TCP protocol. don't talk

The header is the data, that is, the data dropped by the upper layer. The data includes the upper layer protocol header + data, and this is not discussed.

2.1 Unpacking and sharing

How does TCP separate the header from the payload? i.e. how to unpack

After TCP obtains a TCP message from the lower layer, although TCP does not know the specific length of the header, the first 20 bytes of the message are the basic header of TCP, and these 20 bytes cover the header length of 4 bits.

The process of TCP unpacking is as follows:

• After TCP gets a message, it first reads the first 20 bytes of the message,
• And extract the 4-bit header length from it, and then get the size of the TCP header
• Notice: The length of the TCP basic header is 20 bytes, read 20 bytes without thinking
• The 4-bit header length in the TCP header is only 4 bits, that is, the value range of the 4-bit header length is, that is, the 0000 ~ 1111maximum length is 15, and because the basic unit of the 4-bit header length is 4 bytes, the 15*4=60byte
• That is, the maximum length of the TCP header is 60 bytes, and the basic length is 20 bytes, that is, the value range of the header is[20 ~ 60]
• After getting the header length, it can be directly separated from the payload

Notice: Different protocol layers have different names for data packets, which are called data segments at the transport layer (segment), datagrams at the network layer (datagram), and data frames at the link layer(frame)

How does TCP decide which protocol in the upper layer to deliver the payload to? that is, how to share

Every network process in the application layer must bind a port number.

• The server process must explicitly bind a port number (the port number is public).
• The client process is dynamically bound to a port number by the system.

The TCP header covers the destination port number, so TCP can extract the destination port number in the header, find the corresponding application layer process, and then hand over the payload to the corresponding application layer process for processing.

How to unpack and divide has been solved, and the encapsulation is reversed

One question, what does the TCP header ask without the size of the payload? ?

• Because TCP is oriented to byte streams, that is, data is transmitted in the form of byte streams. When TCP receives the data, it will be directly handed over to the upper layer (TCP has ensured that the data arrives in an orderly manner, that is, to provide reliability). How to use and analyze the data is the work of your upper layer, and has nothing to do with my TCP.

Orientation to byte streams and reliability will be discussed later

After receiving a message, how to find the process that was bound to a specific port number? What is the relationship between the network protocol stack and the file

• The mapping relationship between the port number and the process PCB is maintained in the Linux kernel in a hash manner, so the corresponding process PCB can be quickly found through the destination port number, and then the corresponding process can be found
• This also confirms why a port number can be bound to a process
• In the socket code, bind specific port is actually to establish the relationship between process PCB and port

If the process is found, how to give the data to the process? ?

• There is a pointer to the table in the process PCB filles_struct, which contains the file descriptor corresponding to the file opened by the upper layer. The file can be found through the file symbol, and then the buffer of the file can be found, and then the data is put into the file buffer, and the upper layer reads the data from the buffer

Note: The above is just a brief overview, the reality is more complicated

How to understand the header?

I have already talked about UDP in the previous article, so I won’t repeat it

Let's start learning the reliability of TCP

2.2 Talk about reliability

Talk about what is unreliable before talking about reliability

Why is there unreliability in network transmission?

• The various hardware of the computer are not isolated, they are all closely related
• The memory and the peripherals are connected by a "line". This line is called the IO bus.
• The direct connection between the memory and the CPU is also connected by a "wire", which is called the system bus
• In fact, memory and peripherals also have their own protocol, because of the existence of the protocol, the peripherals can be controlled through the protocol, that is, each device has its own standard document. So there is something embedded
• There is a question here, why haven't you heard about the reliability of this type of protocol between memory and peripherals?
• Because the transmission distance is short, this kind of problem does not exist at all

But if it is a network, because the distance between the two hosts is too far, the data may be lost when passing through some intermediate devices, and the signal attenuation leads to bit loss, bit flipping, etc., that is, the interference factors in the transmission process increase.

This kind of problem occurs only because the transmission distance becomes farther.

That is to say, the essence of all network problems is that the transmission distance has become farther, that is, the reason for the unreliable network transmission is that the transmission distance has become farther

What are the phenomena of unreliable network transmission?

• Data packet loss, data out-of-sequence, verification error, data duplication, etc.

The following are solutions to these problems

How does the transmission distance become longer, and is there absolute reliability? ?

For example, two people, A and B, are talking at a distance of 500 meters.

A: B Have you eaten? (A cannot guarantee that this sentence has been received by B.)

Only when B responds can A confirm that his words have been received by B,
but B is not sure that his words have been received by A. Only when A responds again can B confirm that his

last sentence has been received by A. But

there is a problem that the latest content cannot be confirmed by the other party because there is no reply.

So, we think:

• Only when the response is received, the historical information can 100%be received by the other party - confirm the response, reliable
• In the communication between the two parties, there must be no response to the latest data—that is, the reliability of the latest information cannot be guaranteed

Therefore, absolute reliability does not exist, but there is relative reliability, that is, as long as a message receives a response, the reliability of the message can be guaranteed

2.3 TCP working mode

There are two working modes of TCP

The first (not the real working mode of TCP)

For each data segment sent, a confirmation response is required, and the next data segment (serial) is sent after receiving the response. This has a relatively big disadvantage, that is, poor performance

The second (TCP real working mode)

Send multiple data segments at a time (parallel), and then respond to the peer, which can greatly improve performance

The second case is the real working mode of TCP, that is, the mainstream, but there will also be the first working mode, and the first case is rare

Whether the client sends to the server or the server sends to the client, both parties need to respond, that is, the status of the TCP client and the server is equal

Here is just a brief introduction, and the explanation will begin below. This is for the convenience of understanding the serial number and confirming the serial number.

2.4 Acknowledgment response (ACK) mechanism

TCP numbers each byte of data sent out, and this number is called a sequence number.

Each ACK has a corresponding confirmation sequence number, which means telling the sender what data I have received and where you will start sending it next time. Each ack is the sequence number of the last data received last time plus 1

Note: ack is also a TCP message, but there is no data, only the header

2.5 16-bit serial number and confirmation serial number

32-bit serial number

When the two parties are communicating on the network, one party is allowed to send multiple message data continuously to the other party. As long as each message sent has a corresponding response, it can be guaranteed that these messages will be received by the other party.

For convenience, let's take a single send single response as an example:

• When multiple packets are sent continuously, since the paths selected by each packet during network transmission may be different, the sequence in which these packets arrive at the peer host may be different from the sequence in which the packets are sent.
• Packet ordering is also a kind of reliability, so one of the functions of the 32-bit serial number in the TCP header is actually used to ensure the ordering of messages.

TCP numbers each byte of data sent out, and this number is called a sequence number. This has been said above

• Assuming that the sender wants to send 2000 bytes of data, each time 1000 bytes are sent, it will be divided into two TCP packets to send. The sender will set a serial number for each TCP message. For the first message, the serial number will be filled with 1, and for the second message, the serial number will be filled with 1001.
• After receiving these two TCP messages, the receiving end will rearrange the order of the messages according to the serial number in the TCP header (this action is performed at the transport layer), so as to be consistent with the order in which the sending end sends the messages, and then put them into the receiving buffer of TCP.

have to be aware of is, The processing of out-of-order packets by the TCP protocol is completed at the transport layer without the intervention of the application layer.

The data received by the receiving end will be processed and assembled according to the order of the serial number to ensure the integrity and correctness of the data. This can solve the out-of-sequence problem during data transmission and ensure that the data is transmitted to the application layer for processing in the order of the sender.

32-bit confirmation number

The 32-bit confirmation sequence number in the TCP header tells the peer what data I have received so far and where your data should start to be sent next time, each with ACKa corresponding confirmation sequence number

According to the above example, assuming that host B has successfully received the message with serial number 1 and the message with serial number 1001 from host A, host B will send two ACK confirmation messages:

• The confirmation sequence number of the first ACK message is 1001, indicating that the byte data of the sequence number has been successfully received 1-1000.
• The confirmation sequence number of the second ACK message is 2001, indicating that the byte data of the sequence number has been successfully received 1001-2000.

ack confirmation response and confirmation sequence number: ensure that the receiver has received all the messages before the ack sequence number

What should I do if the packet is lost during transmission?

Through the sequence number and confirmation sequence number, you can also judge whether a certain message is lost. What to do will be discussed later

Why use two sets of serial number mechanism?

The sender has a set of serial numbers and confirmation serial numbers, and the receiver also has a set of serial numbers and confirmation serial numbers. Why?

Because TCP is full-duplex and equal in status , both parties may want to send messages to each other at the same time:

• In the messages sent by both parties, not only need to fill in the 32-bit sequence number to indicate the sequence number of the data currently sent by themselves.
• It is also necessary to fill in the 32-bit confirmation sequence number to confirm the data sent by the other party last time, and tell the other party which byte sequence number should start sending next time

2.6 16-bit window size

TCP itself has a receive buffer and a send buffer (full duplex):

• The receive buffer is used to temporarily store received data.
• The send buffer is used to temporarily save the data that has not been sent yet.
• Both buffers are implemented inside the TCP transport layer.

This has been discussed in the previous UDP, so I won’t repeat it

window size

• When the sender wants to send data to the receiver, it puts the data into its own send buffer. But the send buffer has a certain size limit.
• If the receiver can process data faster than the sender can send it, there is a problem: the receiver's receive buffer can fill up.
• In this case, when the sender sends data again, the receiving buffer of the receiver has no space, which will lead to data loss and cause a series of problems, such as packet loss and retransmission.

To solve this problem, the TCP header has a field called the window size, which takes up 16 bits. This window size indicates the size of the remaining space in the current receiving buffer of the receiving end, that is, the ability of the receiving end to receive data currently.

The receiving end can tell the sending end how much space is left in its receive buffer through the window size field. The sender can adjust the speed of sending data according to this window size field:

• If the window size field is large, it indicates that the receiving end has a strong receiving ability, and at this time, the sending end can increase the speed of sending data.
• If the window size field is small, it indicates that the receiving capability of the receiving end is weak, and at this time, the sending end can reduce the speed of sending data.
• If the value of the window size field is 0, it means that the receiving buffer of the receiving end is full, and the sending end should stop sending data at this time

Notice: Fill in your own window size when sending data by yourself, and fill in the window size of the other end when sending data by the other end, so that the two parties exchange the receiving capacity of their own buffers

In short, the window size field plays a role in regulating the data transmission speed between the sending end and the receiving end in TCP, ensuring flow control during data transmission.

2.7 Six flag bits

The role of the six flags:

• URG（Urgent）: Indicates whether the urgent pointer field is valid. If this flag is set, the value of the urgent pointer field will be interpreted as an offset indicating where the urgent data ends.
• ACK（Acknowledgment）: Indicates whether the confirmation sequence number field is valid. If this flag is set, the value in the Acknowledgment Sequence Number field will be interpreted as the sequence number of the next data expected to be received.
  - PSH（Push）: Indicates that the receiving end should hand over the data to the application layer as soon as possible without waiting for the buffer to be filled. This flag is usually set when the sender needs to send data to the receiver immediately.
• RST（Reset）: Indicates to reset the connection. When the receiving end receives an unrecognizable sequence number or receives a message that does not conform to the current connection state, it will send a message with the RST flag to reset the connection.
• SYN（Synchronize）: Indicates that a connection is established. When establishing a TCP connection, both the sending end and the receiving end will send a message with the SYN flag bit to shake hands.
• FIN（Finish）: Indicates that the connection is terminated. When the sender has no data to send, or has completed receiving the data after receiving the FIN message, it will send a message with the FIN flag to end the connection

These six flags are all one bit. If it is 1, it means that the flag is set, and if it is 0, it is false.

Before we start explaining, let's make one clear:

TCP packets have types!

• When the TCP server provides services to a TCP client, the TCP server also provides services to many other clients.
• At this time, TCP will receive a large number of TCP messages, such as connection establishment messages, disconnection messages, ack confirmation messages, connection reset messages, etc.
• The server needs to take different actions according to the type of TCP message. For example, if it is a message to establish a connection, the server will enter the state of three-way handshake instead of reading the message.

How to distinguish these TCP packets? ?

Distinguishing TCP packets can be completed by setting six flags

2.7.1 SYN

SYN（Synchronize）: Indicates that a connection is established.

• The SYN in the message is set to 1, indicating that the message is a connection establishment request message. The two parties enter the three-way handshake state
• SYN is set only during the connection establishment phase, and SYN is not set during normal communication.

After the three-way handshake

2.7.2 FIN

FIN（Finish）: Indicates that the connection is terminated.

• The FIN in the message is set to 1, indicating that the message is a disconnection request message. The two sides entered into a state of four waves
• FIN is set only during the disconnection phase, and FIN is not set during normal communication.

Four Waves of Hands followed by a detailed preface

2.7.3 ACK

ACK（Acknowledgment）: Indicates whether the confirmation sequence number field is valid.

• The ACK in the message is set to 1, indicating that the message can confirm the received message.
• Generally, except for the first request message that does not have ACK set, the rest of the messages will basically set ACK, because the sent data itself has a certain ability to confirm the data sent by the other party, so when the two parties are communicating, they can respond to the data sent last time by the other party.

SYN, FIN and ACK are discussed in detail

2.7.4 PSH

PSH（Push）, PSH is set to 1, indicating that the sender tells the receiver that the data should be handed over to the application layer as soon as possible

• When the sender sets the PSH flag, it will set the PSH flag to 1 in the sent TCP segment. After receiving the message segment with the PSH flag bit set to 1, the receiving end will immediately hand over the data in the message segment to the application layer for processing, without waiting for the buffer to be filled or waiting for a certain delay time.

read/recv read data problem

• When using the read/recv function to read data from the buffer, if there is data in the buffer, the read/recv function will immediately read and return the data. If there is no data in the buffer, the read/recv function will block and wait until there is data in the buffer before reading and returning.
• However, both the receive buffer and the send buffer have a concept of a watermark. The water level is a threshold, only when the amount of data in the buffer reaches or exceeds the water level, the read/recv function will read the data and return. This is to avoid frequent read and return operations, thereby improving the efficiency of reading data.
• For example, suppose the watermark of the TCP receive buffer is 100 bytes. Only when the amount of data in the receiving buffer reaches 100 bytes, the read/recv function will read the 100 bytes of data and return. If there is only a little data in the receive buffer, the read/recv function will not read and return immediately, but wait for more data to arrive in the buffer.
• In addition, when the PSH (Push) flag in the message is set to 1, it actually tells the operating system of the other party to deliver the data in the receiving buffer to the upper layer as soon as possible, even if the amount of data has not reached the watermark.

To sum up, when the read/recv function reads data, the amount of data in the buffer needs to reach a certain amount (water mark) to read, and the PSH flag can affect the timing of data delivery.

2.7.5 URG

URG（Urgent）: Indicates whether the urgent pointer field is valid.

16-bit urgent pointer: identifies which part of the data is urgent data.

• When the two parties are communicating on the network, because TCP guarantees that the data arrives in order, because TCP can rearrange the order of these TCP messages through the 32-bit serial number
• Sometimes the sender may need to send some "urgent data", which needs to be processed quickly by the upper layer of the other party. At this time, you need to set URG to 1, indicating that the urgent pointer takes effect.
• 16-bit urgent pointer to find where the urgent data is, i.e. the offset in the payload
• There is only one urgent pointer, so only one offset can be identified at a time, not a contiguous region. This means urgent data can only be one byte.
• When the receiving end receives a message with the urgent pointer in effect, it will immediately process the urgent data and will not wait for the arrival of subsequent data.

In fact, in the modern TCP protocol, the use of urgent pointers is not common, and it is not used in 99% of the cases.

The recv/send function has parameters for setting URG

The fourth parameter flags of the recv function has an option called MSG_OOB that can be set. OOB is the abbreviation of out-of-band data. Out-of-band data is some important data. Therefore, if the upper layer wants to read urgent data, it can use the recv function to read and set the MSG_OOB option. The corresponding fourth parameter flags of the send function also provides an option called MSG_OOB. If the upper layer wants to send urgent data, it can use the send function Write, and set the MSG_OOB option

.

2.7.6 RST

RST（Reset）: Indicates to reset the connection.

• When the receiving end receives an unrecognizable sequence number or receives a message that does not conform to the current connection state, it will send a message with the RST flag to reset the connection.
• For example, the server suddenly lost power, and the client and the server did not wave four times. At this time, the client still thinks that a connection is established with the server. After the server restarts, the server will receive the TCP packet sent by the client. The server is very confused, you have not established a connection with me, how do you communicate with me, then the server will set RST to 1, send the message to the client, and let the client reconnect with the server.
• and vice versa

When visiting some websites, the following words will appear, which means that the server has sent

the above is the basic header of TCP

2.8 TCP timeout retransmission mechanism

Before talking about this topic, let me talk about a question:

How to treat TCP's receive buffer?

• This receiving buffer is a byte stream, that is, this buffer can be regarded as one char buffer[N], N is the size of the buffer, that is, the receiving buffer can be regarded as an array of fixed size, that is, as long as the data (each byte) in the data has a serial number naturally
• This array serial number provides great convenience for 32-bit serial number and confirmation serial number (the data serial number is very important for 32-bit serial number and confirmation serial number)
• The 32-bit sequence number is used to identify the position of the start byte of each TCP segment in the entire data stream, so that it can be correctly placed in the receiving buffer (placed according to the array sequence number)
• This also means that the receive buffer has no concept of packets or fragments for received data, but instead stores all bytes in the buffer in the order they arrived
• The size of this buffer is 64KB, determined by the 16-bit window size. If the sent data is larger than 64KB, the upper layer needs to process it by itself, divide the data into multiple parts and send each data to be smaller than 64KB

TCP timeout retransmission mechanism

TCP's timeout retransmission mechanism is designed to ensure reliable data transmission . When the sender sends a data segment, a timer is started. If the sender receives the corresponding acknowledgment before the timer expires, the timer will be canceled. If the sender has not received an acknowledgment after the timer expires, the data segment will be resent.

The timeout retransmission mechanism of TCP can be divided into the following steps:

1. Send data segment: The sender sends the data segment to the receiver and starts a timer.
2. Waiting for acknowledgment: The sender waits for the receiver to send an acknowledgment. If the sender receives an acknowledgment before the timer expires, the timer will be canceled.
3. Timer expires: If the sender has not received an acknowledgment after the timer expires, it will consider that the data segment is lost and needs to be retransmitted.
4. Retransmit data segment: The sender resends the lost data segment and starts a new timer.
5. Waiting for acknowledgment: The sender waits for the receiver to send an acknowledgment. If the sender receives an acknowledgment before the timer expires, the timer will be canceled.

This process will be repeated until the sender receives an acknowledgment or reaches the maximum number of retransmissions.

Two cases of packet loss

• After host A sends data to B, the data may not reach host B due to network congestion and other reasons;
• If host A does not receive an acknowledgment from B within a specific time interval, it will resend

However, host A did not receive the acknowledgment from B, it may also be because the ACK was lost

• When packet loss occurs, the sender cannot tell whether the data packet sent is lost or the response packet sent by the other party is lost, because in both cases the sender cannot receive the response message sent by the other party, and the sender can only perform timeout retransmission at this time.
• It should be noted that when the data in the sending buffer is sent out, the data will be temporarily stored in the buffer for a period of time to avoid the need for timeout retransmission. This part of the data in the sending buffer can not be overwritten until the response message of the data is received. (This post-prologue talks about sliding windows in detail)

If the other party's response message is lost and the sender retransmits over time, the receiver will receive a duplicate message data again.

How to deal with this duplicate message?

• But there is no need to worry at this time. The receiver can judge whether the message has been received according to the 32-bit sequence number in the header, so as to achieve the purpose of message deduplication. Deduplication means directly discarding duplicate messages.

How to determine the data timeout time? ? That is, how long is the timer? ?

• Ideally, find a minimum time to ensure that "the confirmation response must be returned within this time"
• But the length of this time varies with the network environment
• If the timeout is set too long, it will affect the overall retransmission efficiency
• If the timeout is set too short, repeated packets may be sent frequently

In order to ensure high-performance communication in any environment, TCP will dynamically calculate the maximum timeout period

• In Linux ( BSD Unixand Windowsthe same is true), the timeout is controlled 500msas a unit, and the timeout time for each timeout retransmission is 500msan integer multiple of
• If there is still no response after retransmission, wait 2*500msand retransmit
• If there is still no response, wait 4*500msfor retransmission, and so on, increasing exponentially
• When a certain number of retransmissions has been accumulated, TCP considers that the network or the peer host is abnormal, and forcibly closes the connection

The content of the article is too much, see you in the next one (the above is already 10,000 words, there are so many TCP contents...)
--------------------- END ----------------------

「 作者 」 枫叶先生
「 更新 」 2023.7.21
「 声明 」 余之才疏学浅，故所撰文疏漏难免，
          或有谬误或不准确之处，敬请读者批评指正。