Top-down understanding of the network (4) - TCP articles
The topic of this series of articles is understanding the web from the top down, and here top-down, as long as it refers to HTTP-based web services. As long as we understand this process from top to bottom, for other applications, only the protocols are different, and the principles are similar. Through the introduction of the previous blogs in this series, we learned that when a domain name is entered in the browser or the app accesses the back-end service interface through a domain name, the domain name will be converted into an IP address. In fact, only the IP address is not enough. In theory, it is necessary to A port number used to identify the corresponding application on the service host. Just in practical applications, the default port number of the HTTP protocol is 80, and the default port number of the HTTPS protocol is 443.
HTTP provides the data definition structure of the application layer (HTTPS adds security protection), and the protocol of the application layer regulates the application interaction format between the server and the client in different devices and network environments. Now, we need to care about how these application data are transmitted between the two ends. I don't know if you remember, we mentioned the layering of the network before. The following figure describes the network layer of each protocol in the TCP/IP protocol suite:
It can be seen that when the application layer assembles the application data, it does not care about the transmission of the data, and the transport layer is responsible for transmitting the data to the target host. HTTP is an application layer protocol based on TCP. Regarding UDP, it is still very different from TCP. We will not discuss it in this blog.
1. Let’s talk about some theory first
Most developers or computer science students are familiar with TCP, but they may only be at the stage where they are not familiar with TCP, its working principle and protocol content are not very clear. In the layered network model, each layer of the protocol will add header information on the basis of the previous layer of data. Learning the protocol is actually learning the meaning and usage of these header information.
I first learned about TCP in the network-related courses at school. Later, when I re-learn computer-related knowledge at work, I will encounter TCP-related theoretical content every time. When almost all teachers introduce TCP, they will first throw the following definitions:
TCP is a connection-oriented, reliable byte stream-based transport layer communication protocol.
There are some keywords in this definition: connection-oriented , reliable , byte-stream based . What do connection-oriented and byte-stream based mean? How is reliability guaranteed? This is the core of this article to discuss.
First of all, let's make some brief introduction to TCP theoretically.
1. About connection-oriented
Connection-oriented mainly means that two TCP applications need to establish a TCP connection before exchanging data with each other. Some descriptions are like establishing a channel on the network before the client and server communicate, and then the communication are based on this channel. If the channel can be established, it can also be closed. When the two communicating devices no longer need to exchange data, the TCP channel can be closed. Later we will describe how the connection is established and how it is disconnected.
2. About reliability
When it comes to reliability, it means that the upper-layer business does not need to care whether the data is lost during the transmission process, whether the other party is sure that it has been completely received, and so on. For example, when we send HTTP requests and receive receipt data, we don't care about the arrival and integrity of the data at all, we just need to wait for the receipt. This is because the reliability of transmission is guaranteed at the TCP layer.
When TCP sends application data, it divides the data to be sent into suitable data blocks and sends them in blocks.
When TCP sends a piece of data, it will wait for the destination's confirmation to receive the message. If the message is not received within a certain period of time, it will enter a timeout or retry logic. TCP ensures the reliable arrival of the data by confirming the message. sex.
TCP will maintain an end-to-end data checksum to detect whether there is an error in the transmission process. If an error is found, the TCP receiver will discard the message, not send an acknowledgment message, and wait for the other party's timeout or Resend logic.
TCP transmits data through the IP protocol of the network layer, and IP datagrams may be out of order. Therefore, TCP needs to rearrange the received data and return the received data to the application layer in the correct order.
Similarly, IP datagrams may also be duplicated, and TCP is also responsible for deduplicating duplicate datagrams.
Finally, TCP also provides flow control, which increases transmission reliability.
3. About byte stream based
When TCP transmits data, it does not interpret the data at the application layer, and TCP does not insert any identifiers into the data byte stream. That is to say, what format of application data is specifically transmitted by TCP is not parsed at the TCP layer, and the byte stream data sent by the sender will also be received in exactly the same way by the receiver, and all interpretations and understandings are applied. layer.
2. Macroscopically look at the TCP communication process
As mentioned earlier, TCP first needs to establish a connection before transmitting application data. The way TCP establishes a connection is through 3 communications, which is called the 3-way handshake of TCP.
In the TCP protocol, there is a Flags field, this field will run through the whole process of our understanding of the TCP protocol from beginning to end. This field will indicate the type of the current TCP packet, there are 6 types as follows:
SYN: A connection is established.
FIN: Disconnect.
ACK: acknowledgment response.
PSH: Data transfer.
RST: Connection reset.
URG: Urgent pointer.
This does not mean that each TCP message can only correspond to a unique type. The Flags field occupies 6 bits of data, each bit corresponds to a state, and the message state can be aggregated. For example, a TCP message is marked as ACK Marked by PSH. Aggregation can reduce the number of TCP packets and improve transmission efficiency. The complete TCP packet format is shown in the following figure:
We will not list the meaning of the complete message for the time being. Just look at the Flags part. You can see that it occupies 6 bits. The first bit is the URG bit, and the last bit is the FIN bit.
1. Connection establishment process
TCP communication is carried out between two ends of the network. To establish a connection, one end needs to initiate a connection first. In an HTTP data request, the client first initiates the connection.
Step 1: The client initiates a SYN-type TCP message, which specifies the port number of the target server and other data.
Step 2: After receiving the connection establishment message from the client, the server replies with a message containing ACK and SYN, indicating that it has received the connection establishment request from the client, and initiates the connection establishment request from the server.
Step 3: After the client receives the response message from the server, it returns an ACK-type message to indicate receipt, and the connection is established.
2. Application data sending process
After the connection is established, the TCP communication process becomes relatively simple. When one party sends data, it will send out the type of PSH message, and the other party needs to reply to the corresponding ACK message after receiving it, and so on and so forth until the data exchange is completed.
3. Disconnection process
Similar to the establishment of a connection, the disconnection also requires an ACK for confirmation. Take an HTTP request as an example:
Step 1: When the server finishes sending data, it will first initiate a FIN type message to disconnect.
Step 2: After the client receives the FIN message from the server, it replies with ACK.
Step 3: The client sends a FIN message to disconnect the client.
Step 4: After the server receives the FIN message from the client, it replies with ACK.
Therefore, the process of TCP disconnection is also figuratively called the 4th wave.
3. In-depth understanding of the workflow of TCP
Now, although we have a general concept of the communication process of TCP on a macro level, it is still too superficial, and we have not covered many core points. Such as timing, reliability, the meaning of TCP header fields, etc. This section also discusses this part in detail.
1. Detailed explanation of TCP message header
Go back to the TCP message diagram above. Let's introduce it in detail below.
Source Port: The source port, occupying 16 bits (two bytes), this field is well understood, that is, the port that sent this message.
Destination Port: Destination port, occupying 16 bits (two bytes), that is, the port to receive this message.
Squence Number: sequence number, occupying 32 bits. TCP is based on byte stream transmission. This sequence number is used to identify the number of the first data byte in the current message. Use this sequence number to count the bytes sent. throughout the communication process. It will be described in detail later.
Acknowledgment Number: Ack sequence number, occupying 32 bits, indicating the number of the next data byte that the party sending the ACK expects to receive.
Data Offset: The data offset, occupying 4 bits, can also be understood as the length of the TCP header, which is used to mark the data configuration. It indicates how many 4 bytes the TCP header occupies. Since the maximum number of 4 bits is 15, the maximum length of a TCP header is 4*15=60 bytes.
Reserved: Reserved field with a length of 6 bits.
Flags: Type field, occupying 6 bits. From low to high, it means FIN, SYN, RST, PSH, ACK, URG.
Window: occupies 16 bits, indicating the size of the sliding window, which is used to tell the buffer size of the sender and receiver. Flow control is reached, with a maximum value of 65535.
Checksum: Checksum, occupying 16 bits, used to check whether there is an error during the transmission of the TCP header information.
Urgent Pointer: Urgent pointer, the type is valid for URG messages, indicating the location of the first urgent data byte.
Options-Padding: Additional options, variable length, when less than a multiple of 32 bits, use 0 padding.
Data: Variable length, upper layer data to be transmitted, can be empty.
2. How is the timing guaranteed?
Guaranteeing timing is an important goal of TCP reliability. The guarantee of timing is mainly guaranteed by the SN (Sequence Number) and AN (Ack Sequence Number) in the TCP header.
Let's first look at SN. SN is a relative concept. When a TCP connection is to be established, the first SYN message initiated by the client will be assigned an SN sequence number, which is the initialization sequence number. After this TCP communication , we will use this initialization sequence number as the standard to calculate the relative SN. It should be noted that the SYN message will also occupy an SN sequence number, and the SN sequence number will be incremented by 1 the next time data is sent.
Let's look at AN again. AN indicates the SN number of the next data I expect to receive. After receiving the data from the sender, it will return an Ack-type message and set AN to the SN number configured by the sender plus 1. .
Through SN and AN, the simplified communication process is as follows: So:
This picture seems to have some twists and turns. After analyzing it a few times, you will have a relatively thorough understanding of the principle of TCP.
3. Some other technical means of reliability
timeout and retries
If everything goes well, the TCP connection can be completed in only 3 steps. But in fact, the network environment in reality is much more complicated, and establishing a connection is not always smooth.
In one case, the port that the client wants to connect to is not listening on the server. At this time, the TCP service of the server host will directly reply with an RST type message, indicating that there is an error in the connection. At this time, the initiator should directly close the connection.
Another situation is that the server host is in an abnormal state, and there may be a network problem. At this time, the initiator cannot wait for any receipt from the server. At this time, it will enter the TCP connection establishment timeout logic, and the first TCP connection establishment retry will be Fires at a timeout of about 6 seconds, the second retry fires after an interval of about 24 seconds, and the third retry fires after an interval of about 76 seconds. The specific timeout retry algorithm we follow will not be expanded here.
RST type reset message
There are many scenarios that may trigger the RST reset message. One is that when we access an invalid port, the server will return an RST message.
Another scenario is that if the connection has been closed, sending data through this connection again will receive a reset connection from RST. This scenario is very common. For example, one end has been restarted for some reason, and the other end does not know that an exception has occurred at the other end. When sending data over the old connection, it will receive an RST packet.
4. Validate the theory from practice
Having said so much theory, we have introduced a lot in the theoretical part, which is enough to understand the communication process of TCP. Below we can verify it through practice.
First, two simple Socket server and client programs can be written locally. Regarding the sample programs, we have already introduced them in the previous HTTP article, and you can directly use the previous sample programs. Use Wireshark to capture the local TCP packets. The complete TCP communication packets between the client and the server are shown in the following figure:
The business logic written by our program is as follows:
1. The client port number is 52079.
2. The server port number is 9001.
3. The client initiates a TCP connection first.
4. The client sends "TCP Customer" data to the server.
5. After the server receives the data sent by the client, it sends the "Hello World" data to the client, and then initiates to close the connection.
6. After the client receives the data returned by the server, it closes the connection.
In this communication process, 12 TCP packets are generated, and we can analyze them one by one.
The first message data is as follows:
It can be seen that this packet is a SYN connection establishment packet initiated by the client, with SN of 3069091127 and AN of 0. Except for some options, no business data is included.
The second message data is as follows:
This packet is an Ack packet initiated by the server, and is also a SYN connection establishment packet of the server. The AN of this packet is 3069091128, and the SN is 3416281738.
The third message data is as follows:
This message is sent by the client, the type is ACK, the SN is 3069091128, and the AN is 3416281739. It can be seen that the SN has been increased, which corresponds to the AN of the second packet.
The fourth message data is as follows:
The type of this message is ACK, which is a special message. You can see that it is sent by the server to the client. The value of the Window field is 6379. The purpose of this message is to specify the cache size. The AN of this packet is 3069091128, and the SN is 3416281739. The SN corresponds to the AN of the third packet.
The data of the fifth message is as follows:
This message is sent by the client, and the type is PSH, that is, it is a data transmission message. You can also see that its content carries 12 bytes of business data, and these 12 bytes of data are "TCP Customer ". SN is 3069091128, AN is 3416281739.
The sixth message is:
This message is an ACK message sent by the server to confirm the 12-byte data sent by the client. The AN is 3069091140 and the SN is 3416281739.
The seventh message is as follows:
It can be seen that the seventh message is the data message that the server actively sends to the client, and the length of the data part is 13 bytes, that is, "HelloWorld!\r\n". Its AN is 3069091140 and its SN is 3416281739.
The eighth message is as follows:
This message is the ACK message of the client's receipt, the SN is 3069091140, and the AN is 3416281752.
The 9th to 12th packets are 4 packets of TCP disconnection. The logic of SN and AN is similar to that of connection establishment, so it will not be analyzed here.
5. Ending
You may have found that, except for the initial connection message, every TCP message has an ACK type, because the ACK bit is set to 1 and there is no additional data and performance consumption. In fact, this article only introduces the TCP protocol from the understanding level. If you want to complete the TCP, you need a book thickness. I believe that if you did not understand the network technology used every day before, and you have read this from the first article of this series of blogs, then in terms of application, you must have a deeper understanding. Later we will continue down and discuss the protocols of the network layer.
Focus on technology, love life, exchange technology, and be friends.
——Hunshao QQ: 316045346