Article directory
foreword
Transport layer: TCP/UDP
The transport layer is responsible for data transmission between two hosts, such as Transmission Control Protocol (TCP), which can ensure that data is reliably sent from the source host to the target host.
The previous articles all explained the content of the application layer. Next, we enter the content of the transport layer. The transport layer is responsible for the transmission of data from the sender to the receiver.
There are two most typical protocols in the transport layer, one is the UDP protocol and the other is the TCP protocol. Before explaining the UDP protocol, let's re-understand the port number:
The port number identifies different application programs for communication on a host. In network communication, a host is identified by an IP value, and different processes on a host are identified by a port value.
In the TCP/IP protocol, a communication is identified by a five-tuple such as source IP, source port number, destination IP, destination port number, and protocol number, as shown in the following figure:
The simple understanding is: the source IP is the IP of your own host, and the destination IP is the IP of the server you want to connect to. The source port number represents the port number of a process opened by the host, which is generally assigned by the operating system, and the destination port number is the port number of a process to be connected to the server. The 6 below TCP in the figure above represents the protocol number.
Port range division
After knowing the well-known port numbers, we need to avoid using these well-known port numbers when we write a program to use the port numbers.
There are two questions below:
1. Can a process bind multiple port numbers?
Can! We must remember that to ensure the uniqueness of the port number to the process, when there are multiple port numbers pointing to a process, it will not affect the uniqueness of the port number to the process.
As shown in the figure above, when a process is bound to multiple port numbers, we find that each different port number will point to a process, so it does not affect the uniqueness of the port number to the process.
2. Whether a port number can be bound by multiple processes
Can't! Of course there are exceptions, but 99% of the time it won't work.
It can be seen that if a port number is bound by multiple processes, it will be messed up and it is impossible to know which process it is.
Next, let's learn two more instructions. In fact, these two instructions were used when we learned sockets before:
netstat:
1. Udp protocol
The UDP protocol end format is as follows:
We said before that when transmitting data from the application layer down, headers need to be added, and when data is transmitted from the data link layer to the application layer, headers need to be removed. The process of removing headers is called unpacking, so how does UDP unpack? What about the package? In fact, the method of UDP unpacking is very simple, only need to fix the length of the header, so when we unpack, we only need to take away the fixed header length, isn’t the payload left? Then how to divide the unpacking? What about? In fact, it is also very simple, because the header of our UDP protocol has a 16-bit destination port number, so the data is directly handed over to the upper-layer specific protocol to complete the process of sharing.
In fact, the so-called header is a structured data. For example, UDP stores the source port, destination port, and UDP length and checksum, as shown in the following figure:
And the principle of adding headers is very simple:
When we call sendto to send data, we first open a part of the space inside the operating system, then calculate the size of the header structure, use a pointer to offset a space just the size of the header structure from the beginning of the space, and then we fill the structure The information in the header is placed in the header, and the sent data can be copied directly in the latter position.
UDP buffer
Our sending and receiving data in the network is not actually sent directly as we thought. When we implemented UDP and TCP before, we found that each call to the sendto interface requires a buffer. This is our user’s buffer. First, the operating system will The data in the user buffer is copied to its own sending buffer, and then sent to the receiving buffer of the operating system of another host through the network. We also need a buffer when we call the receiving function, and the operating system will receive the buffer In this way, half-duplex is realized, and we can not only send, in fact, we can also receive the data sent by the other party in the process of sending, that is to say, our client While sending data to the server, the server can also send data directly to the client, so it is full-duplex.
The most essential difference between our TCP and UDP is that UDP in the above figure can only send as much data as it receives, while TCP can control how much it receives or sends, so TCP is also called Transmission Control Protocol.
Precautions for using UDP:
Two, TCP protocol
We can see that the design of TCP is very complicated compared with UDP. Let's explain the length of the 4-digit header first:
First of all, it can be seen from the figure that the length of the TCP protocol header is 0~31, that is, 32 bits, so how does the TCP protocol unpack? (Separate the header from the payload) First of all, let’s not talk about the options and data lines. There are a total of 5 lines and each line has 32 bits, so the total is 20 bytes, because the tcp protocol has a standard length, that is We just calculated 20 bytes, so to unpack, we first read 20 bytes, then convert the header into a structured data, and immediately extract the 4-digit header length in the standard header (the 4-digit header length is the TCP header Note: the standard tcp length is 20 bytes, but there may be options. The total length of the tcp message = 4-digit header length * 4 bytes, so the range of 4-digit header length is 0*4~15 *4 (bit 1111 (4 bits)), and because our standard length is 20 bytes, the range is [20,60].), once we have the 4-bit header length, we can calculate the remaining The length of the next message, for example, the length of the 4-digit header is x: x*4 (total length)-20==remaining message length, if it is 0, it means that there is no option, and finally only need to process the tcp header, and then The payload is left, so that the package is decomposed. How to divide it is the same as udp, because the message has a destination port number, so it can be delivered directly to the upper layer.
After understanding the 4-bit header length and how TCP unpacks and divides, let's understand the TCP header again:
In fact, it is the same as the UDP header, which is a structure or bit segment, but there are more things in the TCP header:
Let's learn about the reliability of TCP:
First of all, to understand reliability, we must first know why there are unreliable problems?
The main reason for unreliability in the network is that the distance between the hosts is too far. Why is it not unreliable when the same CPU and disk or other hardware IO? If the hard disk, memory, and CPU in a host are all If they are far apart and connected by wires, then it is the same as our network, so the main problem is the length of the distance.
So what are the unreliable scenarios?
There are scenarios such as packet loss, out-of-order, verification errors, and duplication.
So how is tcp reliability guaranteed?
First of all, there is no absolute reliability. Secondly, we believe that only when the response is received, can I 100% confirm the receipt of historical messages. Only when the response is confirmed can it be considered reliable. Why do we say that there is no absolute reliability? There must be the latest data in the communication between the two parties, and the latest data may not be answered.
Let's understand the two concepts of sequence number and confirmation sequence number in the header:
As shown in the figure above, when the real client communicates with the server, most clients will send many data segments to the server at one time, and the server also needs to respond to the data segments sent by the client. In actual network communication, We cannot guarantee that the order in which the data arrives on the opposite side must be the same as the order in which they were sent. This is like when we buy express delivery, we buy the goods in order, but when the express delivery arrives at us, it does not necessarily arrive in the order in which we purchased it. Moreover, our client needs to receive the confirmation from the server. How can the client know the corresponding relationship between the request sent to the server at that time and the response sent by the server now? In layman's terms, the client does not know which message the server confirmed. From the above two problems, it is doomed that the tcp data segment needs to have a way to identify the data segment itself, so only the 32-bit serial number exists in our TCP header.
As shown in the figure above, when we look at each data segment, we should regard the data segment as a header + payload, not simply as a request and response. The confirmation sequence number in the header is to tell the other party that the content of the other party has been received. Our confirmation serial number must be the serial number sent by the other party + 1, because this can ensure that we have received all the messages before the serial number + 1, and let the other party start sending from the serial number + 1 next time, such as sending data at number 10 in the above figure segment, our response should be the 11th confirmation sequence number, and the next client send will start from the 11th. In the picture above, once there is a packet loss problem during our communication process, for example, the data segment sent on the 12th is lost, but the server has already received the data segment on the 13th. At this time, the confirmation number of the server can only be filled with 12, because although we have received the 13th Send the serial number, but because the data segment before the 13th is not continuous, the client must resend it to us from the 12th data segment. Some smart friends may have thought of it. Isn't it enough for our header to use a serial number? When the client sends it, it is the sending serial number, and when the server responds, it is the confirmation serial number. The idea is not but, but it ignores the fact that TCP is full-duplex, because our TCP is full-duplex, so it is very possible to send and receive at the same time, so we must have two serial numbers.
Let's take a look at the 16-bit window size in the header:
We all know that both the client and the server have their own receiving buffer and sending buffer, but how do we know whether the receiving buffer of the other party is full? If it is full, the data we sent will be discarded Is it, so we must get the remaining size of the receiving buffer of the other party. And because TCP is full-duplex, so when we send a message to the other party, the other party may also send us a message, so the other party also needs to know the size of our receiving buffer, so there is a 16-bit window size, and a 16-bit window size. The size is the remaining space of the receiving buffer. Note: Our message must fill in our own 16-bit window size. We put our own in the message and the other party will know how much space we have left after receiving it.
When we learn TCP congestion control later, we will find that when network congestion occurs, the window size at this time is the smaller value of the congestion window size and the receiving buffer window size of the other party.
Let's take a look at the data segment types:
SYN synchronization flag ( request to establish a connection ; we call the one carrying the SYN flag a synchronization segment ): The SYN flag needs to be set to 1 during the tcp three-way handshake, and SYN will only be set when a connection is established.
FIN disconnection flag ( inform the other party that this end is going to be closed , we call the end segment carrying the FIN flag ): When the client exits due to some reasons in the tcp communication, the server will set the FIN flag position at this time If it is 1, it means disconnection. Of course, once FIN is 1, SYN must be set to 0.
ACK confirmation message flag ( whether the confirmation number is valid ): When one end confirms receipt of the message from the other end, it sets the ACK flag to 1 to indicate that the data segment of the other end has been received.
PSH flag ( : Prompt the receiving end application to read the data from the TCP buffer immediately): psh is actually the abbreviation of push. At this time, the sender sets the PSH flag to 1 to urge the receiver to update the receiving buffer size.
URG flag ( whether the urgent pointer is valid ): If there is data in our message that needs to be read as soon as possible, then the URG flag will be set to 1. (The urgent pointer can find which piece of data needs to be read as soon as possible)
RST reset flag ( the other side requests to re-establish the connection ; we call the one carrying the RST flag a reset segment ): the abbreviation of reset. There is a common scenario that when our server and client communicate normally, the server The connection is suddenly disconnected. At this time, the client still thinks that it is connected to the server, but the server thinks that the connection is disconnected, and then the client sends data directly to the server (normally, the connection is established first, that is, after three handshakes, the Send data), at this time, the server will say that you have not established a connection with me, why did you send the data directly, and then send a response to the client to let the client reconnect, at this time, the response message sent by the server to the client Set the RST flag position to 1, indicating that the client should close the connection and re-establish a connection with the server. As shown below:
Let's take a look at the 16-bit urgent pointer in the header:
When there is data that needs to be read as soon as possible in a message, we can find this piece of data through the urgent pointer. The principle of finding this piece of data by the urgent pointer is also very simple, that is, it is realized through the offset of the payload.
2. Timeout retransmission mechanism:
When host A sends data to host B, but has not received a response from host B for a long time, host A will resend the data to host B at this time. However, host A may also be due to the loss of ACK in the response sent by host B. As a result, there is no response due to misidentification. At this time, host B will receive a lot of duplicate data, so the TCP protocol needs to be able to identify those packets as duplicate packets , and discard the duplicates .
3. Re-recognize the TCP three-way handshake:
When our client waved for the first time, it will set the SYN flag in the sent header to 1 to indicate that it wants to establish a connection, and it is in the SYN_SEND stage. The server will send the header sent by the client for the first time to the client. To respond, the SYN flag and ACK confirmation response flag will be set to 1 in the sent header, indicating that the client’s request has been received and the connection is agreed to be established, and the client is in the SYN_RCVD (received) state, and the client receives the server. After confirming the information, the client establishes a successful connection, and then sends a header to the server confirming receipt of the server request, and the server establishes a successful connection after receiving the confirmation information from the client. Note: The three-way handshake does not mean that the connection must be established successfully. The three-way handshake is only the prerequisite for the two parties to establish a connection.
We can find that in the process of three-way handshake, the first two handshakes will be answered by the other party, so whether the first two handshakes fail or succeed, you can know the status of the other party. If the ACK confirmation is not received for one of these reasons, the client will think that the establishment is successful, but the server has not. In this case, the timeout retransmission we mentioned above solves this problem. The client does not receive the A successful response will be resent to the server for reconnection. Therefore: the three-way handshake may not be successful, and the most worrying thing is the loss of the last ACK, but there are supporting solutions. Moreover, the connection needs to be managed. It is managed by the operating system. It is first described and then organized. Maintaining a connection is costly in time and space.
So why do we have to shake hands three times? Can one handshake work?
This is absolutely not acceptable. Suppose a client is multi-threaded and initiates connection requests to the server multiple times. As long as the client sends a SYN message to the server, the server will establish the connection. Once the client machine frequently initiates a connection request to the server SYN message, then the server will consume a lot of resources to maintain the established connection, thus suffering from SYN flood.
So is it okay to shake hands twice?
If it is a two-way handshake, when the client sends a SYN message to the server, the server establishes a connection first after receiving it. Isn’t this the same as a handshake? The client sends a SYN message and the server establishes a connection after receiving it. , and then send a SYN+ACK message to the client. At this time, not only the client is unwilling to establish a connection or does not connect in other cases, but the server is still vulnerable to SYN floods. Can the client establish a connection by itself the first time? Can't! The connection is a matter for both parties. When sending a SYN message to tell the other party to establish a connection, the other party needs to agree and reply to you before the connection can be established, otherwise it will consume resources.
The most important reason why the three-way handshake works is to verify that the full-duplex communication channel is unobstructed with the least cost, and the three-way handshake can ensure that your client establishes the connection before my server establishes a connection, so that we can effectively To prevent a single machine from attacking the server. It's like when a friend tells you that there is a non-spicy chili for you to eat, you will let him eat it to verify yourself before eating it.
4. Re-acquaint TCP with four waved hands:
First of all, when the client does not want to connect, it sends a FIN message to the server. At this time, it is in the first FIN_WAIT1 (waiting to cancel the connection) stage, and then the server responds to the client with an ACK (acknowledgement) after receiving the message from the client. At this time, the server itself is in the CLOSE_WAIT (waiting to close the connection) state, and the client is in the FIN_WAIT2 (waiting for the second time to cancel the connection) stage (the reason for not directly canceling here is that the client has not yet reported that the server also cancels the same) arts). Then the server actively sends a FIN message to the client. At this time, the server itself is in the LAST_ACK (final confirmation) stage, and the client is in the TIME_WAIT () stage. After receiving the request from the server to cancel the connection, the client will serve The end sends an ACK (acknowledgment of receipt) message, and once the server receives the message confirmed by the client, the server will directly close the file descriptor. Note: When disconnecting, either party may disconnect, and the status of whoever sends the FIN message first is FIN_WAIT. That is to say, the party who actively disconnects will be in the TIME_WAIT state after the 4 waved state has been completed, and will maintain the TIME_WAIT state for a period of time. The party that is passively disconnected will enter the CLOSE_WAIT state after waving twice.
Note: If there is a large number of close_wait on our server, it first means that the server has a bug and has not closed the file descriptor. The second server is under pressure and may keep pushing messages to the client, making it too late to close.
Why does the party that actively disconnects in the four waves finally maintain the TIME_WAIT state for a period of time? So how long will it last?
Because the last ACK confirmation message was sent by the party that actively disconnected, if there is no TIME_WAIT time, once the last ACK confirmation message is lost or the other party does not receive it for some reason, and the party that sent the last ACK at this time has already sent the file If the descriptor is closed, the party receiving the ACK needs to resend the FIN segment to the other party because it has not received the ACK, but the other party has already closed the retransmission, so it is meaningless, so this is a bug, and the state of TIME_WAIT is to relieve this Bug, even if one party has sent the last ACK confirmation, it will still wait for a period of time before closing. If the other party has a problem or does not receive the last ACK during this period, then just resend the FIN segment to the other party directly. This maintenance time is generally 2*MSL, and MSL (Maximum Transmission Unit) represents the time from left to right or from right to left, as shown in the figure below:
Each of the two boxes in the figure above is an MSL. In addition to the first reason above, there is a second reason that when the two parties disconnect, there are lingering messages in the network to ensure that the lingering messages are dissipated.
So: 1. Ensure that the last ACK is received by the other party as much as possible.
2. When the two parties are disconnected, there are lingering messages in the network to ensure that the lingering messages are dissipated.
Let's verify the state of the four waved hands:
First, modify the code of the last http, remove unused functions, write an infinite loop in the hander method, and do not actively close the file descriptor:
Then we run it: use telnet to connect to the server, you can see that the port number of telnet is 33366, and the pid is 32759 is a process created by our server (because our server is a multi-process version):
When we use ctrl+] to enter the quit command to disconnect telnet, because telnet actively disconnected, we found that telnet is in the FIN_WAIT2 stage (the FIN_WAIT1 stage did not capture the picture because the time was too fast, and FIN_WAIT2 can already prove that the first exit is in the FIN_WAIT phase), and the process of server 32759 is in the CLOSE_WAIT phase.
Since our server was slowed down, we did not see the TIME_WAIT stage of the party that disconnected first, so let's do it again:
The above picture shows the connection establishment phase
The picture above shows the disconnection stage of telnet, and the server is not disconnected at this time
As can be seen from the figure above, when we quickly shut down the server, the telnet client is immediately in the TIME_WAIT stage, and the server has all exited at this time. After verification, we found that what needs to be remembered in the four-way wave is that the party that is actively disconnected enters the TIME_WAIT phase after four waved hands, and the passively disconnected party enters the CLOSE_WAIT phase after two waved hands.
Summarize
There are many principles of the TCP server, and the next article will continue to be the content of the TCP server.