One article takes you through the computer network (Part 1)

The number of words in the article is about 21,000 words, and it takes about 70 minutes to read. It is recommended to bookmark and read slowly! ! !

1. TCP/IP protocol

1. What are the layers of the TCP/IP network model, and what are their functions?

   application layer

   The top layer, which we can directly access, is the Application Layer . The application software used by our computers or mobile phones is implemented in the application layer.

   Therefore, the application layer only needs to focus on providing application functions for users, such as HTTP, FTP, Telnet, DNS, SMTP, etc.

   The application layer does not need to care about how data is transmitted, and the application layer works in the user state of the operating system, while the transport layer and below work in the kernel state.

   transport layer

   The data packets of the application layer will be passed to the transport layer, and the transport layer ( Transport Layer ) provides network support for the application layer.

   There are two transport protocols in the transport layer, namely TCP and UDP.

   The full name of TCP is Transmission Control Protocol ( Transmission Control Protocol ), and most applications use the TCP transport layer protocol, such as the HTTP application layer protocol. Compared with UDP, TCP has many more features, such as flow control, timeout retransmission, congestion control, etc. These are all to ensure that data packets can be reliably transmitted to the other party.

   UDP is relatively simple, so simple that it is only responsible for sending data packets, and does not guarantee whether the data packets can reach the other party, but it has relatively better real-time performance and high transmission efficiency. Of course, UDP can also achieve reliable transmission, just implement the characteristics of TCP on the application layer, but it is not a simple matter to realize a commercial reliable UDP transmission protocol.

   The data that the application needs to transmit may be very large, and it is difficult to control if it is directly transmitted. Therefore, when the data packet size of the transport layer exceeds MSS (TCP maximum segment length), the data packet must be divided into blocks, so that even if there is a If a block is lost or corrupted, only that one block needs to be resent instead of the entire packet. In the TCP protocol, we call each block a TCP segment ( TCP Segment ).

   Network layer

   We don't want the transport layer protocol to handle too many things. We only need to serve the application well, let it serve as the medium of data transmission between applications, and help realize application-to-application communication, and the actual transmission function is handed over to the next layer. That is the network layer ( Internet Layer ).

   The most common use of the network layer is the IP protocol ( Internet Protocol ). The IP protocol will use the transport layer message as the data part, plus the IP header to assemble the IP message. If the size of the IP message exceeds the MTU (generally in the Ethernet 1500 bytes) will be fragmented again to get an IP packet to be sent to the network.

   The addressing function of the IP protocol is to tell us which direction to go to the next destination, and the routing is to select the path according to the "next destination". Addressing is more like navigating, and routing is more like operating a steering wheel.

   network interface layer

   After the IP header is generated, it is then handed over to the network interface layer ( Link Layer ) to add a MAC header in front of the IP header, and encapsulate it into a data frame (Data frame) and send it to the network.

   The way Ethernet judges the destination of a network packet is different from IP, so a matching method must be used to send the packet to the destination in Ethernet, and the MAC header is used for this, so, in Ethernet Communication uses the MAC address.

   The MAC header is the header used by Ethernet, which contains information such as the MAC address of the receiver and the sender, and we can obtain the MAC address of the other party through the ARP protocol.

   Therefore, the network interface layer mainly provides "link-level" transmission services for the network layer. It is responsible for sending original data packets on the underlying network such as Ethernet and WiFi. It works at the level of the network card and uses the MAC address to identify the network. equipment.

2. Enter the URL to display the webpage, what happened during

   • According to the domain name, perform DNS domain name resolution;
   • Get the resolved IP address and establish a TCP connection;
   • Send an HTTP request to the IP address;
   • The server processes the request;
   • Return the response result;
   • Close the TCP connection;
   • The browser parses the HTML;
   • Browser layout rendering;

   What technology is behind

   1. Check the browser cache to see if there is any cached one, if not

   2. Check the local host file,

   3. Call API, Socket function gethostbyname under Linux

   4. Send a DNS request to the DNS server and query the local DNS server, which uses the UDP protocol

   5. If the ARP address resolution protocol is used to perform ARP query in a subnet, if it is not in a subnet, it needs to perform DNS query on the default gateway. If it cannot be found, it will always search for the root DNS server until it finally gets the IP address. (More than 400 root DNS servers worldwide, managed by 13 different organizations)

   6. At this time, we have the IP address of the server and the default port number. HTTP is 80 by default and https is 443 port number. Yes, first try http and then call Socket to establish a TCP connection.

   7. After the connection is successfully established after three handshakes, start to transmit data. If it is the http protocol, just return and finish.

   8. If it is not the http protocol, the server will return a redirection message starting with 5, telling us that we are using https, which means that the IP has not changed, but the port number has changed from 80 to 443. Okay, let’s wave again four times , done,

   9. Let’s do it again. This time, in addition to changing the above-mentioned port number from 80 to 443, SSL encryption technology will also be used to ensure the security of the transmitted data and ensure that the data will not be modified or replaced during the transmission process.

   10. This time, it is still a three-way handshake, communicate the authentication algorithm, encryption and verification algorithm used by both parties, and also verify the CA security certificate of the other party during the process.

   11. After confirming that it is correct, start communication, and then the server will return some data of the website you want to visit. During this process, the interface will be rendered, involving ajax technology and the like, until finally we see the colorful webpage

2. HTTP/HTTPS protocol

1. What is the HTTP protocol

   HTTP is Hypertext Transfer Protocol, that is HyperText Transfer Protocol .

   The name of HTTP "Hypertext Protocol Transport", it can be broken down into three parts:

   • Hypertext
   • transmission
   • protocol

2. GET and POST

   GET is used to fetch resources, while POST is used to transfer entity bodies.

   1. get is to get data, post is to modify data

   2. Get puts the requested data on the url, separates the URL and the transmission data with ?, and connects the parameters with &, so get is not very safe. And post puts the data in the HTTP package body (request body is relatively safe)

   3. The maximum data submitted by get is 2k (the limit actually depends on the browser), and the post has no limit in theory.

   4. GET generates a TCP data packet, the browser will send the http header and data together, and the server responds with 200 (return data); POST generates two TCP data packets, the browser sends the header first, the server responds with 100 continue, and the browser then Send data, and the server responds with 200 ok (return data).

   5. GET requests will be actively cached by the browser, but POST will not, unless manually set.

   6. Essential difference: GET is idempotent, but POST is not idempotent

      Idempotency here: Idempotence means that one request and multiple requests for a resource should have the same side effects. In simple terms it means that multiple requests to the same URL should return the same result.

   Because of their differences, you should not and cannot use get requests to add, delete, and modify data, which have side effects . Because the get request is idempotent, it will try to retry in the tunnel with bad network . If you use get requests to increase data, there is a risk of repeated operations , and such repeated operations may cause side effects (browsers and operating systems don't know that you will use get requests to increase operations).

3. HTTP caching technology

   For some repetitive HTTP requests, for example, the data obtained by each request is the same, we can cache the "request-response" data locally, then read the local data directly next time, without having to The response of the server is obtained through the network, so the performance of HTTP/1.1 must be improved visible to the naked eye.

   Therefore, the way to avoid sending HTTP requests is through caching technology . There are two implementation methods of HTTP caching, namely mandatory caching and negotiation caching .

   mandatory caching

   Strong caching means that as long as the browser judges that the cache has not expired, it will directly use the browser's local cache. The initiative to decide whether to use the cache lies with the browser.

   Strong caching is implemented using the following two HTTP response header (Response Header) fields, both of which are used to indicate the validity period of resources in the client cache:

   • Cache-Control, is a relative time;
   • Expires, is an absolute time;

   If the HTTP response header has both Cache-Control and Expires fields, Cache-Control has a higher priority than Expires .

   Cache-control has more options and more fine-grained settings, so it is recommended to use Cache-Control to achieve strong caching. The specific implementation process is as follows:

   • When the browser requests to access the server resource for the first time, the server will add Cache-Control to the Response header while returning the resource, and the expiration time is set in the Cache-Control;
   • When the browser requests to access the resource in the server again, it will first calculate whether the resource has expired through the time of requesting the resource and the expiration time set in Cache-Control. If not, use the cache, otherwise request the server again ;
   • After the server receives the request again, it will update the Cache-Control in the Response header again.

   negotiation cache

   Negotiation caching means after negotiating with the server, the negotiation result is used to determine whether to use the local cache.

   Negotiation caching can be implemented based on two headers.

   The first type: If-Modified-Sincethe field and Last-Modifiedthe field in the response header are implemented. The meaning of these two fields is:

   • In the response header Last-Modified: mark the last modification time of this response resource;
   • In the request header If-Modified-Since: When the resource expires and the Last-Modified statement is found in the response header, the Last-Modified time will be included when the request is initiated again. Compare the last modification time of the requested resource (Last-Modified). If the last modification time is newer (larger), it means that the resource has been modified again, and return the latest resource with HTTP 200 OK; if the last modification time is older (smaller), it means There is no new modification of the resource, and the response HTTP 304 goes to the cache.

   The second type: If-None-Matchthe field and ETagthe field in the response header. These two fields mean:

   • In the response header Etag: uniquely identify the response resource;
   • In the request header If-None-Match: When the resource expires and the browser finds that there is an Etag in the response header, it will set the If-None-Match value of the request header to the value of the Etag when it initiates a request to the server again. After receiving the request, the server performs a comparison. If the resource has not changed, it returns 304, and if the resource has changed, it returns 200.

   The first implementation method is based on time, and the second implementation method is based on a unique identifier. Relatively speaking, the latter can more accurately determine whether the file content has been modified, and avoid unreliable problems caused by time tampering.

   The process of negotiating caching when using the ETag field:

   • When the browser requests to access the server resource for the first time, the server will add the ETag unique identifier to the Response header while returning the resource. The value of this unique identifier is generated based on the currently requested resource;

   • When the browser requests to access the resource in the server again, it will first check whether the mandatory cache has expired:

     • If not expired, use the local cache directly;
     • If the cache expires, an If-None-Match field will be added to the Request header, and the value of this field is the unique identifier of ETag;

   • After the server receives the request again,

     It will be compared with the unique identifier generated by the currently requested resource according to the If-None-Match value in the request:

     • If the values ​​are equal, return 304 Not Modified and no resource will be returned ;
     • If they are not equal, return 200 status code and return resource, and add a new ETag unique identifier to the Response header;

   • If the browser receives a 304 request response status code, it will load the resource from the local cache, otherwise it will update the resource.

4. How does HTTP implement long connections? When will it time out?

   By setting Connection: keep-alive in the header (request and response header), the HTTP1.0 protocol is supported, but it is disabled by default. From the HTTP1.1 protocol onwards, the connection is always a long connection by default.

   1. HTTP generally has an httpd daemon process, in which you can set a keep-alive timeout. When the tcp connection is idle for more than this time, it will be closed. You can also set the timeout time in the HTTP header.

   2. The keep-alive of TCP contains three parameters, which can be set in the net.ipv4 of the system kernel: when the TCP connection is idle and the tcp_keepalive_time is idle, a detection packet will be generated. If no ACK is received from the other party, it will Send it again every tcp_keepalive_intvl, until tcp_keepalive_probes are sent, the link will be discarded.

   （1）tcp_keepalive_intvl = 15
   （2）tcp_keepalive_probes = 5
   （3）tcp_keepalive_time = 1800

   In fact, HTTP does not have long and short links, only TCP does. TCP long connections can reuse one TCP link to initiate multiple HTTP requests, which can reduce resource consumption. For example, if you request HTML once, you may need to request subsequent JS/CSS/pictures, etc.

5. Introduction, advantages and disadvantages of HTTP 1.0/1.1/2.0/3.0

   HTTP/1.0

   In May 1996, the HTTP/1.0 version was released. In order to improve the efficiency of the system, HTTP/1.0 stipulates that the browser and the server only maintain a short connection. Each request of the browser needs to establish a TCP connection with the server, and the server completes the request processing. Immediately after disconnecting the TCP connection , the server does not keep track of each client nor record past requests.

   In HTTP/1.0, the browser and the server only maintain a short-term connection, and the connection cannot be reused. That is to say, only one request can be sent per TCP connection. After the data is sent, the connection is closed. If you want to request other resources, you must create a new connection.

   We know that the establishment of a TCP connection requires a three-way handshake, which is a time-consuming process. Therefore, the performance of the HTTP/1.0 version is relatively poor.

   HTTP/1.1

   The most prominent advantages of HTTP are "simple, flexible and easy to expand, widely used and cross-platform".

   The HTTP protocol has a double-edged sword with advantages and disadvantages , namely "stateless, clear text transmission", and at the same time there is a major disadvantage "insecure".

   In order to solve the defects of HTTP/1.0, HTTP/1.1 was born in 1999. Compared with HTTP/1.0, the main improvement is the introduction of persistent connections. The so-called persistent connection means that the TCP connection is not closed by default and can be multiplexed by multiple requests .

   When the client and server find that the other party has not been active for a period of time, they can actively close the connection. Or the client actively tells the server to close the connection during the last request.

   HTTP/1.1 version also introduces the pipeline mechanism (pipelining), that is, in the same TCP connection, the client can send multiple requests at the same time. This further improves the efficiency of the HTTP protocol.

   With persistent connections and pipelines, the efficiency of HTTP is greatly improved. However, the server is still executed sequentially, and there is still room for improvement in efficiency.

   HTTP/2

   HTTP/2 is the first update of the HTTP protocol since the release of HTTP 1.1 in 1999, mainly based on the SPDY protocol.

   HTTP/2 In order to solve the efficiency problem still existing in HTTP/1.1, HTTP/2 adopts multiplexing . That is, in a connection, both the client and the browser can send multiple requests or responses at the same time, and there is no need for one-to-one correspondence in order. A prerequisite for this is that HTTP/2 has binary framing , that is, HTTP/2 will divide all transmitted information into smaller messages and frames (frame), and encode them in binary format.

   And this layer responsible for splitting, assembling requests and binary frames is called the binary framing layer .

   In addition, there are some other optimizations, such as header compression, server push, etc.

   Header compression is to compress the dialogue between the boss and employees.

   Server-side push means that employees send some things that the boss may ask to the boss's mobile phone (cache) in advance. In this way, the boss can directly read the text message (cache) when he wants to know.

   HTTP/2 performance improvements over HTTP/1.1:

   • head compression
   • binary format
   • concurrent transfer
   • The server actively pushes resources

   What's wrong with HTTP/2?

   HTTP/2 solves the problem of head-of-line blocking in HTTP/1 through the concurrency capability of Stream, which seems to be perfect, but HTTP/2 still has the problem of "head-of-line blocking", but the problem is not at the level of HTTP. But at the TCP layer.

   HTTP/2 is based on the TCP protocol to transmit data. TCP is a byte stream protocol. The TCP layer must ensure that the received byte data is complete and continuous, so that the kernel will return the data in the buffer to the HTTP application. Then when the "first 1 byte of data" does not arrive, the last byte of data received can only be stored in the kernel buffer, and only when the 1 byte of data arrives can the HTTP/2 application layer receive data from the kernel. Get the data, this is the HTTP/2 head-of-line blocking problem.

   HTTP3.0

   Although HTTP/2 has the ability to transmit multiple streams concurrently, but the transport layer is the TCP protocol, so there are the following defects:

   • Head-of-line blocking , HTTP/2 multiple requests run in one TCP connection, if the TCP segment with a lower sequence number is lost during network transmission, even if the TCP segment with a higher sequence number has been received, the application layer cannot start from This part of the data is read in the kernel, from the perspective of HTTP, multiple requests are blocked;
   • TCP and TLS handshake delay , TCP three-way handshake and TLS four-way handshake, a total of 3-RTT delay;
   • Connection migration requires reconnection . When a mobile device switches from a 4G network environment to WiFi, since TCP confirms a TCP connection based on quadruples, changes in the network environment will result in changes in the IP address or port, so TCP can only Disconnect and then re-establish the connection, the cost of switching the network environment is high;

   HTTP/3 replaced the transport layer from TCP to UDP, and developed the QUIC protocol on the UDP protocol to ensure reliable data transmission.

   Features of the QUIC protocol:

   • There is no head-of-line blocking , there is no dependency between multiple Streams on the QUIC connection, they are all independent, and there will be no underlying protocol restrictions. Packet loss occurs in a certain stream, only this stream will be affected, and other streams will not be affected ;
   • The establishment of the connection is fast , because QUIC contains TLS 1.3 inside, so only 1 RTT is needed to complete the establishment of the connection and the TLS key negotiation "simultaneously", and even in the second connection, the application data packet can shake hands with QUIC ( Connection information + TLS information) are sent together to achieve the effect of 0-RTT.
   • Connection migration , the QUIC protocol does not use a quadruple to "bind" the connection, but to mark the two endpoints of the communication through the "connection ID". The client and the server can each choose a set of IDs to mark themselves, so even if After the network of the mobile device changes, the IP address changes. As long as the context information (such as connection ID, TLS key, etc.) is still retained, the original connection can be reused "seamlessly" to eliminate the cost of reconnection;

   In addition, QPACK of HTTP/3 synchronizes the dynamic tables of both parties through two special unidirectional streams, which solves the head-of-queue blocking problem of HPACK of HTTP/2.

6. Introduction to HTTPS

   HTTPS is not a new protocol, but allows HTTP to communicate with SSL (Secure Sockets Layer) first, and then SSL and TCP to communicate, that is to say, HTTPS uses tunnels for communication . By using SSL, HTTPS has encryption (anti-eavesdropping), authentication (anti-spoofing), and integrity protection (tamper-proof).

7. How does HTTPS ensure the security of data transmission, and what is the overall process? (How does SSL work to ensure security)

   (1) The client initiates an SSL connection request to the server;

   (2) The server sends the public key to the client, and the server saves the only private key

   (3) The client uses the public key to encrypt the symmetric secret key for communication between the two parties and sends it to the server

   (4) The server uses its own unique private key to decrypt the symmetric key sent by the client,

   (5) For data transmission, both the server and the client use the same public symmetric key to encrypt and decrypt the data, which can ensure the security during the data sending and receiving process, even if the third party obtains the data packet, it cannot be encrypted , decryption and tampering.

   Because digital signatures and abstracts are key weapons for certificate anti-counterfeiting. "Summary" is to calculate a fixed-length string through the hash algorithm for the transmitted content. Encrypt this digest with the private key of the sender, and the result after encryption is a "digital signature"

   The basic idea of ​​the SSL/TLS protocol is to use public key encryption, that is to say, the client first asks the server for the public key, and then encrypts the information with the public key. After receiving the ciphertext, the server decrypts it with its own private key.

   Supplement : The four-way handshake of SSL/TLS, the current mainstream answers on the Internet are repeating Mr. Ruan Yifeng's blog, which belongs to the answer of TLS version 1.0, using the RSA key exchange algorithm. But now that TLS 1.2 has become the mainstream, using the ECDHE algorithm, it should be better if the interviewer can tell the answer of this version.

8. What is SSL/TLS

   SSL stands for Secure Sockets Layer. It is a protocol for encrypting and authenticating data sent between an application (such as a browser) and a web server. Authentication, Encryption The encryption mechanism of Https is a hybrid encryption mechanism that uses both shared key encryption and public key encryption.

   Functions of the SSL/TLS protocol: authenticate users and services, encrypt data, and maintain data integrity. Application layer protocol encryption and decryption require two different keys, so it is called asymmetric encryption; both encryption and decryption use the same key key

   Symmetric encryption: The advantage is that the encryption and decryption efficiency is usually relatively high. HTTPS is based on asymmetric encryption, and the public key is public.

9. TLS handshake process

   How does the TLS protocol solve the risks of HTTP?

   • Information encryption : HTTP interaction information is encrypted, so third parties cannot be stolen;
   • Verification mechanism : Verify whether the information has been tampered with by a third party during the transmission process, and if it has been tampered with, there will be a warning prompt;
   • Identity certificate : Prove that Taobao is the real Taobao;

   In the handshake process of TLS, each "box" is a record ( record ). The record is the basic unit of TLS sending and receiving data, similar to the segment in TCP. Multiple records can be combined into one TCP packet to send, so usually the TLS handshake can be completed after "four messages", that is, a delay of 2 RTTs is required , and then HTTP messages can be sent in a secure communication environment to achieve HTTPS protocol.

   HTTPS is an application layer protocol. It needs to complete the establishment of the TCP connection first, and then go through the TLS handshake process before establishing a secure communication connection.

10. HTTPS RSA handshake process

    HTTPS adopts a mixed encryption mechanism, uses asymmetric key encryption to transmit symmetric keys to ensure the security of the transmission process , and then uses symmetric key encryption to communicate to ensure the efficiency of the communication process .

    Ensure the transmission security process (actually the rsa principle):

    1. Client gives the protocol version number, a random number generated by the client (Client random), and the encryption method supported by the client.
    2. Server confirms the encryption method used by both parties, and gives a digital certificate and a random number (Server random) generated by the server.
    3. The client confirms that the digital certificate is valid, then generates a new random number (Premaster secret), and uses the public key in the digital certificate to encrypt the random number and send it to the server.
    4. Server uses its own private key to obtain the random number (Premaster secret) sent by Client.
    5. According to the agreed encryption method, Client and Server use the previous three random numbers to generate a "session key" (session key), which is used to encrypt the entire subsequent dialogue process.

    The biggest problem with using the RSA key agreement algorithm is that it does not support forward secrecy . Because the client sends the random number (one of the conditions used to generate the symmetric encryption key) to the server using public key encryption, after the server receives it, it will decrypt it with the private key to get the random number. So once the private key of the server is leaked, all TLS communication ciphertexts intercepted by third parties in the past will be cracked.

    In order to solve this problem, the ECDHE key agreement algorithm appeared later.

11. HTTPS ECDHE handshake process

    The ECDHE algorithm uses the ECC elliptic curve characteristics on the basis of the DHE algorithm, and can calculate the public key and the final session key with less calculation.

    TLS first handshake

    The client will first send a " Client Hello " message, which contains the TLS version number used by the client, the list of supported cipher suites, and the generated random number (*Client Random*) .

    TLS second handshake

    When the server receives the "greeting" from the client, it also needs to return the gift, and will return a " Server Hello " message. The message contains the TLS version number confirmed by the server, and a random number (*Server Random*) is also given , and then from the client Select an appropriate cipher suite from the list of cipher suites on the client side.

    Then, in order to prove its identity, the server sends a " Certificate " message, and the certificate will also be sent to the client.

    This step is very different from the RSA handshake process, because the server selects the ECDHE key agreement algorithm, so it will send the " Server Key Exchange " message after sending the certificate.

    This process server does three things:

    • The elliptic curve named x25519 is selected , and the selected elliptic curve is equivalent to the base point G of the elliptic curve. These will be disclosed to the client;
    • Generate a random number as the private key of the server-side elliptic curve and keep it locally;
    • Calculate the elliptic curve public key of the server based on the base point G and the private key , which will be disclosed to the client.

    In order to ensure that the public key of the elliptic curve is not tampered with by a third party, the server will use the RSA signature algorithm to sign the public key of the elliptic curve of the server.

    Then came the " Server Hello Done " message, the server and the client said: "These are the information I provided, the greeting is over."

    So far, the two TLS handshakes have been completed. At present, the client and the server share the following information in plain text: Client Random , Server Random, the elliptic curve used, the base point G of the elliptic curve, the public key of the elliptic curve of the server , These pieces of information are very important and are the materials for subsequent generation of session keys.

    TLS third handshake

    After the client receives the certificate from the server, it is natural to verify whether the certificate is legal. If the certificate is legal, then the server has no problem with the identity. The process of verifying the certificate will go through the certificate chain to verify step by step to confirm the authenticity of the certificate, and then use the public key of the certificate to verify the signature, so that the identity of the server can be confirmed. After the confirmation is correct, you can continue to go down.

    The client will generate a random number as the private key of the client's elliptic curve, and then generate the client's elliptic curve public key according to the information given by the server, and then send it to the server with a " Client Key Exchange " message.

    After calculating the session key, the client will send a " Change Cipher Spec " message to tell the server to use a symmetric algorithm to encrypt the communication.

    Then, the client will send an " Encrypted Handshake Message " message, make a summary of the previously sent data, and then encrypt it with a symmetric key, and let the server do a verification to verify whether the symmetric key generated this time can be used normally .

    TLS fourth handshake

    Finally, the server will also perform the same operation, sending " Change Cipher Spec " and " Encrypted Handshake Message " messages. If both parties verify that the encryption and decryption are OK, the handshake is officially completed. Then, you can send and receive encrypted HTTP requests and responses normally.

12. The difference between RSA and ECDHE handshake process

    • The RSA key agreement algorithm "does not support" forward secrecy, and the ECDHE key agreement algorithm "supports" forward secrecy;
    • The RSA key agreement algorithm is used, and the application data can only be transmitted after TLS completes the four-way handshake. For the ECDHE algorithm, the client can send encrypted HTTP data in advance without waiting for the last TLS handshake of the server, saving a The round-trip time of the message (this is stipulated in the RFC document, the specific reason is not explained in the document, so I don't quite understand this);
    • Using ECDHE, in the second TLS handshake, there will be a "Server Key Exchange" message sent by the server, but there is no such message in the RSA handshake process;

13. FIN_WAIT_2, CLOSE_WAIT state and TIME_WAIT state

    • FIN_WAIT_2：
      • half-closed state.
      • The party that sent the disconnection request still has the ability to receive data, but has no ability to send data.
    • CLOSE_WAIT state:
      • The party that passively closes the connection will immediately respond with an ACK packet after receiving the FIN packet, indicating that the disconnection request has been received.
      • If the party that passively closes the connection has remaining data to send, it will enter the CLOSE_WAIT state.
    • TIME_WAIT state:
      • Also called 2MSL waiting state.
      • If the client directly enters the CLOSED state, if the server does not receive the last ACK packet, it will resend the FIN packet after the timeout. At this time, because the client is already CLOSED, the server will not receive the ACK but the RST . Therefore, the purpose of the TIME_WAIT state is to prevent the last handshake data from reaching the other party and triggering the retransmission of FIN preparations.
      • During the 2MSL time, the same socket can no longer be used, otherwise it may be confused with the old connection data (if the socket of the new connection and the old connection are the same).

14. Symmetric key encryption and asymmetric key encryption

    Symmetric Key Encryption

    Symmetric-Key Encryption (Symmetric-Key Encryption), encryption and decryption use the same key.

    • Advantages: fast operation
    • Disadvantage: cannot securely transmit the key to the communicating party

    Asymmetric Key Encryption

    Asymmetric key encryption, also known as public key encryption (Public-Key Encryption), uses different keys for encryption and decryption.

    The public key can be obtained by everyone. After the communication sender obtains the receiver's public key, it can use the public key to encrypt , and the receiver can use the private key to decrypt after receiving the communication content .

    In addition to being used for encryption, asymmetric keys can also be used for signing. Because the private key cannot be obtained by others, the sender of the communication uses its private key to sign, and the receiver of the communication uses the sender's public key to decrypt the signature to determine whether the signature is correct.

    • Advantages: the public key can be transmitted to the communication sender more securely;
    • Disadvantage: slow operation speed.

15. How to optimize HTTPS

    For the direction of hardware optimization, because HTTPS is computationally intensive, you should choose a CPU with stronger computing power, and it is best to choose a CPU that supports the AES-NI feature . This feature can optimize the AES symmetric encryption algorithm at the hardware level and speed up application data. encryption and decryption.

    For the direction of software optimization, if possible, upgrade the software to a newer version, such as upgrading the Linux kernel 2.X to 4.X, and upgrading openssl 1.0.1 to 1.1.1, because the new version of the software will not only provide New features, but also fix the problems of the old version.

    For the direction of protocol optimization:

    • The key exchange algorithm should choose the ECDHE algorithm instead of the RSA algorithm, because the ECDHE algorithm has forward security, and the client can send encrypted application data after the third handshake, saving 1 RTT.
    • Upgrade TLS1.2 to TLS1.3 , because the handshake process of TLS1.3 only needs 1 RTT, and the security is stronger.

    For the direction of certificate optimization:

    • The server should choose ECDSA certificate instead of RSA certificate, because under the same security level, the key length of ECC is much shorter than that of RSA, which can improve the efficiency of certificate transmission;
    • The server should enable the OCSP Stapling function, and the server will obtain the OCSP response in advance, and cache the response result, so that there is no need to visit the CA server during the TLS handshake, which reduces the overhead of network communication and improves the efficiency of certificate verification;

    When reconnecting to HTTPS, we can use some techniques to allow the client and server to use the session key used in the last HTTPS connection to directly restore the session without going through the complete TLS handshake process again.

    Common session reuse technologies include Session ID and Session Ticket. With session reuse technology, when HTTPS is reconnected again, only 1 RTT is needed to restore the session. For TLS1.3 using the Pre-shared Key session reuse technology, only 0 RTT is needed to restore the session.

    Although these session reuse technologies are easy to use, they have certain security risks. Not only do they not have forward security, but they also have the risk of replay attacks. Therefore, a reasonable expiration time should be set for the session key.

16. WebSocket protocol

    Websocket is a network technology for full-duplex communication between the browser and the server , which belongs to the application layer protocol . It is based on the TCP transport protocol and reuses the HTTP handshake channel to make up for the lack of persistent communication capabilities of the HTTP protocol.

    • ws default port: 80
    • wss default port: 443
    • Websocket handshakes through the HTTP protocol .

    What are the characteristics of websocket?

    1. Save resource overhead, HTTP requests must carry a complete header every time, this overhead is significantly reduced;
    2. Stronger real-time performance. Since the protocol is full-duplex communication, the server can actively push data to the client. Compared with HTTP requests, it needs to wait for the client to initiate a request before the server can respond, and the delay is significantly less;
    3. Keep the connection status and be able to record the user status, and some status information can be omitted during communication, unlike HTTP that needs to carry user authentication information every time;
    4. Better binary support, Websocket defines binary frames, which can handle binary content more easily than HTTP.

17. How HTTP implements long connection

    HTTP long connection

    • After the browser makes an HTTP session visit to the server, it will not directly close the connection, but will keep it for a period of time by default, then the next time the browser continues to visit, it will use this connection again.
    • In HTTP/1.1the version, the default connection is a long connection, we can Connection: keep-alivespecify it through the field.

    TCP keep-alive mechanism

    • Why should there be a keep-alive mechanism?

      • The first point is naturally the theme of our article. Through the keep-alive mechanism, we can ensure that the connection between the two parties will not be released
      • The second point is that in other cases, if the client or server has an error or crashes, then this keep-alive mechanism can be used to detect a problem with network communication, and then the wrong connection can be released.

    • Keep Alive Mechanism

      First of all, the working principle of the keep-alive mechanism is that by setting a keep-alive timer on the server side, when the timer starts working, it will regularly send a keep-alive detection TCP message to the other end of the network communication. If an ACK message is received , then it proves that the other party is alive and can continue to maintain the connection; otherwise, it proves that the network is faulty.

      The above is just a brief introduction at the principle level. According to the literature [1], we can understand the detailed content:

      • If there is no action on a given connection for two hours, the server sends a probe segment to the client. The client host must be in one of the following 4 states.
      • State 1: The client host is still running normally and reachable from the server. The client's TCP response is normal, and the server knows that the other party is working normally. The server resets the keep-alive timer after two hours. If there is application traffic over this connection before the two-hour timer expires, the timer is reset 2 hours in the future after the data exchange.
      • State 2: The client host has crashed and is shutting down or restarting. In either case there is no response from the client's TCP. The server will not be able to receive a response to the probe and will timeout after 75 seconds. The server sends a total of 10 such probes, each with an interval of 75 seconds. If the server does not receive a response, it assumes that the client host is closed and terminates the connection.
      • State 3: The client host has crashed and has been restarted. At this point the server will receive a response to its keep-alive probe, but this response is a reset, causing the server to terminate the connection.
      • State 4: The client host is running normally, but the slave server is unreachable. This is the same as state 2, since TCP cannot tell the difference between state 4 and state 2, all it can find out is that no response to the probe was received.

18. The difference between HTTP and HTTPS

    The Http protocol runs on TCP and is transmitted in clear text. Neither the client nor the server can verify the identity of the other party; Added encryption and authentication mechanisms to HTTP. There are following differences between the two:

    1. Different ports: Http and Https use different connection methods, and the ports used are also different. The former is 80, and the latter is 443;

    2. Resource consumption: Compared with HTTP communication, Https communication will consume more CPU and memory resources due to encryption and encryption processing;

    3. Overhead: Https communication requires a certificate, and the certificate generally needs to be purchased from a certification authority;

    4. Security: The HTTP connection is very simple and stateless; the HTTPS protocol is a network protocol constructed by the TLS+HTTP protocol that can perform encrypted transmission and identity authentication. It is safer than the HTTP protocol. The encryption mechanism of Https is a shared secret　
    . A hybrid encryption mechanism that uses both private key encryption and public key encryption.

    What are the application scenarios of websocket?

    • Instant messaging, live streaming, games, online collaboration tools (Tencent documents, etc.), real-time data pull and map push

19. How many HTTP request methods are there

    The first line of the request message sent by the client is the request line, which contains the method field.

    According to the HTTP standard, HTTP requests can use several request methods.

    HTTP1.0 defines three request methods: GET, POST and HEAD methods.

    HTTP1.1 adds six new request methods: OPTIONS, PUT, PATCH, DELETE, TRACE and CONNECT methods.

    serial number	method	describe
    1	GET	Request the specified page information and return the entity body.
    2	HEAD	Similar to a GET request, except that there is no specific content in the returned response, which is used to obtain the header
    3	POST	Submit data to a specified resource to process a request (such as submitting a form or uploading a file). Data is included in the request body. POST requests may result in the creation of new resources and/or the modification of existing resources.
    4	PUT	The data sent from the client to the server replaces the content of the specified document.
    5	DELETE	Requests the server to delete the specified page.
    6	CONNECT	Reserved in the HTTP/1.1 protocol for proxy servers that can pipe connections into.
    7	OPTIONS	Allows clients to view server performance.
    8	TRACE	Echoes the requests received by the server, mainly for testing or diagnosis.
    9	PATCH	Complementary to the PUT method for partial updates to known resources.

3. TCP three-way handshake/four-way wave

1. The process of TCP three-way handshake and four-way wave

   Three-way handshake process

   • Initial state : the client is in closed(关闭)state and the server is in listen(监听)state.
   • The first handshake : the client sends a request message SYN = 1to seq = xsend the synchronization sequence number and initialization sequence number to the server, and the client is in the SYN_Sendstate after sending. (Verify the sending ability of the client and the receiving ability of the server)
   • The second handshake : After receiving SYNthe request agrees to the connection, it will respond with its own synchronization sequence number SYN(服务端) = 1, initialization sequence number seq = y, confirmation sequence number (the data packet expected to be received next time) ack = x+ 1and a confirmation number ACK = 1message. The server is SYN_Receivestatus. (Here comes the problem. After two handshakes, think from the perspective of the client: I send and receive ok, and the send and receive of the server are also ok. But think from the perspective of the server: Oh, my server receives ok, But I don't know if my sending is ok or not, and I don't know how your receiving ability is? So bro, you need to shake hands three times to send me a message to tell me. If you don't tell me, in case I I think you ran away, and then I may continue to send you another time for security reasons, to see if you can get back to me.)
   • The third handshake : SYN + ACKAfter knows that it can send the next sequence of data packets next time, and then sends the synchronization sequence number ack = y + 1and the sequence number of the data packet seq = x + 1and confirmation number ACK = 1confirmation packet as a response, and the client turns to for establishedstatus. (Thinking from the perspective of both sides, each is ok)

   Why do we need a three-way handshake, not two?

   To clarify this issue, we need to first understand what the purpose of the three-way handshake is, and whether we can achieve the same purpose with only two handshakes.

   • The first handshake: the client sends a network packet, and the server receives it. In this way, the server can conclude that the sending capability of the client and the receiving capability of the server are normal.
   • The second handshake: the server sends a packet, and the client receives it. In this way, the client can conclude that the receiving and sending capabilities of the server and the receiving and sending capabilities of the client are normal. However, at this time, the server cannot confirm whether the receiving ability of the client is normal.
   • The third handshake: the client sends a packet, and the server receives it. In this way, the server can conclude that the client's receiving and sending capabilities are normal, and the server's own sending and receiving capabilities are also normal.

   Analyze the reasons for the three-way handshake in three aspects:

   • The three-way handshake can prevent the initialization of repeated historical connections (the main reason)
   • Three-way handshake can synchronize the initial serial numbers of both parties
   • Three-way handshake can avoid resource waste

   When TCP establishes a connection, the three-way handshake can prevent the establishment of historical connections, reduce unnecessary resource overhead on both sides, and help both parties initialize sequence numbers synchronously . Sequence numbers can ensure that data packets are not repeated, not discarded, and transmitted in order.

   Reasons for not using "two-way handshake" and "four-way handshake":

   • "Two handshakes": It is impossible to prevent the establishment of historical connections, which will cause a waste of resources on both sides, and cannot reliably synchronize the serial numbers of both parties;
   • "Four-way handshake": The three-way handshake has theoretically established the least reliable connection, so there is no need to use more communication times.

   four wave process

   • Initialization state : Both the client and the server are in the connected state, and then four disconnection operations will be performed.
   • The first breakup : Both the client and the server can initiate the first breakup, because TCP is full-duplex.

   If the data sent by the client has been sent, send FIN = 1 to tell the server that all the data on the client has been sent , and you can close the receiving on the server , but if your server has data to send to the client, the client Still acceptable. At this time, the client is in the FIN = 1 state waiting for the server to confirm the release of the connection.

   • The second breakup : After the server receives the client’s release request connection, it knows that the client has no data to send to itself , and then the server sends ACK = 1 to tell the client that it has received the information you sent me . At this time, the server In CLOSE_WAIT waiting to close state. (The server first responds to the client, I know, but the server's ability to send data is about to be turned off, so the third time will come.)
   • The third breakup : At this time, the server has sent all the data to the client, and then sends a FIN = 1 to tell the client that all the data on the server has been sent , and you can also close the connection to receive data on the client . At this time, the server state is in the LAST_ACK state, waiting to confirm whether the client has received its request. (The server waits for the client to reply whether it has been received. If not, the server does not know whether the client is hung up or what is going on, so the server does not dare to close its receiving ability, so the fourth time coming.)
   • The fourth breakup : At this time, if the client receives the information sent by the server, it sends ACK = 1, telling the server that the client has received your information. There is a delay wait of 2 MSL .

   Why do you need to wave four times

   Because when the server receives the SYN connection request message from the client, it can directly send the SYN+ACK message. Among them, the ACK message is used for response, and the SYN message is used for synchronization . But when the connection is closed, when the server receives the FIN message, it may not close the SOCKET immediately, so it can only reply with an ACK message first, telling the client, "The FIN message you sent has been received". I can only send the FIN message until all the messages on my server have been sent, so they cannot be sent together. So four waves are required.

2. Introduction to TCP protocol

   TCP (Transmission Control Protocol) is a connection-oriented, reliable, byte stream-based transport layer communication protocol.

   features

   • TCP is connection-oriented .
   • Each TCP connection can only have two endpoints, and each TCP connection can only be point-to-point ( one-to-one );
   • TCP provides a reliably delivered service . The data transmitted through the TCP connection has no error, no loss, no repetition, and arrives in order;
   • TCP provides full-duplex communication . TCP allows the application processes on both sides of the communication to send data at any time. Both ends of the TCP connection are equipped with a sending buffer and a receiving buffer, which are used to temporarily store the communication data between the two parties;
   • Oriented to byte streams . A "stream" in TCP refers to a sequence of bytes flowing into or out of a process. The meaning of "byte stream-oriented" is: Although the interaction between the application program and TCP is one data block (of different sizes) at a time, TCP only regards the data handed over by the application program as a series of unstructured byte streams.

3. What happens when TCP three-way handshake and four-way handshake are lost

   Three-way handshake lost

   The first handshake is lost, what happens?

   If the client fails to receive the SYN-ACK message from the server (the second handshake), it will trigger the "timeout retransmission" mechanism to retransmit the SYN message, and the serial number of the retransmitted SYN message is the same . Each timeout is twice as long as the previous time .

   The second handshake is lost, what happens?

   Because the second handshake message contains the ACK confirmation message of the first handshake to the client, if the client has not received the second handshake for a long time, then the client thinks that its own SYN message ( The first handshake) is lost, so the client will trigger the timeout retransmission mechanism and retransmit the SYN message .

   Then, because the second handshake contains the server's SYN message, when the client receives it, it needs to send an ACK confirmation message to the server (the third handshake), and the server will think that the SYN message has been received by the client. end received.

   Then, if the second handshake is lost, the server will not receive the third handshake, so the server will trigger a timeout retransmission mechanism to retransmit the SYN-ACK message .

   Therefore, when the second handshake is lost, both client and server retransmit

   The third handshake is lost, what happens?

   After the client receives the SYN-ACK message from the server, it will return an ACK message to the server, which is the third handshake, and the client state enters ESTABLISHthe state

   Because the ACK of the third handshake is the confirmation message of the SYN of the second handshake, so when the third handshake is lost, if the server side fails to receive the confirmation message, it will trigger a timeout restart. The transmission mechanism retransmits the SYN-ACK message until the third handshake is received, or the maximum number of retransmissions is reached.

   Note that the ACK message will not be retransmitted. When the ACK is lost, the corresponding message will be retransmitted by the other party .

   wave four lost

   The first wave is lost, what happens?

   If the first wave is lost, and the client fails to receive the ACK from the passive party, it will trigger the timeout retransmission mechanism and retransmit the FIN message. The number of retransmissions is controlled by tcp_orphan_retriesparameters

   When the number of times the client retransmits the FIN message tcp_orphan_retriesexceeds , it will no longer send the FIN message, and will wait for a period of time (the time is twice the time of the previous timeout), if it still fails to receive the second wave, Then go directly to closethe state .

   The second wave is lost, what happens?

   When the server receives the first waving from the client, it will return an ACK confirmation message first, and the connection of the server enters CLOSE_WAITthe state .

   We also mentioned earlier that the ACK message will not be retransmitted, so if the server's second wave is lost, the client will trigger the timeout retransmission mechanism and retransmit the FIN message until it receives the server's first wave. Wave twice, or reach the maximum number of retransmissions.

   The third wave is lost, what happens?

   When the server (passive closing party) receives the FIN message from the client (active closing party), the kernel will automatically reply ACK, and the connection is in CLOSE_WAITthe state of

   At this time, the kernel has no right to replace the process to close the connection, and the process must actively call the close function to trigger the server to send a FIN message.

   When the server is in the CLOSE_WAIT state and the close function is called, the kernel will send a FIN message, and the connection will enter the LAST_ACK state at the same time, waiting for the client to return ACK to confirm the connection is closed.

   If the ACK is not received for a long time, the server will resend the FIN message, and the number of retransmissions is still tcp_orphan_retriecontrolled by the s parameter, which is the same as the control method of the number of retransmissions of the FIN message resent by the client.

   The fourth wave is lost, what happens?

   When the client receives the FIN message of the third hand wave from the server, it will return an ACK message, that is, the fourth hand wave, and the client connection enters TIME_WAITthe state

   In the Linux system, the TIME_WAIT state will last for 2MSL before entering the closed state.

   Then, before the server (passive closing party) does not receive the ACK message, it is still in the LAST_ACK state.

   If the ACK message of the fourth hand wave does not reach the server, the server will resend the FIN message, and the number of retransmissions is still controlled by tcp_orphan_retriesthe parameters

4. 2MSL wait state

   2MSL wait state

   The TIME_WAIT state also becomes a 2MSL wait state. Each specific TCP implementation must choose a maximum segment lifetime MSL (Maximum Segment Lifetime), which is the longest time in the network before any segment is discarded. This time is limited because the TCP segment is transmitted in the network as an IP datagram, and the IP datagram has a TTL field that limits its lifetime.

   For a specific implementation of the given MSL value, the principle of processing is: when TCP performs an active close and sends back the last ACK, the connection must stay in the TIME_WAIT state for 2 times the MSL. This allows TCP to resend the final ACK in case this ACK is lost (the other end times out and resends the final FIN).

   Another result of this 2MSL wait is that during the 2MSL wait for this TCP connection, the sockets that define this connection (the client's IP address and port number, the server's IP address and port number) cannot be used. This connection can only be used after the end of 2MSL.

   Why does the TIME_WAIT state need to go through 2MSL to return to the CLOSE state

   Theoretically, after all four messages are sent, it can directly enter the CLOSE state, but the network may be unreliable, and the last ACK may be lost. So the TIME_WAIT state is used to resend the ACK message that may be lost .

   The ACK = 1 sent by the client to the server is lost, and the server waits for 1MSL not received , and then needs 1MSL to resend the message . If the message from the server is received again, restart the 2MSL timer and send a confirmation request . The client only needs to wait for 2MSL. If it does not receive the message from the server again, it means that the server has received its own confirmation message; at this time, both parties have closed the connection, and the TCP has broken up four times.

5. Introduction to TIME_WAIT

   MSLIt is the Maximum Segment Lifetime, the maximum lifetime of a packet , which is the longest time any packet exists on the network, and the packet will be discarded after this time. Because the TCP message is based on the IP protocol, and there is a TTLfield , which is the maximum number of routes that the IP datagram can pass through. This value will be reduced by 1 every time it passes through a router that processes it. When the value is 0, the datagram will will be discarded, and an ICMP packet will be sent to notify the source host at the same time.

   The difference between MSL and TTL: The unit of MSL is time, while TTL is the number of route hops. Therefore, the MSL should be greater than or equal to the time when the TTL is 0 to ensure that the packet has been destroyed naturally.

   The value of TTL is generally 64, and Linux sets MSL to 30 seconds, which means that Linux thinks that the time for a data packet to pass through 64 routers will not exceed 30 seconds, and if it exceeds, it will consider that the packet has disappeared in the network .

   TIME_WAIT waits for twice the MSL. A more reasonable explanation is: There may be data packets from the sender in the network. When these data packets from the sender are processed by the receiver, they will send a response to the other party, so you need to wait back and forth . 2 times the time .

   Why is the TIME_WAIT state needed?

   Only the party that initiates the closing of the connection will have TIME-WAITthe state .

   The TIME-WAIT state is required for two main reasons:

   • Prevent the data in the historical connection from being incorrectly received by the subsequent connection with the same quadruple;
   • Ensure that the party that "passively closes the connection" can be closed correctly;

6. What's the harm in having too much TIME_WAIT?

   There are two main hazards of excessive TIME-WAIT states:

   • The first is to occupy system resources, such as file descriptors, memory resources, CPU resources, thread resources, etc.;
   • The second is to occupy port resources. Port resources are also limited. Generally, the ports that can be opened are 32768～61000, and the range can also be specified by net.ipv4.ip_local_port_rangeparameters .

7. How to optimize TIME_WAIT?

   Here are several ways to optimize TIME-WAIT, both of which have advantages and disadvantages:

   • Open net.ipv4.tcp_tw_reuse and net.ipv4.tcp_timestamps options;
   • net.ipv4.tcp_max_tw_buckets
   • SO_LINGER is used in the program, and the application is forced to use RST to close.

8. What is a semi-join queue

   After the server receives the SYN from the client for the first time, it will be in the SYN_RCVD state. At this time, the two parties have not fully established their connection. The server will put the request connection in this state in a queue. We call this queue Semi-join queue .

   Of course, there is also a full connection queue , that is, the three-way handshake has been completed, and those who have established a connection will be placed in the full connection queue. If the queue is full, packet loss may occur.

   Here is a question about the number of retransmissions of SYN-ACK: After the server sends the SYN-ACK packet, if it does not receive the client confirmation packet, the server will retransmit it for the first time. retransmission times. If the number of retransmissions exceeds the maximum number of retransmissions specified by the system, the system will delete the connection information from the semi-connection queue. Note that the waiting time for each retransmission is not necessarily the same, and generally increases exponentially, for example, the interval is 1s, 2s, 4s, 8s...

9. What are the common TCP connection states

   • CLOSED: initial state.
   • LISTEN: The server is in the listening state.
   • SYN_SEND: The client socket performs a CONNECT connection, sends a SYN packet, and enters this state.
   • SYN_RECV: The server enters this state after receiving the SYN packet and sending the server SYN packet.
   • ESTABLISH: Indicates that the connection is established. The client enters this state after sending the last ACK packet, and the server enters this state after receiving the ACK packet.
   • FIN_WAIT_1: The party that terminated the connection (usually the client) enters after sending a FIN message. Wait for the other party to FIN.
   • CLOSE_WAIT: (assuming the server) waits for the closing stage after receiving the client FIN packet. After receiving the other party's FIN packet, it is natural to reply to the ACK packet immediately, indicating that the disconnection request has been known. However, whether the party immediately disconnects (sends a FIN packet) depends on whether there is still data to be sent to the client. If so, it will be in this state before sending the FIN packet.
   • FIN_WAIT_2: At this time, it is in a semi-connected state, that is, one party requests to close the connection and waits for the other party to close. The client receives the ACK packet from the server, but does not immediately receive the FIN packet from the server, and enters the FIN_WAIT_2 state.
   • LAST_ACK: The server launches the last FIN packet, waits for the last client ACK response, and enters this state.
   • TIME_WAIT: The client receives the FIN packet from the server and immediately sends an ACK packet for final confirmation. The 2MSL time after that is called the TIME_WAIT state.

10. What information is in the TCP header

    • Sequence number (32bit): The byte number of the byte stream in the transmission direction. The initial sequence number will be set to a random initial value (ISN), and then each time data is sent, the sequence number value = ISN + the offset of the data in the entire byte stream. Assuming A -> B and ISN = 1024, the first segment of data 512 bytes has arrived at B, then the sequence number of the second segment of data is 1024 + 512. It is used to solve the problem of out-of-order network packets.
    • Confirmation number (32bit): The receiver's response to the sender's TCP segment, whose value is the received sequence number + 1.
    • Header Length (4bit): Indicates how many 4 bytes * header length, the maximum is 15, that is, 60 bytes.
    • Flag bit (6bit):
      • URG: indicates whether the urgent pointer is valid.
      • ACK: Mark whether the confirmation number is valid (confirmation message segment). Used to solve the problem of packet loss.
      • PSH: Prompt the receiver to read data from the buffer immediately.
      • RST: Indicates that the other party is required to re-establish the connection (reset segment).
      • SYN: Indicates a request to establish a connection (connection segment).
      • FIN: means to close the connection (disconnect the segment).
    • Window (16bit): Receive window. It is used to inform the other party (sender) how many bytes of data can be received in the buffer of this party. Used to solve flow control.
    • Checksum (16bit): The receiving end uses CRC to check whether the entire message segment is damaged.

11. What are RTO, RTT and timeout retransmission?

    • Timeout retransmission: If the sender does not receive a confirmed message for a long time after sending the message, it needs to resend the message. There are several possible situations:
      • The sent data failed to reach the receiving end, so the other party did not respond.
      • The receiving end receives the data, but the ACK packet is lost during the return process.
      • Receiver rejects or discards data.
    • RTO: The time between the last time data is sent, because no ACK response has been received for a long time, and the next retransmission. is the retransmission interval.
      • Usually the RTO of each retransmission is twice the previous retransmission interval, and the unit of measurement is usually RTT. Example: 1RTT, 2RTT, 4RTT, 8RTT...
      • Stop retransmission after the number of retransmissions reaches the upper limit.
    • RTT: The time interval between sending data and receiving the other party's response, that is, the round-trip time of the datagram in the network. The size is not stable.

12. TCP retransmission

    One of the ways TCP achieves reliable transmission is through sequence numbers and acknowledgment responses.

    In TCP, when the data from the sender arrives at the receiving host, the receiving host will return an acknowledgment message, indicating that the message has been received.

    Common retransmission mechanisms:

    • timeout retransmission
    • fast retransmit
    • SACK
    • D-SACK

    timeout retransmission

    One of the methods of the retransmission mechanism is to set a timer when sending data. When the specified time is exceeded and no ACKconfirmation data will be resent, which is what we often say Timeout retransmission .

    TCP will timeout and retransmit in the following two situations:

    • packet loss
    • Acknowledgment lost

    RTTIt refers to the difference between the time when the data is sent and the time when the confirmation is received , that is, the round-trip time of the packet.

    The timeout retransmission time is RTOrepresented by (Retransmission Timeout timeout retransmission time).

    • When the timeout time RTO is large , the retransmission will be slow, and it takes a long time to retransmit, which is inefficient and poor in performance;
    • When the timeout time RTO is small , it may lead to retransmission without loss, so the retransmission will be fast, which will increase network congestion, resulting in more timeouts, and more timeouts will lead to more retransmissions.

    According to the above two situations, we can know that the value of the timeout retransmission time RTO should be slightly greater than the value of the round-trip RTT of the message .

    fast retransmit

    TCP has another fast retransmit (Fast Retransmit) mechanism , which is not time-driven, but data-driven retransmission .

    The working mode of fast retransmission is that when three identical ACK packets are received, the lost packet segment will be retransmitted before the timer expires.

    The fast retransmission mechanism only solves one problem, that is, the timeout problem, but it still faces another problem. That is, when retransmitting, whether to retransmit one or retransmit all the problems.

    SACK method

    There is another way to implement the retransmission mechanism called: SACK(Selective Acknowledgment), selective acknowledgment .

    This method needs to add SACKsomething , which can send the information of the received data to the "sender" , so that the sender can know which data has been received and which data has not been received. Knowing this information, it is possible to retransmit only the lost data .

    Duplicate SACK

    Duplicate SACK is also called D-SACKthat it mainly uses SACK to tell the "sender" which data has been received repeatedly.

    D-SACKThere are several advantages:

    1. It can let the "sender" know whether the sent packet is lost, or the ACK packet responded by the receiver is lost;
    2. You can know whether the data packet of the "sender" is delayed by the network;
    3. You can know whether the data packet of the "sender" has been copied in the network;

    Under Linux, this function can be turned on/off by net.ipv4.tcp_dsackparameter (it is turned on by default after Linux 2.4).

13. TCP sliding window

    TCP uses a sliding window mechanism to implement flow control. Sliding window (Sliding window) is a flow control technique. In the early network communication, the communicating parties would not consider the congestion of the network and send data directly. Since everyone does not know the network congestion situation and sends data at the same time, the intermediate nodes block and drop packets, and no one can send data. There is a sliding window mechanism to solve this problem.

    In TCP, a sliding window is used for transmission control. The size of the sliding window means how much buffer the receiver still has for receiving data. The sender can determine how many bytes of data should be sent by the size of the sliding window. When the sliding window is 0, the sender generally cannot send datagrams, except for two cases. One case is that urgent data can be sent, for example, allowing the user to terminate the running process on the remote machine. Alternatively, the sender can send a 1-byte datagram to inform the receiver to re-declare the next byte it wants to receive and the sender's sliding window size.

14. TCP flow control

    TCP uses sliding windows to implement flow control. Flow control is to control the sending rate of the sender to ensure that the receiver has time to receive. The window field in the confirmation message sent by the receiver can be used to control the window size of the sender, thereby affecting the sending rate of the sender. Setting the window field to 0 means that the sender cannot send data.

    1. The purpose is that the receiver informs the sender of the maximum amount of data that it can receive through the TCP header window field, so as to solve the problem that the receiver cannot receive because the sending rate is too fast. So flow control is point-to-point control.
    2. TCP is a duplex protocol, and both parties can communicate at the same time, so the sender and the receiver each maintain a sending window and a receiving window.
       • Sending window: used to limit the size of data that the sender can send, where the size of the sending window is controlled by the window field in the TCP segment returned by the receiving end, and the receiving end informs the sender of its own buffer through this field (subject to system, hardware and other restrictions) size.
       • Receive window: used to mark the size of data that can be received.
    3. TCP is stream data, and the data stream sent out can be divided into the following four parts: sent and confirmed part | sent unacknowledged part | unsent but sendable part | unsendable part, where sending window = sent but not Confirmation part + unsent but sendable part. Received data flow can be divided into: received | not received but ready to receive | not received not ready to receive. Receive window = not received but ready to receive part.
    4. For the data in the sending window, the sending window is only moved when the ACK response of a certain segment of data sent by the receiving end is received, and the left edge is close to the data that has just been confirmed. The receiving window only moves the receiving window when data is received and the leftmost is continuous.

15. TCP congestion control

    The purpose of congestion control is to prevent the overload of network resources (routers, switches, etc.) caused by excessive data injection into the network. Because congestion control involves the overall network link, it belongs to global control. Congestion is controlled using congestion windows.

    Congestion control mainly consists of four algorithms: 1) slow start, 2) congestion avoidance, 3) congestion occurrence, and 4) fast recovery.

    Slow start algorithm – Slow Start

    The so-called slow start means that the TCP connection has just been established, and the speed is increased little by little to test the bearing capacity of the network, so as not to directly disturb the order of the network channel.

    Slow start algorithm:

    1. After the connection is established, the congestion window cwnd is initialized with a size of 1, indicating that data of an MSS size can be transmitted.
    2. Whenever an ACK is received, the size of cwnd is increased by one, which increases linearly.
    3. Whenever a round-trip delay time RTT (Round-Trip Time) passes, the size of cwnd is directly doubled, multiplied by 2, and exponentially increased.
    4. There is also a ssthresh (slow start threshold), which is an upper limit. When cwnd >= ssthresh, it will enter the "congestion avoidance algorithm"

    Congestion Avoidance Algorithm – Congestion Avoidance

    As mentioned above, when the congestion window size cwnd is greater than or equal to the slow start threshold ssthresh, it enters the congestion avoidance algorithm. The algorithm is as follows:

    1. An ACK is received, then cwnd = cwnd + 1 / cwnd
    2. Whenever a round-trip delay time RTT is passed, the size of cwnd is increased by one.

    After the slow start threshold, the congestion avoidance algorithm can avoid window congestion caused by too fast window growth, but slowly increase and adjust to the optimal value of the network.

    Algorithm when congestion occurs

    Generally speaking, TCP congestion control defaults that network packet loss is caused by network congestion, so the general TCP congestion control algorithm uses packet loss as a signal for the network to enter a congested state. There are two ways to judge packet loss, one is when the timeout retransmission RTO [Retransmission Timeout] times out, and the other is when three repeated ACKs are received.

    Timeout retransmission is an important mechanism for the TCP protocol to ensure data reliability. The principle is to start a timer after sending a piece of data. If the ACK message for sending the datagram is not received within a certain period of time, then the data is resent. until the sending is successful.

    But if the sender receives more than 3 repeated ACKs, TCP realizes that the data has been lost and needs to be retransmitted. This mechanism does not need to wait until the retransmission timer expires, so it is called fast retransmission. After fast retransmission, the slow start algorithm is not used, but the congestion avoidance algorithm is used, so this is also called the fast recovery algorithm.

    When the timeout retransmission RTO[Retransmission Timeout] times out, TCP will retransmit the data packet. TCP thinks this situation is worse and reacts more strongly:

    • Due to packet loss, set the slow start threshold ssthresh to half of the current cwnd, ie ssthresh = cwnd / 2.
    • cwnd reset to 1
    • Enter the slow start process

    The earliest TCP Tahoe algorithm only used the above-mentioned processing method, but because everything was restarted once a packet was lost, cwnd was reset to 1 again, which was not conducive to the stable transmission of network data.

    Therefore, the TCP Reno algorithm is optimized. When receiving three repeated confirmation ACKs, TCP starts the Fast Retransmit algorithm without waiting until the RTO times out before retransmitting:

    • The size of cwnd is reduced to half of the current size
    • ssthresh is set to the reduced cwnd size
    • Then enter the fast recovery algorithm Fast Recovery.

    Fast Recovery Algorithm – Fast Recovery

    TCP Tahoe is an early algorithm, so there is no fast recovery algorithm, but the Reno algorithm has it. Before entering fast recovery, cwnd and ssthresh have been changed to half of the original cwnd. The logic of the fast recovery algorithm is as follows:

    • cwnd = cwnd + 3 MSS, the reason for adding 3 MSS is because 3 repeated ACKs are received.
    • Retransmit the packets specified by DACKs.
    • If more DACKs are received, the size of cwnd is increased by one.
    • If a new ACK is received, indicating that the retransmitted packet was successful, exit the fast recovery algorithm. Set cwnd to ssthresh, then enter the congestion avoidance algorithm.

16. The difference between flow control and congestion control

    • Flow control belongs to the negotiation between the communication parties; congestion control involves the overall communication link.
    • Flow control requires both parties to maintain a sending window and a receiving window. For any party, the size of the receiving window is determined by itself, and the size of the sending window is determined by the window value in the TCP segment responded by the receiver; the size of the congestion window for congestion control changes Self-adaptive adjustment is made by tentatively sending a certain amount of data to detect network conditions.
    • Actual final sending window = min{flow control sending window, congestion window}.

17. How does the TCP protocol ensure reliable transmission

    • Confirmation and retransmission : The receiver will confirm when it receives the message, and the sender will retransmit if it does not receive the confirmation after sending for a period of time.
    • Data verification : The TCP packet header has a checksum, which is used to verify whether the packet is damaged.
    • Reorder out-of-order packets : Since TCP segments are transmitted as IP datagrams, and IP datagrams may arrive out of order, TCP segments may also arrive out of order. TCP will reorder the out-of-order data before handing it over to the application layer;
    • Flow control : When the receiver has no time to process the sender's data, it can prompt the sender to reduce the sending rate through the sliding window to prevent packet loss.
    • Congestion control : When the network is congested, the congestion window is used to reduce data transmission and prevent packet loss.
    • **Discard duplicate data:** For duplicate data, it can discard duplicate data;
    • **Response mechanism:** When TCP receives data from the other end of the TCP connection, it will send an acknowledgment. This acknowledgment is not sent immediately, usually delayed by a fraction of a second;

18. How to optimize TCP

    The strategy for improving TCP is described from three perspectives, namely:

    • Performance improvement of TCP three-way handshake;
    • TCP four-wave performance improvement;
    • Performance improvements for TCP data transfers;

    Performance improvement of TCP three-way handshake

    Client Optimization

    When the client initiates a SYN packet, it can tcp_syn_retriescontrol .

    Server-side optimization

    When the server SYN semi-join queue overflows, subsequent connections will be discarded. You can netstat -sobserve overflow of the semi-join queue. If the overflow of the SYN semi-join queue is serious, you can tcp_max_syn_backlog、somaxconn、backlogadjust the size of the SYN semi-join queue through parameters.

    The number of retransmissions of the server replying SYN+ACK is controlled by tcp_synack_retriesthe parameter . If you suffer from a SYN attack, you should set tcp_syncookiesthe parameter to 1, which means that the syncookie function is enabled only after the SYN queue is full, which can ensure the successful establishment of a normal connection.

    When the server receives the ACK returned by the client, it will move the connection into the accpet queue and wait for the accpet() function to be called to take out the connection.

    You can ss -lntview the length of the accept queue of the server process. If the accept queue overflows, the system will discard the ACK by default. If you can tcp_abort_on_overflowset it to 1, it means to use RST to notify the client that the connection establishment failed.

    If the accpet queue overflows seriously, you can increase the queue size through backlogthe parameters , and the accept queue length depends on min(backlog, somaxconn).somaxconn

    Bypass the three-way handshake

    The TCP Fast Open function can bypass the three-way handshake, reducing the HTTP request time by 1 RTT. This function can be tcp_fastopenenabled , and must be supported by both the server and the client.

    Performance improvement of TCP four wave

    Both the client and the server can actively disconnect. Usually, the party that closes the connection first is called the active party, and the party that closes the connection later is called the passive party.

    For the optimization of the TCP four-way wave, we need to adjust the system TCP kernel parameters according to the change of the four-way wave state of the active side and the passive side.

    [External link image transfer failed, the source site may have an anti-leeching mechanism, it is recommended to save the image and upload it directly (img-vIOGjqUw-1678081583394) (https://cdn.xiaolincoding.com/gh/xiaolincoder/ImageHost/%E8% AE%A1%E7%AE%97%E6%9C%BA%E7%BD%91%E7%BB%9C/TCP-%E5%8F%82%E6%95%B0/39.jpg)]

    Active Side Optimization

    If the party that initiated the FIN message to disconnect the connection fails to receive the ACK reply from the other party, it will retransmit the FIN message, and the number of retransmissions is determined by tcp_orphan_retriesthe parameter

    When the active party receives the ACK message, the connection enters the FIN_WAIT2 state. Depending on the closing method, the optimization method is also different:

    • If this is a connection closed by the close function, it is an orphan connection. If no FIN message from the other party is received within tcp_fin_timeoutseconds , the connection will be closed directly. At the same time, in order to deal with orphan connections occupying too many resources, tcp_max_orphansthe maximum number of orphan connections is defined, and the connection will be released directly when it exceeds.
    • On the contrary, the connection closed by the shutdown function is not limited by this parameter;

    When the active party receives the FIN message and returns an ACK, the active party's connection enters the TIME_WAIT state. This state will last for 1 minute. In order to prevent the TIME_WAIT state from occupying too many resources, a tcp_max_tw_bucketsmaximum number is defined, and the connection will be released directly when it exceeds.

    When there are too many TIME_WAIT states, you can tcp_tw_reusealso tcp_timestampsreuse the ports in TIME_WAIT state as new connections as clients by setting and to 1. Note that this parameter is only applicable to clients.

    Passive side optimization

    The response to the passively closed connection is very simple. After replying ACK, it enters the CLOSE_WAIT state, waiting for the process to call the close function to close the connection. Therefore, when there are a large number of connections in the CLOSE_WAIT state, the problem should be found in the application.

    After the passive party sends the FIN message, the connection enters the LAST_ACK state, and when the ACK is not received, the FIN message will be resent under the control of tcp_orphan_retriesthe parameters .

    Performance improvements for TCP data transfers

    [External link image transfer failed, the source site may have an anti-leeching mechanism, it is recommended to save the image and upload it directly (img-yvueg1Zy-1678081583395) (https://cdn.xiaolincoding.com/gh/xiaolincoder/ImageHost/%E8% AE%A1%E7%AE%97%E6%9C%BA%E7%BD%91%E7%BB%9C/TCP-%E5%8F%82%E6%95%B0/49.jpg)]

    The reliability of TCP is realized through the ACK confirmation message, and relies on the sliding window to improve the sending speed and take into account the processing ability of the receiver.

    However, the default maximum value of the sliding window is only 64 KB, which does not meet the requirements of today's high-speed network. In order to increase the sending speed, the upper limit of the sliding window must be tcp_window_scalingincreased up to 1GB.

    The sliding window defines the maximum number of bytes of flight packets in the network. When it exceeds the bandwidth-delay product, the network is overloaded and packet loss occurs. And when it is smaller than the bandwidth-delay product, the network bandwidth cannot be fully utilized. Therefore, the setting of the sliding window must refer to the bandwidth-delay product.

    The kernel buffer determines the upper limit of the sliding window, and the buffer can be divided into: send buffer tcp_wmem and receive buffer tcp_rmem.

    Linux will dynamically adjust the buffer, and we should set the upper limit of the buffer to the bandwidth-delay product. The adjustment function of the sending buffer is automatically enabled, while the receiving buffer needs to be enabled by setting tcp_moderate_rcvbuf to 1. Among them, the adjustment is based on the TCP memory range tcp_mem.

    However, it should be noted that if the socket in the program sets SO_SNDBUF and SO_RCVBUF, the dynamic adjustment function of the buffer will be turned off, so it is not recommended to set them in the program, but to let the kernel automatically adjust it.

19. UDP protocol

    Provides a connectionless , best-effort data transmission service ( the reliability of data transmission is not guaranteed ).

    features

    (1) UDP is a connectionless transport layer protocol;

    (2) UDP uses best-effort delivery and does not guarantee reliable delivery;

    (3) UDP is message-oriented, and the messages delivered by the application layer are not merged or split, and the boundaries of the original messages are retained;

    (4) UDP has no congestion control, so even if the network is congested, the sending rate will not be reduced;

    (5) UDP supports one-to-one, many-to-many interactive communication;

    (6) The header overhead of UDP is small, only 8 bytes.

20. How UDP achieves reliable transmission

    UDP is not a connection protocol, and has the advantages of less resource consumption and fast processing speed. Therefore, UDP is usually used when transmitting audio, video and ordinary data, because even if a small amount of packets are lost, it will not have a large impact on the acceptance result. Impact.

    The transport layer cannot guarantee the reliable transmission of data, it can only be realized through the application layer. The implementation method can refer to the tcp reliable transmission method, but the implementation is not at the transport layer, and the implementation is transferred to the application layer.

    The easiest way is to imitate the reliable transmission of TCP at the transport layer at the application layer. The following does not consider congestion handling, a simple design of reliable UDP.

    • 1. Add seq/ack mechanism to ensure that data is sent to the peer
    • 2. Add sending and receiving buffers, mainly for user timeout retransmission.
    • 3. Add timeout retransmission mechanism.

    Detailed description: When the sending end sends data, a random seq=x is generated, and then each piece is allocated seq according to the data size. After the data arrives at the receiving end, the receiving end puts it in the cache and sends an ack=x packet, indicating that the other party has received the data. After receiving the ack packet, the sender deletes the data corresponding to the buffer. When the time is up, the scheduled task checks whether the data needs to be retransmitted.

    At present, the following open source programs use udp to realize reliable data transmission. * RUDP, RTP, UDT* respectively .

21. The difference between TCP and UDP

    (1) TCP is reliable transmission, UDP is unreliable transmission;

    (2) TCP is connection-oriented, UDP is connectionless;

    (3) TCP transmits data in order, and UDP does not guarantee the order of data;

    (4) TCP does not save data boundaries, and UDP retains data boundaries;

    (5) TCP transmission speed is slower than UDP;

    (6) TCP has flow control and congestion control, UDP does not;

    (7) TCP is a heavyweight protocol, and UDP is a lightweight protocol;

    (8) The TCP header is 20 bytes longer, and the UDP header is 8 bytes shorter;

    TCP application scenarios:

    Scenarios with relatively low efficiency requirements but relatively high accuracy requirements. Because operations such as data confirmation, retransmission, and sorting are required during transmission, the efficiency is not as high as that of UDP. To give a few examples: file transfer (high accuracy and high requirements, but the speed can be relatively slow), accepting mail, remote login.

    UDP application scenarios:

    Scenes with relatively high efficiency requirements and relatively low accuracy requirements. To give a few examples: QQ chat, online video, VoIP (instant messaging, high speed requirements, but occasional intermittent is not a big problem, and the retransmission mechanism cannot be used here), broadcast communication (broadcast, multi broadcast)

    Common protocols based on TCP and UDP

    HTTP, HTTPS, FTP, TELNET, SMTP (Simple Mail Transfer Protocol) protocols are based on the reliable TCP protocol. DNS, DHCP, TFTP, SNMP (Simple Network Management Protocol), RIP based on unreliable UDP protocol

22. TCP sticky packet and unpacking

    TCP sticky packet

    TCP sticky packet refers to that several packets of data sent by the sender are glued into one packet when received by the receiver. From the receiving buffer, the head of the next packet of data is immediately followed by the tail of the previous packet of data.

    • Sticky packet problem caused by TCP connection multiplexing .
    • Because TCP uses the Nagle algorithm by default, this algorithm will cause sticky packets.
      • The next packet will only be sent if the previous packet is confirmed;
      • Collect multiple small packets and send them together when an acknowledgment arrives.
    • Sticky packet problem caused by too large data packet .
    • Flow control and congestion control may also cause sticky packets.
    • The receiver does not receive the packets in the buffer in time, causing multiple packets to be received

    Solved :

    1. It is caused by the Nagle algorithm problem, and the algorithm needs to be properly closed according to the application scenario
    2. Tail marker sequence. Packet boundaries are represented by special identifiers, such as \n\r, \t, or some hidden characters.
    3. Header tags are received in steps. Add the data length to the header of the TCP message.
    4. The application layer sends data at a fixed length .
    5. Special character control;

23. Difference Between URI and URL

    URL, that is, Uniform Resource Locator (Uniform Resource Locator), URL is actually the URL we usually enter when surfing the Internet. It identifies an Internet resource and specifies the method to operate it or obtain the resource. For example, the URL https://leetcode-cn.com/problemset/all/ identifies a specific resource and indicates that a certain form of the resource can be obtained from the corresponding location through the HTTP protocol.

    A URI is a Uniform Resource Identifier, and a URL is a subset of a URI, both of which define what a resource is, and a URL also defines how the resource can be accessed. URI is a semantic abstract concept, which can be absolute or relative, while URL must provide enough information to locate and is absolute. Simply put, as long as a resource can be uniquely identified, it is a URI, and on the basis of the URI, a resource access method is given as a URL.