In-depth analysis of the secret behind the Http request (on)

Preface

We have used the browser for so long, but do we know what happened behind the input of a URL to the interface response? Why do we enter www.baidu.com and it knows that we are looking for Baidu's address? We all know that the http request is triggered after we press Enter, but what is the http request and why can it request our data? As a programmer, exploring and understanding the online world we live in, I think it’s still very good

Due to the limited space, it is impossible to clarify all the details. Therefore, try to analyze the content that we usually come into contact with. Some things that do not need to be understood are briefly explained. At the same time, according to the different purpose of each person to obtain information, I will first briefly summarize the entire process of Http request, and then conduct specific analysis and elaboration. Everyone, according to your own time and purpose, whether you need to read the whole article, if it is useful to you, remember to click three consecutive yo~

Overview

From the time we enter the URL or click on a link, to finally receive the message returned by the web server and display it is roughly divided into the following processes.

1. Parse the HTTP request ;
2. Create HTTP connection ;
3. Request sending and response .
4. Disconnect .

Let's start with a detailed explanation of what happened in each process. This article focuses on the analysis of steps 1, 2 and steps 3 and 4 in the next article.

Before explaining, let me talk about an example around us.

We want to send a letter to someone. What do we need to do? First of all, we must know where to send it and who to send it to. This is the most basic problem, but this is actually a problem. How do we determine this address? Can we ensure that everyone knows where this is? Then we definitely need a standardized address , such as XXX District, XXX City, XXX Province, etc., but what if we only know a certain macro location ? For example, XXX company, but don't know the specific house number, what to do? Then when we send a letter, we either clarify the specific street number under Baidu, and then write it, or the post office staff will automatically find out which street and number when we enter the address for us. After clarifying the location and addressee, we can write the content of the letter , then put it in an envelope , and finally submit it to the post office . At this point, we have finished sending the letter, just wait for the reply.

Having said so much, you may be a little confused. Is there any connection between this and our network request sending? Let me speak slowly.

One, parse the HTTP request

Before writing this article, I read a lot of articles describing the http request, the first step is to establish a TCP connection. But in fact, we have overlooked a question of principle. We want to build a road. We must know where we built this road, right? That is to clarify our destination. Therefore, the first thing to do after an Http request occurs is to parse the request link, and then analyze our purpose information, such as the ip address, the file path to be accessed, and the port of the server.

Http request link resolution

Before starting the explanation, let's first understand what the network links we are in contact with every day are like. The following uses an http link as an example:

http://www.yoka.com/dna/222/192/index.html

The above link mainly contains these pieces of content: protocol type, domain name/IP address, port number, file path. The above link can be split as follows:

协议类型：http
域名/IP地址：www.yoka.com
端口号：web服务器默认80端口，可省略不写
文件路径：/dna/222/192/index.html

agreement type

There are many types of protocols supported by the browser: http, https, ftp, news, mailto, etc. News and mailto are rarely used, so I won’t talk about it here. The ftp protocol is a file transfer protocol, which is mainly used when uploading and downloading files, and it is not elaborated too much. Mainly talk about the difference between http and https.

Http protocol : Hypertext transfer protocol, a stateless, application layer protocol based on request and response, often based on TCP/IP protocol to transmit data, the most widely used network protocol on the Internet, all WWW files must Follow this standard. At present, the mainstream version of the http protocol is http2.0, which supports multiplexing, server push, header compression, binary protocol, etc. Multiplexing refers to the initiation of multiple requests through a single HTTP/2 connection request- In response to a message, multiple request streams share a TCP connection , so that multiple stays in parallel are achieved instead of relying on the establishment of multiple TCP connections.

Https protocol : HTTPS is a transmission protocol for secure communication through a computer network. It communicates via HTTP and uses SSL/TLS to establish a full channel to encrypt data packets. The incoming content is encrypted and mixed encryption technology is used. The middleman cannot directly view the plaintext content. At the same time, it supports the verification of the server's identity. The client is authenticated through the certificate to access its own server, which can also effectively ensure the integrity of the data.
Reference: HTTP and HTTPS protocols, just read one article

Domain name/IP address & DNS

Under normal circumstances, most of the web links are accessed by domain names. This is because the domain name can represent the characteristics of the corresponding website, such as:, www.baidu.comand if you use the ip address to display, a string of numbers is not only inconvenient to remember, but also has no special features. . However, the address of our web server cannot be found correctly through the domain name, which requires the use of DNS (Domain Name Service System) .

Every one of us will have a DNS client, called resolver for short, on our computer . Querying the IP address corresponding to the domain name is the process of domain name resolution. When we initiate an http request, the final connection processing will actually be executed by our operating system. At this time, the operating system will initiate a domain name resolution request to the DNS server through the resolver, and the DNS server will return the corresponding IP address after analyzing the domain name. The resolver will take out the IP address and write it into the memory address specified by the browser. But to access the web server we need to know its IP address, how do we know the IP address of the DNS server? This is actually matched on our computer itself, as follows:

At this point, we can get the IP address of the web server normally. But why can the IP address find our corresponding server address? This again requires an understanding of the IP address. All our computers are assigned an IP address, and the ip address is mainly composed of two parts: network number and host number . The network number is a subnet number of the network where the host is located, and the host number corresponds to our computer. An IP address is a string of 32-bit numbers, 8 bits are divided into four groups, and then divided by dots, the proportion of the host number and network number in the IP address is not fixed, so a subnet is required Mask to identify. details as follows:

1. IP address subject identification method

10.11.12.13

2. Method of identifying the subnet mask in the same format as the IP address

//后者为子网掩码，子网掩码为1的部分（十进制转比特）标识网络号，为0标识主机号
//所以下面的ip地址，网络号为10.11.12，主机号为13
10.11.12.13/255.255.255.0   

It is worth mentioning that the bits in the back part of the host are all 0 or all one means two special meanings, 0 means the entire subnet instead of a certain device in the subnet, and 1 means all of the subnet The device sends a packet, that is, broadcast.

At this point, we have obtained the ip address corresponding to our domain name, and we also know the destination of our request this time, and the rest is to bridge the water and create a communicable connection between the sender and the server.

Two, create an HTTP connection

When we already know the ip address of the server, our final request is actually realized by the bottom layer of the operating system. The protocol stack inside the operating system is used here . The protocol stack is mainly divided into two when processing network requests. Steps: Create a socket and connect to the server . Here first put a rough operating system communication hierarchy diagram.

Create socket

We want to create a connection pipeline. The most important thing is the data entry and exit at both ends of the pipeline. These entrances and exits are called sockets. The browser creates sockets by calling the socket program in the Socket library. After the sockets are created, they can Connect the pipes. The specific process is as follows:

1. The server starts and creates the socket, enters the waiting state, and waits for the client to connect to the pipeline.
2. The client creates a socket, then extends the pipe, and finally the pipe connects to the socket on the server side.
3. Send and receive data.
4. The client or server disconnects, and both parties delete the socket.

What exactly is a socket? It can be understood that they are just a string of data information that exists in the memory space of the operating system . When the socket is created, the system will open up a memory space for a period of time and write the initial state of the socket (note that the connection is at this time The information IP address, port number, etc. have not been passed into the socket, but simply created an initial state), and a descriptor is assigned to each socket to distinguish between different sockets. This information is used to control communication operations , mainly including the IP address, port number, and status of the communication operation of the communication partner . Control information can be divided into two categories. One is that when a connection request is created, these control information will be placed in the header of the connection information (TCP header), and each request needs to bring this control information. The details are as follows: The

main things you need to know above are the following:

• 32-bit acknowledgment number (ACK number): The receiver uses the number of bytes to inform the sender that all data has been received.
• 32-bit serial number: The sender informs the receiver that the data sent by the network packet is equivalent to the first few bytes of all data.
• ACK: Identifies that the received data sequence number field is valid, generally indicating that it has been accepted by the receiver
• SYN: The sender and receiver mutually confirm the serial number, indicating the connection operation.
• FIN: indicates disconnection. When
  actually sending, it is actually like the following:
  

There is another type of information stored in the socket to control the operation of the protocol stack.
When the protocol stack creates a connection, it needs to operate based on these control information. In the window system, you can actually view the contents of the socket through commands. The specific commands are:netstat -ano

connect to the server

After the socket is created, the browser will call the connect application in the Socket library to connect the local socket and the server socket. When calling connect, the server's ip address and port number will be transmitted together. Into. The details are similar to the following:

connect(<描述符>, <服务器的端口和ip>,...)

The connection is mainly to do the following things:

1. Tell the server's IP address and port number to the protocol stack;
2. The client communicates a request to start communication to the server;
3. Create a memory space and buffer for temporarily storing data sent and received;

Among them, the client communicates the request to start communication to the server first by creating a header representing the connection control information at the TCP module, which is the content of the table above. Through TCP header information, you can find the port numbers of the sender and receiver, and then pass them down to the IP module. Add the IP header information in front of the request and perform the return operation of the network packet. Finally, the network report will be To reach the server through the network, the server analyzes the IP and TCP modules layer by layer, finds the corresponding socket information, and then matches the sockets in the waiting state in the server to find the same port number. At this point, the first handshake of the three-way handshake commonly referred to in network communication is started . For the remaining two handshake, the data processing process is similar to the first one, so I will not elaborate on it. To put it briefly here, the server returns the received response. After the client receives the response returned by the server, the connection is successful. Finally, the client sends a feedback request confirming the reception to the server according to the returned response request. To tell the server that the response packet just now has been received. The connection here is truly complete!

As for how the client knows that the connection is successful, we need to use the request header that we will include in every request, and the control identifiers ACK and SYN, etc., are roughly as follows:

At this point, the client has successfully connected to the server. The simple summary is: the browser first parses our request url (usually http://ip address/accessed file path), according to the content of the url (if Is a domain name, such as www.baidu.com, you need to obtain the ip address corresponding to the domain name from the DNS server) call the application of the operating system (socket and connect in the Socket library) to create a socket, and pass The protocol stack goes down layer by layer to complete the TCP connection.

Originality is not easy, welcome one-click triple connection~

Reference materials:

"How the Internet is Connected"

TCP header format