1. Introduction to HTTP protocol
In the real network environment, the structure transmission of TCP/IP five-layer network transmission model is adopted.
Physical Layer -> Data Link Layer -> Network Layer -> Transport Layer -> Application Layer
1. Application layer: The application layer is the topmost layer of the model, which provides a way for users to communicate with the network. The application layer includes various applications, including web browsers, emails, file transfer protocols, etc. These applications can transfer data by constructing datagrams in the application layer and calling the transport layer.
2. Transport layer: manages data transmission between two nodes and is responsible for reliable transmission. The transport layer does not pay attention to the path of intermediate transmission, but only pays attention to the starting point and end point and ensures that the data is reliably transmitted to the target address.
3. Network layer: address management, recording the ip of the source and destination host and the ip of the destination host, routing and forwarding: responsible for selecting a reliable and efficient transmission path for two nodes - path planning.
4. Data link layer: Responsible for the transmission of data frames between connected devices, mainly focusing on the transmission between two adjacent device nodes, such as data transmission between connected switches and network cards through network cables/optical fibers/network interfaces.
5. Physical layer: infrastructure equipment for network communication: a series of transmission media such as network and optical fiber.
The upper-layer protocol calls the lower-layer protocol, and the lower-layer protocol provides services for the upper-layer protocol, and cross-layer calls are not allowed.
Such data is transmitted in the network according to this attribute protocol format.
The HTTP protocol belongs to the application layer of the TCP/IP five-layer network protocol model
There are currently three major versions of the HTTP protocol:
Both HTTP/1 and HTTP/2 transmit data based on the TCP Transmission Control Protocol. The latest version of HTTP/3 is based on the UDP protocol. Most of the current version used to transmit data on the Internet is HTTP / 1.1
What exactly is the HTTP protocol? ——What is the application layer protocol?
The application layer protocol can be understood as a format for organizing data, which requires the sender and receiver to agree on the corresponding format.
For example (hypothesis): QQ application layer datagram
Application layer protocols, in most scenarios, are designed by programmers themselves according to actual demand scenarios, but there is a scenario where in the circle of programmers, the level of expertise varies, and some application layer protocols designed by programmers are very It is easy to use. Some programmers design application layer protocols with many bugs. In this case, some big guys have invented some useful protocols. Other programmers can refer to the big guys’ designed protocols to transmit data. While using the Big Brother protocol, you can also allow the protocol to transmit various custom data information according to actual needs. One of the most famous is the HTTP protocol, which describes how the application layer organizes data.
Protocol format - how the data is organized.
For example: UDP transport protocol: header (source port, destination port, length, checksum) + payload
TCP transport protocol:
UDP / TCP / IP protocols are all "binary" protocols, and data needs to be understood from a binary perspective.
The HTTP protocol is a text format protocol (converted to binary transmission), so we only need to understand the format of the text.
How to see the HTTP message format?
We can use some "packet capture tools" to obtain device interaction data, and can capture data packets of the http protocol.
If you only want to capture packets for HTTP/HTTPS protocol data, you can use the tool Fiddler . If you need to capture UDP / TCP / IP / mac addresses... you can use the tool wireshark .
Fiddler (download address: https://www.telerik.com/fiddler/)
The principle of the packet capture tool: when data is transmitted in the network, it will pass through many transit devices, such as: router addressing, and the data will be transmitted through these devices. The packet capture tool can be considered as a transit device. When the data passes through this device, it can naturally For "capture", the data transmission is transmitted according to the standard protocol, and it can be parsed according to a certain format. Of course, https will encrypt the data for transmission. Although it is also possible to capture the data packets of https, if there is no certain "means" "I can't understand the data (analysis in the next article), only the two ends of the data transmission are analyzed through the agreed ciphertext.
2. HTTP protocol request
HTTP (Hypertext Transfer Protocol) is an application layer protocol based on request-response mode, which is used to transmit data in the Web. An HTTP request is a request sent by the client to the server, and the request includes three parts: the request line, the request header, and the request body.
For example: use the Sogou search engine to search for a "puppy", then the browser will automatically package the search record into an http data packet, and send it to the Sogou server as a request. After parsing, the keyword "puppy" can be obtained, and the server can use the puppy as a keyword to perform fuzzy matching on the database. After getting the data from the database, it can continue processing according to the obtained data, such as building a page Ah, finally pack the data into http format and return it to the client as a response, so we can see the information about the puppy on the page.
ask:
The data after query = is the keyword "puppy" we requested from the server. Here we just escape these characters. Essentially, we take out the Ascll code of the character to be escaped and express it in hexadecimal. At the same time, adding % is the processing of each byte, so it is such a result (details are described below).
The http request is divided into four parts: 1. The first line of the request, 2. The header, 3. A blank line, 4. The body of the request, depending on the request format, there may not be any.
2.1 The first line of the request
The first line contains three parts:
2.1.1 HTTP method, method
HTTP defines methods (also known as actions or verbs) that describe the type of communication between a client and a web server.
Here are some common methods.
The method in the request , for example, GET request: want to get some resources from the server, it is natural to imagine the scene of the search box.
There are many methods of the HTTP protocol, the most commonly used methods are GET and POST.
2.1.2 URL (Uniform Resource Locator)
URL stands for Uniform Resource Locator, which describes in detail where the resources on the network to be accessed are.
It is necessary to clarify who the resource host is, and also to specify the resources on the host. When opening a web page to search through a browser, fill in the "URL" in the address bar.
1. Protocol scheme name
Describe the format of the current URL under the organization of that protocol. By specifying the protocol, the user's browser or other applications can correctly open and load resources according to the protocol. For example, if the protocol scheme of the URL is HTTP, the browser will use the HTTP protocol to establish a connection with the server, obtain resources and present them on the user's screen. If the protocol scheme is FTP, the browser will use the FTP protocol to access resources. The protocol scheme name in the URL is very important to access and load the resource correctly.
2. Login information
In the early http format, the user name and password will be displayed in the URL for identity verification, but now the https protocol is basically used to encrypt the transmitted data with ssl.
3. Server address
To transfer resources in the network, the IP address is really indispensable. The IP address is the unique identity of the device in the network. To establish a connection between two devices, they need to find each other in the vast network world. Under normal circumstances, what we see are domain names, such as: www.baidu.com, which is the IP address of Baidu server, which involves DNS domain name resolution, which essentially combines the domain name (certain format) with the IP address (192.168.255.255 ) to form a mapping, because using a domain name is friendly to users and easy to remember. From a certain point of view, it is more beautiful than the string of IP numbers. The domain name will eventually be translated into an IP address by DNS. .
4. Port number
The port number indicates the application program on the current device, for example: the port number of the http server: 80, the http server is essentially an application program for code editing, which is convenient for sending and parsing datagrams in http format, and tomcat is more commonly used in Java , he is an http server written in Java code, the default binding port number is 8080, and a set of APIs (interfaces) for operating the http protocol——servlet
Every networked application is automatically bound to a port number when it is started, which is used as the identity of the application on the device. For example, messages from qq will not be received by WeChat, and messages from WeChat will not be received by WeChat. qq, but the messages of these two applications must be received by the device we use, and then the port number can be obtained according to the analysis, and then we can further decide which application to analyze the data.
The port number is omitted in most cases, because the browser will automatically assign a default port according to the protocol scheme name . For URLs that start with http, port 80 is used as the default value by default, and for URLs that start with https, the default is Use port 443.
5. Hierarchical file paths
As an application, the server can store resources in hard disks or databases. For example, when the Baidu browser searches for "puppy", the Baidu server will search for relevant information about puppies locally, or from the hard disk? - File path: D:/Pictures/Puppy.jpg, or from the database? Database/DataTable/Field(Puppy).
The above IP address + port + hierarchical file path can actually describe a specific resource on the network.
6. Query String
In essence, it is a browser/client, and the customized information passed to the server is often used to describe the further resources that you want to obtain. For example: in the above example, if you search for a puppy, the puppy will be used as a query keyword, and the server will decide to use it Those data are responded to the client, and the content of the query string is the structure of key-value pairs——key : value For example: name = Zhang San, and the corresponding value can be obtained according to the key. The structure of these key-value pairs is completely programmed Custom decisions.
For example: the login interface will always consist of: user account, password
Programmers can set a set of key-value pairs for this login interface, username = ?, password = ?
After the user enters the data, the program can obtain the user data to construct the key-value pair. There is a way to obtain data on the server side, which is to obtain the value "Zhang San" through the hard-coded key value (key) username. password = 123456.
The content of the query string is a structure of key-value pairs. Keys and values are separated by "=", key-value pairs are separated by "&", and query strings and paths are separated by "?".
7. Fragment identifier
It is used to identify a specific location or fragment in a resource, usually used inside a web page, and can control the browser to scroll to the relevant location.
URL summary:
According to the standard URL structure, it is indeed more complicated, but there are four parts that are most closely related to program development:
1. IP address/domain name
2. Port number (usually not displayed)
3. Paths with hierarchy
4. queryString (query string)
1~3 describe the specific resources in the network, and 4 describes further requirements based on this resource
When queryString contains special characters, such as: / , : , ? , &, = These characters have special meanings in the URL, so special characters need to be escaped to avoid server parsing errors. We can use the encoding method of url encode in the process of escaping. Use url decode for the query string during parsing.
URLEncode is an encoding method for encoding special characters in URLs. There are some special characters in the URL address, such as "space", "&", "#", etc. These special characters need to be encoded during the submission process, otherwise the URL address may not be parsed correctly. After URLEncode is used to encode the URL, special characters can be replaced with "%" plus the hexadecimal representation of its corresponding ASCII code.
For example, for the character string "hello world", the result of URLEncode encoding is "hello%20world", where "%20" means "0x20" in hexadecimal ASCII code of a space.
For the GET method request of http, in fact, the queryString in the URL is used to send some information to the server, so it is necessary to have the awareness of escaping special characters during development.
2.1.3 Version number
The version number of the HTTP protocol appears in the request and response, and is used to indicate the version of the HTTP protocol used by the request or response. This is because different HTTP protocol versions adopt different protocol specifications, so the formats, supported methods, and status codes of request and response headers may be different. Therefore, specifying the HTTP protocol version can ensure that the HTTP protocol used by the client and server The versions are consistent to avoid errors caused by different versions. In addition, the protocol version used can also be easily identified by including the protocol version number in the request and response, which facilitates the maintenance and evolution of the protocol.
In HTTP/1.1, the HTTP version number information is included in both request and response headers.
To give a simple example, there are three versions of the HTTP protocol, HTTP1 and HTTP2 are implemented based on the TCP protocol, and HTTP3 is implemented based on the TCP protocol, so it is necessary to include the version number in an http datagram
2.2 Request header (header)
The request header contains many lines, each line is a key-value pair, use ":" between key and value
The request header of the HTTP protocol is a key-value pair structure sent by the client to the server. It contains some relevant information about the request sent by the client, such as the request method, the requested resource address, the HTTP version, the time of the request, and the request encoding method, etc. These request header information can help the server better understand the intent and needs of the client's request , and the corresponding reply will be more accurate and efficient. The number of key-value pairs here is not fixed, and different key-value pairs represent different meanings.
2.2.1 Key-value pairs in the common header section
The following key-value pairs can refer to the picture captured by the GET or POST request in the context of the blogger.
Host : www.baidu.com Host description IP address + port number
The domain name is converted into an IP address through DNS resolution, and the IP and the domain name are in a mapping relationship.
Content-Length: Describes the length of the data in the body of the text. According to this information, you can accurately read the complete data from the body. It involves the processing of IO streams. It is transmitted using the TCP protocol and is based on byte streams. based on.
TCP is a byte stream-oriented protocol that solves the problem of sticky packets -> reasonably design the application layer protocol to clarify the boundaries between packets.
use separator
use length
If there are currently several GET requests in the TCP receiving buffer,
When the application reads the request, a blank line is used as the delimiter.
If there are currently several POST requests, it has reached the TCP buffer~~
At this time, there is still a body after the blank line ~~ After the application reads the blank line, it needs to continue reading data of several lengths according to the length indicated by Content-Length ~~
Content-Type: Indicates the data format in the body of the request, commonly used in json format, or the front-end form form submission, you can also specify the character set at the same time: UTF-8
User-Agent( UA)：
Indicates what kind of device the current user uses to surf the Internet, mainly describing the information of the operating system and browser
The purpose of setting UA in the early website development is to solve the compatibility problem of the page. After the client sends a request, the server decides what kind of page to return according to different operating systems and different browser versions——HTML + CSS + JavaScript, such as size, interface layout.
Today, with the rapid development of the Internet, the functional differences between mainstream browsers are very small. Ten years ago, the compatibility of browsers was still a big issue for front-end development. Now front-end development still needs to consider compatibility issues, but compared to before It is indeed a lot more convenient-responsive layout in front-end development, UA has a new mission now, that is, it can be used to distinguish between PC and mobile. The page effects displayed by the browsers on these two devices are Obviously, there is a way that the front end will make two pages, judge the device according to the requested UA, and then decide to respond to that page.
Through Content-Type: the browser can determine which format to use to identify the data. If it is the front-end code, it can directly render the page and display it on the user interface.
Referer: Describes that the current page is redirected from a page
Referer: This key-value pair does not necessarily exist. If you directly enter the address from the address bar, or click on the favorites, there will be no Referer at this time.
If you search for "puppy" in Sogou's search engine , the referer field in the request at this time can perfectly show that the request comes from Sogou
What is the use of this key-value pair? There is a very commonly used function, the advertising system, which is billed according to clicks. There are advertising links on the browsing page. After clicking, a request will be triggered, and the server can identify it according to the Referer key-value pair received from the request. Incoming requests, so as to achieve the counting function. For example, on the Sogou platform, how many users have seen and clicked on the advertisement I placed, and after the statistics are calculated, I need to pay the Sogou platform~
Of course, is it possible that the http request was intercepted by others and tampered with the content in the Referer? Data is encrypted, can be intercepted, and is almost impossible to tamper with.
Cookie is a persistent data storage mechanism provided by the browser to the page . For the security of user data, the browser does not allow the front-end code to access the data on the user's device by default, preventing malicious code, crawling user data, or implanting viruses , Malicious deletion of data, etc., but sometimes it is necessary to store some data persistently on the page to facilitate future access to the webpage, such as: account information, careful friends can find that after logging in to some websites once, the next visit does not need to log in, directly Just enter the homepage, in fact, this verification process has always existed, but the verification request is sent in a place where the user cannot see, which saves the user from the complicated login and improves the user experience.
First of all, different browsers have different cookies. Cookies will allocate a space on the hard disk for each domain name (for example: www.baidu.com). It has a certain isolation to ensure that the code can only process the data in the cookie.
Where does the cookie data come from? When the user logs in to the webpage and sends a request, the website server can get the user's account information. First, it will determine whether the account exists, and secondly, verify the correctness of the account password. After the verification is successful, you can give some feedback The user's private information, such as avatar, nickname, detailed information, history, etc. Behind the account information is a key-value pair, the account information is the key, and other user-related information is the value. This key-value pair is session (session), and the server manages many sessions.
The first login to the server will create a new session based on the account, key-value pair, account information is the key, and then the server will return the account information to the client, and the client will store the data in the cookie. From then on, the user through the domain name (for example Baidu) will send requests with account information, and the server can directly get the user account information for automatic verification, eliminating the need for user login verification every time. Through the session (session), you can get the user's personal information.
2.3 Request Body
The request body (body) is the location where the HTTP protocol uses the POST method to transfer data . If the GET method is used to transfer data, you only need to use the queryString (query string) in the URL , so the request body (body) is optional. Not necessarily.
Note: There is a blank line between the request header (header) and the request body (body), and the blank line here is equivalent to the end mark of the header.
2.4 Classic interview questions: talk about the difference between the GET method and the POST method
1. Priority conclusion: There is no essential difference between GET and POST
- From the semantics of words, the GET method is usually used to fetch data, and the POST method is used to upload/submit data, but in fact GET is often used to upload data, and POST is also used to obtain data.
- The scene that GET can use can also be replaced by POST, and the scene used by POST can also replace GET.
2. The difference in details
- Usually, a GET request has no body (text) part. GET transmits data to the server through the queryString in the URL. It is a key-value pair structure. The key and value are linked by =, and each key-value pair is Use & between the connection,
- Normally, a POST request has a body (text) part, and POST passes data to the server through the body, but the queryString (query string) is not included in the POST request URL
- This is just the usual case - according to the standard, it is not a mandatory difference, but just a customary usage. As an application layer protocol, we can customize many places. You can make the GET request bring a body request, or let the POST use queryString, or even design an application layer protocol according to actual needs.
3. GET requests are generally idempotent, and POST requests are generally not idempotent (not mandatory, idempotent or not is just a suggestion)
Idempotent:
Every time the same input, the output result is determined, what does it mean? GET requests data, and the server responds. If I request the same data from the server every time, at this time, from the perspective of the entire network transmission, it will undoubtedly bring invalid overhead. Therefore, the request sent by GET is then given by the server. The response can be cached, for example: I enter a dog in the search box, then the server returns the information about the dog to the browser, and the browser displays the results on the page, and the information about the dog is cached by the browser at this time, Next time I will continue to search for puppies in the search box, this time the result will be directly taken from the cache and displayed on the page, which will avoid the overhead of network transmission, then if I just want him to get it from the server every time For the latest data, what should I do? Either re-enter the webpage, or press Ctrl + F5 to force the browser to refresh-get the data directly from the server.
Not idempotent:
For the same input every time, the input result obtained is uncertain, and the data is directly obtained from the server every time.
Expand knowledge:
There is no upper limit to the length of the data passed by the GET method, and there is no upper limit to the data passed by the POST method.
The official HTTP standard document clearly states that there is no limit to the length of the URL, and the GET method transmits data through the queryString in the URL. Of course, this is based on the standard of the HTTP protocol.
If you use GET to transmit a large amount of data , it will inevitably lead to a very long URL, and the entire address bar will be very unsightly.
If POST is used to transmit data , the user cannot see the data in the body on the interface, that is to say, no matter what data is transmitted in the body, the impact on the user is minimal.
3. HTTP response
3.1 First line
The first line of the HTTP protocol response contains three parts:
1. Version number: HTTP/1.1, HTTP/2, HTTP/3.0 ... describe the version of the protocol
2. Status code: (this is unknown to ordinary people) indicates the result of the requested page, whether it was successful, failed, or some other reason...
3. Description of the status code: a paragraph of English words, which simply describes the status to the user.
3.1.1 Status code and its description
200 OK
The browser gets the desired information
302 Move temporarily (temporarily redirected)
Redirection describes where to jump to next. It is often used in the login interface to automatically jump to the home page after successful login.
The header part of the response message will contain a Location field, indicating which page to jump to
Understanding redirection:
It is equivalent to the "call forwarding" function in mobile phone number communication. For example, I changed the number, but it was too late to remind the friends in my address book. In order to avoid, my friends couldn't reach me when they called me, so I went to the business hall to handle a call Transfer, when my friend calls my old number, it will be automatically transferred to my new mobile number.
301 Moved Permanently (permanent redirection)
When the browser receives this response, subsequent requests will be automatically changed to a new jump address, which is quite extreme. 301 also uses the Location field to indicate the new address to be redirected to.
404 not Found
After the client sends a request to the server, the server cannot parse the request, or does not receive the request (URL input error), most of the time the resource you want to not exist.
403 forbidden
Indicates that the resource you want to access exists, but you do not have permission to access it. This is a sad story~~
405 Method Not Allowed
We have already learned the methods supported in HTTP, including GET, POST, PUT, DELETE, etc.
Assuming that the client uses the queryString in the URL of the GET method to transmit data, if the server can only parse the HTTP datagram of the POST method and get the data from the body, it will feel a bit pretentious. The server does not support parsing the datagram . So it will respond with a 405 status code. This situation is rare in reality. Generally, the front and back ends have agreed on the communication method, and will go through a series of debugging and testing, but if we write it ourselves As for the website, such code may be written if one is not careful.
Reason: For example, try to use GET to access other people's servers, but maybe they only support POST, so return 405
500 internal server Error
There is a problem with the server itself, which means that there is a bug, which causes the server to crash abnormally and cannot continue to run correctly. It is also rare in reality. It is easy to see when you write the code yourself.
504 Gateway Timeout
When the server load is relatively heavy, it takes a long time for the server to process a single request, which may lead to a timeout. A simple understanding is that multiple clients access the server at the same time, causing the server to handle the overload situation (concurrent execution, knowledge related to multi-threading), which is also rare in reality. When the server processes requests concurrently, it is easy to be overloaded and the server is busy.
Status code summary
3.2 Response header (header)
It is also a key-value pair structure, each key-value pair occupies one line, and each key and value is separated by: space, the number of key-value pairs in the response header is also uncertain, and different key-value pairs represent different meaning. You can refer to the response header section of the request.
A blank line means split header and body
3.2 Response body (body)
The response body is the specific data returned by the server to the client, and the specific format of the body depends on the Content-Type field
1. text/html : Indicates that a webpage text is returned, and the browser can parse it into a webpage
2. text/css : Describes the front-end style files
3. application/javascript: Describes the actions of the webpage~ Front-end three-piece set
4. application/ json :
JSON format refers to JavaScript Object Notation (JavaScript Object Notation), which is a lightweight data format for data exchange and a key-value pair structure. Isn’t there a saying that front-end + json + back-end = full stack, enough to see the status of data in json format in actual development~
The basic JSON format structure is as follows:
{
"key": "value",
"key2": "value2",
"key3": {
"key4": "value4",
"key5": "value5"
}
}
It can contain data types such as objects, arrays, strings, numbers, Booleans, etc. Example:
{
"name": "John",
"age": 30,
"city": "New York",
"pets": [
{
"name": "Fluffy",
"species": "cat"
},
{
"name": "Fido",
"species": "dog"
}
],
"isMarried": true
}When the hardships are all over, I will tell you the way to come