"How the Internet is Connected" Notes Collection

Recently, I have encountered some network-related terms like proxy, gateway, and IPV4 in many aspects, but I don’t know what they mean, so I created this blog, mainly to record learning. Some knowledge points in the book "How are Networks Connected" are very suitable for beginners who are new to the Internet. They can understand some of the basic knowledge, at least they will not be as confused as I was before
( We will continue to update the content in the future, as well as add red highlights and add some expanded knowledge and hyperlinks)

1.1 Generate HTTP request message

1.1.1 The journey of exploration begins by entering the URL

First of all, we need to know the website address we enter in the browser, which is called URL to be precise. In addition to the common HTTP (Hypertext Transfer Protocol), the following are examples of URLs of other protocols:

1. HTTPS (Hypertext Transfer Protocol Secure): A version of the HTTP protocol used to transfer data securely.
2. FTP (File Transfer Protocol): A protocol used for file transfer, often used to upload and download files.
3. SMTP (Simple Mail Transfer Protocol): The protocol used to send email.
4. POP (Post Office Protocol): The protocol used to receive email.
5. IMAP (Internet Message Access Protocol): A protocol also used to receive email. Unlike POP, IMAP allows users to keep a copy of their email on the server.
6. Telnet (Terminal Network): A remote login protocol that can be used to remotely control and transfer data between computers.
7. SSH (Secure Shell): A secure remote login protocol that can be used to remotely control and transfer data between computers.

It is worth mentioning that the browser does not only have the function of accessing the Web server. It can not only be used to download and upload files on the FTP server, but also has the function of an email client. It can be said that the browser is a server with multiple clients. It is a comprehensive client software with terminal functions, and he needs something to judge which function should be used to access the corresponding data, and these different URLs and protocols allow users to access resources on the Internet in different ways and means.

1.1.2 The browser must first parse the URL

The first step for the browser to do is to parse the URL to generate a request message sent to the Web server. Taking the following situation of accessing the Web server as an example, it will first split out the various elements. , you can understand the meaning of the URL by splitting the elements

Therefore, we can understand that the URL in the picture means that you want to access www.lab.glasscom.com. The path name on the Web server is /dir/file1.html file, that is, the file file1.html located in /dir/ directory D

1.1.3 When the file name is omitted

Sometimes we will see some different URLs, here are some common examples

1. http://www.lab.glasscom.com/dir/

Ending with "/" means that the file name that should be after /dir/ is
omitted

In fact, we will set the default file name to be accessed when the file name is omitted on the server. This setting differs from server to server. In most cases, it is a file name such as index.html or default.htm.

2. http://www.lab.glasscom.com/

Similar to the previous one, the following file name is also omitted, and the file set by the server will be accessed by default.

3. http://www.lab.glasscom.com

This time even the "/" at the end is omitted. Even writing like this, which even omits the directory name, is allowed. In order to prevent confusion, when there is no path name, it means accessing the default file A set in advance in the root directory, that is, /index.html or /default.htm files

4. http://www.lab.glasscom.com/whatisthis

This one is a bit special: we shouldn't always treat whatisthis as a file name. Generally speaking, this situation will be handled according to the following convention: if a file named whatisthis exists on the Web server, treat whatisthis as the file name; if a directory named whatisthis exists, treat whatisthis as the directory name deal with

You cannot create two files and directories with the same name, so don't worry about ambiguity.

1.1.4 Basic ideas of HTTP

First, let’s use a simple example to explain what the HTTP protocol is:
The basic idea of ​​HTTP is that the browser (client) sends a request to the Web server, and the Web server The requested resource is sent back to the browser as an HTTP response. This process can be compared to the process of users shopping in a store:

1. User enters the store: analogy to browser connecting to web server
2. The user selects the product and provides the order: analogy is to send an HTTP request to the browser. The request includes the resources to be accessed and other information.
3. The store receives the order and processes it: analogy to a web server handling HTTP requests
4. The store packages the goods and hands them to the user: analogously to the web server sending the requested resources back to the browser as an HTTP response. The response includes response status code, response header, response body, etc.

The message of this request contains two parts: "what to do" and "what to do". Among them, "what to do" is called the URL, and "what to do" we call it the method. Below The table lists the main methods. You can use it to understand what operations can be performed through the methods.

The subsequent operations are very simple. After receiving the HTTP request message, the web server will parse it. The content includes URI and methods to determine what the request is to access and what operation to perform. The server stores the execution results in the response message. The response message includes status code, header fields and web page data, indicating the success or error of the operation. After the client receives the response message, the browser reads the required data from it and displays it on the screen.

When encountering a file that cannot be found, 404 Not Found will be displayed.

The above methods are commonly used GET and POST methods. With these two methods, we can obtain web page data from the web server and send the information in the web page input box to the web server. In fact, the HTTP protocol also contains a lot of Possibility, I won’t go into details here.

1.1.5 Generate HTTP request message

After parsing the URL, the browser determines the web server and file name, and then generates an HTTP request message based on this information. In fact, the format of HTTP messages is strictly regulated, so the browser will generate request messages according to the prescribed format.

There are many methods, so which one should we choose? In fact, it depends on the working status of the browser. The browser works not only after you enter the URL in the address bar at the top, but also when you click on the web page. hyperlinks, etc., and which method to use is determined based on the scenario.

The specific format is as shown below:

Here you can also add some distinctions between methods in the form, but the author feels it is a bit complicated and will add more in the future

1.1.6 You will receive a response after sending a request

After we send the above request message, the Web server will return a response message. The format and basic idea of ​​the response message are the same as the request message, and the difference is only in the first line. In the response message, the first line of content is the status code and response phrase, which are used to indicate whether the execution result of the request was successful or an error occurred.
To put it simply: the status code is for the program to see, and the response phrase is for humans to see.

• The status code is a numeric code used to indicate the result of the request execution. For example, 200 indicates success, 404 indicates resource not found, 500 indicates internal server error, etc. These status codes allow the program to automatically identify the execution results and handle them accordingly.
• In contrast, the response phrase is a piece of text that describes the result of the request execution, such as "OK" indicating success, "Not Found" indicating that the resource was not found, and "Internal Server Error" indicating an internal server error, etc. The purpose of a response phrase is to provide people with easy-to-understand information to help them understand the results of the execution of the request

Only one URI can be written in each request message, so only one file can be obtained at a time. If multiple files need to be obtained, a separate request must be sent for each file. For example, if a web page contains 3 pictures, then a total of 4 requests need to be sent to the web server to obtain the web page and obtain the pictures.

The web server does not care whether the files obtained by these four requests are on one web page or on different web pages. Its task is to return a response to each separate request.

1.1.7 Complete process diagram

Next, we will show an example of interactive messages between the browser and the web server.

1.2 Query the DNS server for the IP address of the web server

1.2.1 Basic knowledge of IP addresses

The browser can parse URLs and generate HTTP messages, but it does not itself have the function of sending messages to the network, so this function needs to be entrusted to the operating system. When entrusting the operating system to send a message, what must be provided is not the domain name of the communication object, but its IP address, so let's first introduce what an IP address is.

TCP/IP is the network structure used by the Internet and company intranets. This network consists of multiple small subnets connected through routers to form a large network. Each device has a unique IP address, similar to real-life addresses. Through the IP address, we can determine the location of the accessed object server and send the message to the server. The process of message delivery is to forward the message from one subnet to another through routers until the message finally reaches the destination. (It can also be simply understood that you need to fill in the specific house number so that the express delivery can be delivered to you correctly)

The network number is used to identify the network where a host is located, and the host number is used to identify a specific host in the network.

After understanding the basic idea of ​​IP address in TCP/IP, let us take a look at the actual IP address
As shown in Figure 1.9, the actual IP address is a string of 32 Bit numbers are divided into 4 groups according to a group of 8 bits (1 byte), which are expressed in decimal and separated by dots. This is the IP address format we often see, but with this string of numbers alone we cannot distinguish which part is the network number and which part is the host number. In the IP address rules, the network number and host number are connected to a total of 32 bits, but the specific structure of these two parts is not fixed. When building a network, users can decide the distribution relationships between them, such as:

Suppose a network administrator wants to divide a network into subnets and assign each subnet a unique network number. He can choose to use the first few bits of the IP address for the network number and then use the remaining bits for the host number. For example, he could assign the first 8 bits to the network number and then use the remaining 24 bits as the host number. This means that the network can support up to 2^8 subnets, and each subnet can support up to 2^24 hosts.

Another network administrator might want to use the first three bytes of the IP address for the network number and then use the last byte as the host number. Doing so can provide larger networks with more IP addresses. For example, this distribution scheme can support up to 2^24 networks, each of which can support up to 256 hosts.

These are just two examples. Network administrators can determine the allocation relationship between network numbers and host numbers based on the needs and size of the network. Therefore, we also need additional additional information to represent the internal structure of the IP address

This additional information is called the subnet mask. The part of the subnet mask 1 represents the network number, and the subnet The part with a mask of 0 indicates the machine number
For the IP address 10.11.12.13/24, /24 indicates that the number of digits in the network address is 24, that is, the first 3 bytes are the network number. The last 1 byte is the host number. Convert the first 3 bytes to binary and get 00001010.00001011.00001100. Then in the last byte, all the previous 24 bits are set to 1 and all the following 24 bits are set to 0. The subnet mask is 11111111.11111111.11111111.00000000. Convert The decimal system is 255.255.255.0
These are just different in writing, but the meaning is exactly the same

host number of IP address

All 0: indicates the entire subnet
All 1: indicates sending packets to all devices on the subnet, that is, "broadcast"

1.2.2 Reasons for sharing domain names and IP addresses

The TCP/IP network determines the communication object through the IP address, so you cannot send a message to the other party without knowing the IP address, but just like it is difficult to remember a phone number, you have to remember a long list IP addresses composed of numbers are also difficult, so why not just use the name to determine the communication object? It is technically possible, but the length of the IP address is 32 bits. , that is, 4 bytes. In contrast, the shortest domain name is dozens of bytes, and the longest can even reach 255 bytes. In other words, using an IP address only requires processing 4-byte numbers, while a domain name needs to process dozens to 255 bytes of characters, which increases the burden on the router and takes longer to transmit data, so This is not a suitable design

So a solution was created at this time, where people use the name and the router uses the IP address. In order to bridge the gap between the two, there needs to be a mechanism that can query IP addresses by name, or query names by IP address, so that the problem can be solved perfectly without making sacrifices on both sides of the human and machine. This mechanism is DNS

1.2.3 The Socket library provides the function of querying IP addresses

The method of querying the IP address is very simple. Just ask the nearest DNS server what the IP address of a certain website is, and it will tell you. So how does the browser issue a query to the DNS server?
For a DNS server, there must be a corresponding DNS client on our computer, and the part equivalent to the DNS client is called a DNS resolver, or simply a resolver. The operation of querying an IP address through DNS is called domain name resolution, so the person responsible for performing the resolution operation is called a resolver.
The parser is actually a program, which is included in the Socket library of the operating system. The library is a collection of common program components, and other applications need to use the components.
Libraries have many benefits:

1. Building applications using ready-made components saves programming effort
2. Using the same components allows for program standardization

The Socket library is a collection of program components used to call network functions

1.2.4 Issue queries to DNS servers through resolvers

Usage of the parser is very simple. The programs in the Socket library are all standard components, and you only need to call them from the application.

After calling the parser, the parser will send a query message to the DNS server, and then the DNS server will return a response. information. The response message contains the queried IP address, and the parser will take out the IP address and write it into the memory address specified by the browser. Next, when the browser sends a message to the web server, it only needs to take out the IP address from the memory address and hand it to the operating system together with the HTTP request message

When querying an IP address based on a domain name, the browser uses the resolver in the Socket library

1.2.5 Internals of the parser

When a web application (in our case, the browser) calls the parser, the program's control flow is transferred to Internals of the parser.

When control passes to the resolver, the resolver generates query messages to be sent to the DNS server. This process is similar to the process in which the browser generates an HTTP request message to be sent to the Web server. The parser will generate a piece of data indicating "Please tell me the IP address of www.lab.glasscom.com" based on the DNS specifications, and will It is sent to the DNS server. The operation of sending messages is not performed by the parser itself, but is delegated to the protocol stack within the operating system. This is because, like the browser, the parser itself does not have the ability to use the network to send and receive data. After the resolver calls the protocol stack, the control flow will transfer again, the protocol stack will perform the operation of sending the message, and then send the message to the DNS server through the network card

Protocol stack: the network control software inside the operating system, also called "protocol driver", "TCP/IP driver", etc.

By the way, when sending a message to the DNS server, of course we also need to know the IP address of the DNS server. It’s just that this IP address is set in advance as a setting item of TCP/IP, and there is no need to query it again.

1.3 Relay of DNS servers around the world

1.3.1 Basic working of DNS server

The basic job of the DNS server is to receive the query message from the client, and then return a response based on the content of the message
The query message from the client contains the following 3 types of information

1. Domain name
   Name of server, mail server (the part after @ in the email address)
2. Class
   In the early design of DNS, DNS was designed to run on different networks, which might have different naming conventions and protocols. To distinguish these networks, DNS introduced the Class field. The Class field defines the scope of a resource record, that is, the network type or zone to which it applies. Three classes are defined in the original DNS protocol: IN (Internet), CS (CSNET) and CH (CHAOS).
   However, with the popularity of the Internet, IN Class has become the only widely used Class type. Therefore, modern DNS protocols usually ignore the Class field, setting it to the default IN type. Therefore, when we do not specify a Class in a DNS query, it usually defaults to the IN type.
3. Record type
   indicates what type of record the domain name corresponds to. For example, when the type is A, it means that the domain name corresponds to the IP address; when the type is MX, it means that the domain name corresponds to the mail server. For different record types, the information returned by the server to the client will be different

The DNS server will look up the corresponding record from the domain name and IP address comparison table and return the IP address

1.3.2 Hierarchical structure of domain names

Because there are too many servers in the Internet, it is impossible to store them all in one DNS server, so we distribute and store the information in multiple DNS servers. These DNS servers cooperate with each other to find the information to be queried.

All information in the DNS server is stored in a hierarchical structure according to the domain name. The domain name is separated by periods. In the domain name, the position to the right indicates the higher level. The information of each domain is stored in the corresponding hierarchical DNS server. A domain is indivisible, but sub-domains can be created under it to be assigned to different organizations to achieve hierarchical management of domain names.

For example, in the domain name www.nikkeibp.co.jp, the jp at the top represents the domain assigned to the country of Japan; the co at the lower level is the domain classified in Japan, representing the company; and the nikkeibp at the lower level is assigned to a certain company. domain; the lowest www is the name of the server

1.3.3 Find the corresponding DNS server and obtain the IP address

This part mainly introduces how to find the IP address of the target server through DNS. First, we need to know the IP address of the DNS server responsible for managing the target domain name. Since there are tens of thousands of DNS servers on the Internet and it is impossible to search them one by one, a level-by-level registration method can be used. That is, the IP address of the DNS server of the lower-level domain is registered to the upper-level DNS server, and the IP address of the upper-level DNS server is registered to the higher-level DNS server, and so on. In this way, the IP address of the lower-level DNS server can be queried through the upper-level DNS server and a query request can be sent to it.

In the Internet, the top-level domain is not the top-level domain. There is also a first-level domain, called the root domain. The root domain does not have its own name like com and jp, so it is often omitted when writing domain names. However, the DNS server of the root domain stores the information of DNS servers such as com and jp. Therefore, starting from the root domain, we can follow the clues to find the DNS server of any domain.

In order for any DNS server to find and access the root domain DNS server, the root domain's DNS server information needs to be stored in all DNS servers on the Internet. In this way, as long as the client can find any DNS server, it can find the root domain DNS server through it, and then follow the path to find a target DNS server located in the lower layer. There are only 13 IP addresses assigned to root domain DNS servers in the world, and these addresses rarely change, so it is not difficult to save these addresses in all DNS servers.

1.3.4 Speed ​​up DNS server response through caching

In fact, the way DNS works on the real Internet has the following characteristics:

1. One DNS server can manage information for multiple domains, and upper-level domains and lower-level domains may share the same DNS server.
2. The DNS server has a caching function that can remember previously queried domain names and related information. Caching can reduce the time required for queries.
3. The information in the cache is set with an expiration date. After the expiration date, the data will be deleted from the cache.
4. When a DNS server responds to a query, it tells the client whether the result of the response came from the cache or from the DNS server responsible for managing the domain name.

Caching: refers to the technology of storing used data in a high-speed storage device closer to the place where the data is used, in order to improve the speed of subsequent access. This technology has many applications, such as caching between CPU and memory, caching between disk and memory, etc. Caching in the network is also a common technology used to improve access speed.

1.4 Delegate the protocol stack to send messages

1.4.1 Overview of data sending and receiving operations