Table of contents
1.3. Domain name resolution process
1.4. Use the dig tool to analyze the DNS process
1.5. The value of the DNS protocol
2.1. Positioning of ICMP protocol
3.2. NAT IP conversion process
3.3.1. Address translation table
3.4. Defects of NAT technology
3.5. NAT penetration technology
4. Summary of network protocols
1.DNS protocol
• DNS (Domain Name System, Domain Name System) protocol is an application-layer protocol used to convert domain names into IP addresses.
• The DNS protocol belongs to the application layer protocol, and its domain name resolution function is realized by UDP.
1.1. DNS background
In TCP/IP, a program on a host in the network is determined by means of an IP address and a port number. But the IP address is a long series of numbers, which is not easy for people to remember, so people invented something called the host name, and used the hosts folder to describe the correspondence between the host name and the IP address.
Originally, the hosts file was managed by the Internet Information Center (SRI-NIC).
• If a new computer is to be connected to the network, or a computer's IP is changed, it is necessary to apply for changing the hosts file at the information center.
• Other computers also need to regularly download and update the new version of the hosts file to access the Internet correctly.
• When a user accesses an Internet service through a domain name, he will first find the corresponding IP address in the local hosts file through the domain name, and then use this IP address to access the corresponding service.
But this is too troublesome, so the DNS system was born.• An organization's system management agency maintains the correspondence between the IP and host name of each host in the system.
• If a new computer is to be connected to the network, or a computer's IP is changed, the corresponding information needs to be registered in the database.
• When a user accesses Internet services through a domain name, the DNS server will be automatically queried, and the DNS server will retrieve the database to obtain the corresponding IP address.
So far, our computer still retains the hosts file. The hosts file generally stores the mapping between the host name and the IP address. Users can also add domain name and IP mapping relationship in the hosts file independently. In the process of domain name resolution The content of the hosts file will be searched first.Note: Use the cat /etc/hosts command to view the contents of the hosts file.
1.2. Domain Name Introduction
A domain name is a hierarchical structure name used to identify the host name and the organization to which the host belongs, such as www.baidu.com.
• com: A first-level domain name, indicating that this is a business domain name. There are also .net (network provider) and .org (open source organization or non-profit organization) at the same level.
• baidu: Second-level domain name, generally corresponding to the company name.
• www: just a customary usage. When people used the domain name before, they often named it in a format similar to ftp.xxx.xxx/www.xxx.xxx to indicate the protocol supported by the host. This part can also be omitted.
1.3. Domain name resolution process
After entering the url in the browser, if the url contains a domain name, domain name resolution is required.
• First, it will check whether there is a corresponding record in the DNS cache of the browser. If the record is found, the corresponding IP address can be obtained directly to complete the resolution.
• If it is not found in the browser's DNS cache, it will query the DNS cache in the operating system, and if the corresponding IP address is found, the resolution will be completed.
• If it is not found in the DNS cache of the operating system, it will search the local hosts file, and complete the resolution if the corresponding IP address is found.
• If it is not found in the local hosts file, it will search in the local DNS server. The local DNS server IP address is generally provided by the local network service provider, such as telecom, mobile and other companies, and is usually automatically assigned through DHCP. At present, the public NDS 8.8.8.8 provided by Google and the domestic public DNS 114.114.114.114 are mostly used. If there is a cache of the corresponding domain name in the local DNS server, the corresponding IP address is returned directly to complete the resolution.
• If it is still not found in the local DNS server, then the local DNS server will take the domain name to the root DNS server to inquire, and the root DNS server will tell the local DNS the IP address of the top-level domain name server.
• After the local DNS obtains the IP address of the top-level domain name server, it will take the domain name to the top-level DNS server, and the top-level domain name server will tell the local DNS the IP address of the authoritative domain name server.
• The local DNS server takes the domain name to the authoritative domain name server, queries the IP address corresponding to the domain name, and finally returns the IP address corresponding to the domain name to the browser. At this time, the entire domain name resolution process is completed.
1.4. Use the dig tool to analyze the DNS process
We can use the dig tool to view the process of domain name resolution, for example, to view
www.baidu.comthe process of Baidu domain name resolution.The analysis results of the dig tool are as follows:
Interpretation of results:
• At the beginning is the version number of the dig tool.
• The second part is the details returned by the server, where the status parameter is NOERROR, which means the query is successful.
• QUESTION SECTION indicates the domain name to be queried.
• ANSWER SECTION indicates the result of the query. First, www.baidu.com is queried as www.a.shifen.com, and finally www.a.shifen.com is queried as two specific IP addresses.
• At the bottom are some result statistics, including query time and DNS server address, etc.
1.5. The value of the DNS protocol
Value one:
Domain name resolution technology is a very important strategic-level technology. This does not mean that its technical threshold is high, but its function and mode are very important.
The domain name resolution service was first owned by the United States. If the United States imposes sanctions on other countries and stops the domain name resolution service, then all companies’ websites will not be able to use domain names to access, which will directly affect the company’s profits and even the country’s development.
Many other countries are opposed to domain name resolution services being occupied by the United States. After several meetings, the information center finally decided that the domain name resolution server allows other countries to mirror. Today, our country also has a national information center, which will consider the security of domain name resolution.
Value two:
Today's Internet market generally has its own server rooms in multiple regions. For example, company A has its own server rooms in Beijing, Inner Mongolia, and Guizhou. The three server rooms correspond to three IPs, that is, three domain names. However, a company will only provide one domain name to the outside world, so there is a problem that all users visit one of the domain names, that is, they all visit one server room while the other two server rooms are idle.
In fact, the domain name provided by Internet Dachang corresponds to multiple IP addresses. When the client performs domain name resolution, it will be randomly assigned an IP address. This IP address may correspond to any server room. In this way, according to the domain name resolution, the load balancing of server room access is realized.
2. ICMP protocol
ICMP (Internet Control Message Protocol) Internet Control Message Protocol, used to transfer control information between IP hosts and routers, is a TCP/IP protocol.
2.1. Positioning of ICMP protocol
In the TCP/IP four-layer model, the network protocol stack is divided into application layer, transport layer, network layer and data link layer from top to bottom.
Among them, the most typical protocols of the application layer are HTTP, HTTPS and DNS, etc., the most typical protocols of the transport layer are TCP and UDP, the most typical protocol of the network layer is IP, and the most typical protocol of the data link layer is the MAC frame protocol. There are also two protocols at the network layer called ICMP and IGMP.
Although ICMP, IGMP and IP protocols belong to the network layer protocol, ICMP protocol and IGMP protocol belong to the upper layer protocol of IP.
• That is to say, the upper-layer protocol of IP is not necessarily the protocol of the transport layer directly, and the upper-layer protocol of IP may also belong to the protocol of the network layer, but it is located in the upper layer of IP.
• Similarly, the ARP protocol and RARP protocol in the data link layer, although these two protocols and the MAC frame protocol belong to the data link layer, these two protocols belong to the upper layer protocol of the MAC frame.
2.2. ICMP function
The main functions of ICMP include:
• Confirm whether the IP packet reaches the destination address successfully.
• Notify the cause of IP packet discard during transmission.
• ICMP can only be used with IPv4. If it is IPv6, ICMPv6 needs to be used.
for example:
For example, when host A is sending data to host B, host B has been offline for some reason.
When the sent data packet arrives at the ingress router of the LAN where host B is located, in order to obtain the MAC address of host B, the ingress router will send an ARP request packet to host B, but because host B is offline, the router is sending ARP multiple times After requesting a packet but not getting a response, it will return an ICMP Destination Unreachable packet to host A. At this time, host A knows that the data it sends cannot reach host B.
2.3. ICMP protocol format
The ICMP protocol format is as follows:
ICMP is roughly divided into two types of messages.
• One is for notifying the cause of an error.
• One class is for diagnostic queries.
Common types of ICMP packets are as follows:
2.4.ping command
The ping command is implemented based on the ICMP protocol, and is usually used to test whether the communication channel between the local host and another host is normal.
For example, use the ping www.baidu.com command to test whether the communication channel between the local host and the Baidu server is normal.
• Note that the domain name of Baidu is pinged here, and the domain name will be resolved into an IP address by DNS.
• The ping command can not only verify the connectivity of the network, but also count the response time and TTL (Time To Live in IP packets, time to live).
• The ping command will first send an ICMP Echo Request to the peer.
• After the peer end receives it, it will return an ICMP Echo Reply.
Question: The port number corresponding to telnet is 23, and the port number corresponding to ssh is 22. What is the port number corresponding to ping?
Answer: This is a trap set up by the person who asked the question. The ping command is based on the ICMP protocol at the network layer, and the port number belongs to the content of the transport layer. Therefore, the ICMP protocol does not care about such information as the port number at all.
Therefore, the ping command actually bypasses the transport layer. In Linux, there is actually a set of network programming interfaces that bypass the transport layer, called native sockets.
2.5. traceroute command
The traceroute command is also implemented based on the ICMP protocol. The traceroute command can traverse all the routers through which the data packet is transmitted to the target host.
For example, use the traceroute www.baidu.com command to traverse all routers through which data packets are sent to the Baidu server.
Brief description of the principle:
• The bottom layer of the traceroute command is actually implemented by increasing the time-to-live (TTL) value.
• Because every time a data packet passes through a router, its TTL value will decrease by 1. When the TTL value decreases to 0, the corresponding routing device will discard the data packet and send an ICMP TTL data packet to the sending host.
• Therefore, the bottom layer of the traceroute command can send multiple data packets, and set different TTL values for these data packets, and finally the host can obtain a series of data packet paths.
3. NAT technology
NAT (Network Address Translation, Network Address Translation) technology is the main means to solve the shortage of IP addresses, and can effectively avoid attacks from outside the network, hide and protect computers inside the network.
3.1.NAT technical background
In the IPv4 protocol, the shortage of IP addresses is a big problem, and NAT technology is currently the main means to solve the shortage of IP addresses, and it is an important function of routers.
• During external communication, NAT can convert private IP to global IP through a series of replacement operations, that is to say, NAT is a technical method for mutual conversion of private IP and global IP.
• A router equipped with NAT software is called a NAT router. All hosts using private IPs must convert their private IPs to global IPs on the NAT router when communicating with the outside world.
• In many schools, families, and companies, the IP set for each terminal is a private IP, and the global IP is only set on the router or necessary server.
• The global IP is required to be unique, but the private IP is not required, and the same private IP in different LANs is completely unaffected.
3.2. NAT IP conversion process
Assume that there are three hosts A, B, and C in a local area network, and there is a server in the public network. Taking host A accessing this server in the public network as an example, let’s take a look at the IP address of the data packet during transmission. conversion process.
The process of data packets from LAN to public network:
When host A initiates a data request to the server, the conversion process of the IP address in the data packet is as follows:
• At the beginning, the source IP address in the data packet is the private IP address of host A, and the destination IP address is the public IP address of the server.
• When the data packet passes through the NAT router, the router will replace the source IP address of the data packet with its own WAN port IP address. At this time, the source and destination IP addresses of the data packet are both public network IP.
• The data packet is forwarded through various routes in the Internet, and finally reaches the server host.
After the server receives and processes the data request from host A, it will respond to the request from host A.The process of data packets from the public network to the LAN:
When the server responds to host A, the conversion process of the IP address in the data packet is as follows:
• At the beginning, the source IP address in the data packet is the public IP address of the server, and the destination IP address is the WAN port IP address of the router.
• The data packet is forwarded through various routes in the Internet and reaches the NAT router of the LAN where host A is located. At this time, the router will replace the destination IP address of the data packet with the private IP address of host A.
• Finally, the router will forward the data packet to host A in the LAN.
It should be noted that when host A initiates a data request to the server, the source IP address in the data packet is replaced with the WAN port IP address of the NAT router, which means that the router initiates a data request to the server instead of host A, so The destination IP address of the response packet sent by the server should be the IP address of the WAN port of the NAT router.
3.3.NAPT technology
3.3.1. Address translation table
• When a host in the local area network wants to access the external network, the NAT router will replace the source IP address of these data packets with its own WAN port IP address.
• When the response data is sent from the external network, the NAT router will replace the destination IP address of the response data packet with the IP address of the corresponding host in the LAN.
How does the NAT router determine which host in the LAN should forward the response packet received from the external network?• Actually inside the NAT router, there is an automatically generated table for address translation.
• The translation table maintains the mapping relationship between the private IP of the host in the LAN and the corresponding public IP in the external network.
• When a host in the LAN initiates a data request to the external network for the first time, the mapping relationship in the table will be generated.
• For example, when a TCP connection is established, the corresponding mapping relationship will be established, and after the TCP connection is disconnected, the corresponding mapping relationship will be deleted.
In the example just now, when host A initiates a data request to the server for the first time, the following mapping relationship will be established in the router.
When the NAT router receives the response data sent by the server to host A, it can know that the response data is sent to host A in the LAN by looking up the table.
However, if the conversion table only maintains the mapping relationship between the private IP of the host in the LAN and a public IP in the corresponding external network, some problems will arise.
If both host A and host B in the local area network are accessing the server at the same time, then the following two pairs of mapping relationships will be established in the conversion table at this time:
At this time, this conversion table can only guarantee the uniqueness from left to right, but not from right to left. When the server sends response data, the destination IP address in the data packet is the WAN port of the router. At this time, the NAT router cannot judge whether the data packet should be forwarded to host A or host B, and NAPT technology is needed at this time.
3.3.2.NAPT technology
NAPT (Network Address Port Translation, Network Address Port Translation), can map multiple internal addresses to a legal public network address.
• When a host in the local area network sends data to the external network, the router will replace the source IP address of the data packet with its own WAN port IP address, and establish a mapping relationship between the host's private IP and the corresponding public network IP .
• However, if multiple hosts in the LAN access the same external network service at the same time, when the router receives the response data from the external network, the router cannot determine which host in the LAN the response data should be forwarded to, because the All the data packets of the host are sent by the router instead, so the destination IP address of the response data packet sent is the WAN port IP address of the router.
• Therefore, when NAPT establishes the mapping relationship of the conversion table, in addition to establishing the mapping relationship between the private IP in the LAN and the corresponding public network IP, it will also add a port number selected by the NAT router.
• At this time, when multiple hosts in the local area network access the same external network service at the same time, although the destination IP address of the response data sent from the external network is the WAN port IP of the router, the response data sent to different hosts in the local area network correspond to The destination port numbers of different hosts are different, and at this time, the router can distinguish the data packets sent to different hosts by means of IP+Port.
For example, host A and host B in the LAN are both accessing the same server, and the port numbers they use to access the server are both 1025.• Assume that the data packet sent by host A reaches the router first. At this time, the router replaces the source IP address of the data packet with its own WAN port IP address. Since port 1025 used by the router to access the server is not used, the data packet The source port number of can be unchanged.
• When the data packet sent by host B arrives at the router, the router also replaces the source IP address of the data packet with its own WAN port IP address, but at this time the port 1025 used by the router to access the server has already been used by host A , so the router will reselect a port number to replace the source port number of the data packet.
At this time, the following two-team mapping relationship will be established in the conversion table:
At this time, this conversion table can not only guarantee the uniqueness from left to right, but also guarantee the uniqueness from right to left.
• When the response data sent by the server arrives at the router, although the data packets sent by the server to host A and host B correspond to the same destination IP address.
• However, the router uses its own port 1025 instead of host A for data requests, and uses port 1026 instead of host B for data requests.
• So now the router can continue to judge whether the data packet should be forwarded to host A or host B according to the source port number of the data packet, replace the destination IP address and destination port number in the data packet, and then forward it to the local area network the corresponding host.
Talk about routers:
A router is a device that works at the network layer. The load forwards data packets from one network to another, but it cannot be narrowly considered that a router can only work at the network layer.
When a NAT router forwards data, it not only has the ability to replace the source and destination IP addresses of data packets, but also may replace the source and destination port numbers of data packets if necessary, and the port number is actually a concept of the transport layer.
In order to dynamically manage IP addresses, most routers have a DHCP function, and DHCP is actually a protocol at the application layer.
Therefore, the current routers do not only provide services related to the network layer, but may be involved in all layers of routers in the network protocol stack.
3.4. Defects of NAT technology
The replacement between private IP and public network by NAT technology mainly relies on the network address translation table maintained in the NAT router, but this translation table also reflects some defects of NAT:
• It is impossible to establish a connection to the internal host or server from the outside of the NAT, because the outside cannot know the internal private network IP. It is impossible to actively establish a connection with the internal host or server (this defect can be solved by NAT penetration technology).
• The generation and destruction of translation tables requires additional overhead.
• Once the NAT device is abnormal during the communication process, all TCP connections will be disconnected even if there is hot backup.
3.5. NAT penetration technology
NAT penetration technology:
• NAT traversal technology enables hosts outside the LAN (outside NAT) to actively access private IP hosts inside the LAN (inside NAT).
• NAT traversal technology is actually to let the hosts inside the NAT visit the hosts outside the NAT at regular intervals. As long as the hosts inside the NAT access the hosts outside the NAT, a translation table will be established in the NAT router, so that if the hosts outside the NAT want to actively access the hosts inside the NAT It can also be accessed directly.
• If you want to perform NAT penetration, you can download some internal network penetration software on the NAT internal host. These software can help us regularly access the NAT external host, that is, let the NAT router establish a corresponding translation table.
• Recommended software for intranet penetration: NATAPP, Intranet Cloud, cpolar, Peanut Shell, Flying Pigeon, Net Cloud, etc.
• If our host achieves intranet penetration, then the host can also act as a server, provide services to the corresponding host, and be accessed by the corresponding host.
3.6. NAT and proxy server
The function of the proxy server (Proxy Server) is to act as a proxy for network users to obtain network information, and the proxy server is divided into forward proxy and reverse proxy.
Forward proxy:
The forward proxy is a server located between the client and the target server. The client does not directly access the target server, but first visits the proxy server. The proxy server replaces the client to access the corresponding target server, and the target server The response result is returned to the client.
For example, the company generally has its own server, when we use the company's intranet to access the Internet.
• The data request we initiate from the external network will first be forwarded to the company's server, and then the company's server will access the external network instead of you.
• After the company's server receives the response data corresponding to the external network, the company's server will forward the data to you.The benefits of forward proxy:
(1) One of the biggest benefits of forward proxy is that it can speed up resource access. For example, if a large number of employees in the company want to access the same resource on the external network, the forward proxy server can cache the corresponding resource locally. At this time, when other people want to access the resource, they can obtain it directly on the forward proxy server. , without the need to access the external network again.
(2) The forward proxy can filter or intercept access requests (the campus network intercepts illegal requests).
(3) The forward proxy can perform identity authentication (login to the campus network).
(4) The intranet is protected to a certain extent.
reverse proxy:
A reverse proxy is also a server located between the client and the target server. For the client, the reverse proxy server is equivalent to the target server. The user does not need to know the address of the target server, and the user only needs to visit the reverse proxy server The services provided by the target server can be obtained.
The reverse proxy is also a server located between the client and the target server. The client directly initiates a data request to the reverse proxy server, and then the reverse proxy server forwards the client's data request to the real target server for processing. After the data is processed, the reverse proxy server returns the data result to the client.
For example, the server corresponding to the domain name www.baidu.com is actually a reverse proxy server.
• There is actually not only one server inside Baidu, but people in different regions can enjoy the services provided by Baidu by visiting www.baidu.com. In fact, what we visit is Baidu's reverse proxy server.
• When this reverse proxy server receives the client's data request, it will forward our data request to a server within Baidu for data processing, and then return the data processing result to the client.Benefits of reverse proxy:
(1) The reverse proxy can play the role of load balancing. For example, if a reverse proxy server is not set up, when users visit Baidu, they will randomly access a certain server inside Baidu. At this time, some servers may be under too much pressure, while some servers are idle. After setting up the reverse proxy server, we can use some methods to make the user's data request fall on each server more evenly.
(2) The reverse proxy can also play a role in security protection (the security level of the reverse proxy is much stronger than that of the forward proxy). With the direction proxy server, we don't need to directly expose the information corresponding to the server that provides the service. In addition, when an illegal request is sent to the reverse proxy server, the reverse proxy server is equivalent to a layer of software barrier, which can be used in Deploy some protective measures in the reverse proxy server (such as black and white lists, etc.), so that these illegal requests can be filtered out in the reverse proxy server without affecting the internal servers that actually provide services.
Note: The main job of the proxy server is to forward data, so the pressure on the proxy server to process data will not be particularly high, and there can be multiple proxy servers, so there is no need to worry about the overload of the proxy server.
Similarities and differences between forward proxy and reverse proxy:
The same points of forward proxy and reverse proxy:
• Both the forward proxy server and the reverse proxy server are located between the client and the server.
• The main work of the forward proxy server and the reverse proxy server is to forward the client's request to the server, and then forward the server's response to the client.The difference between forward proxy and reverse proxy:
• The forward proxy is the proxy of the client, helping the client to access server resources that it cannot access, while the reverse proxy is the proxy of the server, helping the server to do load balancing and security protection.
• The forward proxy is generally set up by the client. For example, the company's forward proxy server is set up by the company as the client, while the reverse proxy is generally set up by the server. For example, Baidu's reverse proxy server is set up by Baidu as the server. of.
• In the forward proxy, the server does not know who the real client is, and the server thinks that the forward proxy server is the real client, while in the reverse proxy, the client does not know who the real server is, and the client thinks the reverse proxy server is the real client. A proxy server is a real server.
The difference between NAT and proxy server:
Both NAT and proxy server initiate data requests to the server instead of us, but they have the following differences:
• In terms of application, NAT equipment is one of the basic network equipment, and it solves the problem of IP shortage, while proxy server is closer to specific applications, such as bypassing the wall through the proxy server, and accelerators like Xunyou are also The proxy server to use.
• In terms of underlying implementation, NAT works at the network layer and directly replaces IP addresses, while proxy servers often work at the application layer.
• From the scope of use, NAT is generally deployed at the egress of the LAN, and the proxy server can be used as a proxy in the LAN, or in the WAN, or across networks.
• From the perspective of deployment location, NAT is generally integrated on hardware devices such as firewalls and routers, while a proxy server is a software program (such as Nginx and Apache) that needs to be deployed on a server.
Forward proxy server application:
All our network requests will be forwarded through the operator's server, and our host cannot access foreign websites such as www.google.com, because the operator's server will intercept domain names like www.google.com, when we visit www.qq.com and other domestic websites, the operator's server will forward it.
Hong Kong and Taiwan can access foreign websites. If we buy a server in Hong Kong (this server can access foreign websites), when we want to access the external network, we can first send the access request to this Hong Kong server. When the access request is sent, the access request is encrypted (the purpose is to encrypt the domain name). After the access request arrives at the operator's server, the operator's server cannot recognize the destination domain name of the request and will forward it. The Hong Kong server can proceed after receiving the access request. Decryption (because the access host and the Hong Kong server are both ours, they can be encrypted and decrypted normally), then the Hong Kong server visits the foreign website, and returns the response result to our requesting host in the same way.
Note: We don't need to do the above work by ourselves, we can find out how to realize it by ourselves.
Public IP of the cloud server:
When we write the socket code, we cannot directly bind the public network IP of the cloud server. We can understand that the public network IP given to us is the IP of the reverse proxy server in the cloud server room, and what we use is not the reverse proxy server IP. To the proxy server, but to a certain server host in the server room, the IP address of the server host is not the public network IP for us, so we cannot bind directly when writing the socket.
4. Summary of network protocols
Application layer:
• The role of the application layer: responsible for communication between applications, and complete a series of services required for business processing.
• Be able to design corresponding application layer protocols according to their own needs.
• Understand the HTTP protocol.
• Understand the principles and workflow of DNS.
Transport layer:
• The role of the transport layer: responsible for providing services for the communication between two network communication processes, and more for providing some strategies for data transmission.
• Understand the concept of port numbers.
• Know the TCP protocol, understand the reliability of the TCP protocol, and understand the state transition of the TCP protocol.
• Master the characteristics of TCP connection management, acknowledgment response, timeout retransmission, sliding window, flow control, congestion control, delayed response, and piggyback response.
• Understand that TCP is byte-oriented, and understand the problem and solution of sticky packets.
• Can realize reliable transmission based on UDP.
• Understand the impact of MTU on UDP/TCP.
Network layer:
• The role of the network layer: to determine an appropriate path in a complex network environment.
• Understand IP address, understand the difference between IP address and MAC address.
• Understand the IP protocol format.
• Understand how to solve the problem of insufficient number of IPs, master the two schemes of network segment division, and understand private IP and public IP.
• Understand the IP address routing process at the network layer, and understand how a data packet reaches the final destination across network segments.
• Understand why IP packets are fragmented.
• Understand the ICMP protocol.
• Understand how NAT devices work.
data link layer:
• The role of the data link layer: responsible for data transfer between two devices in the LAN.
• Understand the principles of Ethernet communication
• Ethernet frame format.
• Understand MAC addresses.
• Understand the ARP protocol.
• Understand MTU.