Toutiao released 5 minutes and was recommended by a million people. What is the charm of this interesting network protocol document compiled by Huawei's 18th-level engineer for three years!

Preface

Although everyone has studied network protocols in college, I must feel that there are many knowledge points about network protocols and they are very complicated. I was confused when I was learning, and I was even more confused in practice. Once I encountered a network problem in my work, there was basically no way to solve the problem except for a few simple pings. However, when picking up a book to study, or reading some official documents, a variety of rare professional vocabulary immediately rushes toward you. Every time you understand a vocabulary, you have to read multiple articles and read multiple books, resulting in an even A short article on network technology will take several weeks to read.

This is a serious blow to everyone's self-confidence, and it is easy for people to lose themselves in the ocean of technology, resulting in the urge to "go from the door to giving up"!

The network protocol is different from the ever-changing cutting-edge technology. Its changes are relatively small. Once you master it to a certain degree, you will always benefit. The technology has changed quickly. In recent years, OpenStack, docker, Mesos, kubernetes, microservices, serverless, AIops, etc. Technology emerges endlessly, making most technicians overwhelmed. However, after mastering the basic knowledge, they found that many technologies look “vigorous”. Taking off their coat, the essence is actually the operating system computer network, algorithm and data structure, compilation principle, computer composition and system structure.

If the foundation is laid, the biggest benefit is that after the latest technology comes out, as long as it is learned in a short period of time, it will be easy to learn, and it will be able to maintain the ability to learn quickly in the wave of new technology.

Since the network protocol is the foundation and cannot be bypassed, it is still so difficult, but it doesn’t change much after the trip, and the benefits are getting bigger and bigger, so why not write a document and give you some lessons to learn from, and help you master the network as soon as possible What about the agreement?

So, today we will learn about the network protocol from the three parts of the catalog, the main contents and the summary. I hope you will like it! !

table of Contents

main content

The content of this article is divided into nine chapters to introduce you:

Chapter 1 Overview of Communication Protocol.

1.1 Why learn network protocols

1.2 The true meaning of network layering, to summarize the content of this section, to understand the working mode of the network protocol, there are the following two tips.

• Always imagine yourself as a program that processes network packets: how to get network packets, how to process them according to the rules, and how to send them out.
• Always keep in mind a principle: As long as the package runs on the network, it is complete. There can be a lower layer without an upper layer, and there is absolutely no upper layer without a lower layer.

1.3 ifconfig: A familiar and unfamiliar command line. Through the study of this section, I hope you can remember the following knowledge points, which can be used later:

• The I address has the positioning function, and the MAC address is similar to the ID number, without the positioning function.
• CIDR can be used to determine whether it is a local address.
• IP addresses are divided into public IP addresses and private IP addresses. The following chapters will talk about "going abroad", which is related to this.

1.4 DHCP and PXE: How did the IP address come from, and why did it disappear? This section is summarized as follows:

• DHCP mainly leases client IP addresses. This process is very similar to renting a house. Negotiations, signing contracts, and lease renewals are required. Broadcasting cannot "grab orders".
• DHCP will recommend the "decoration team" PXE to the client to install the operating system, which is very useful in the field of cloud computing.

Chapter 2 from the second to the third floor.

2.1 From the physical layer to the MAC layer: How to network and play online games in the dormitory, there are 3 important points to remember in this section:

• The MAC layer is used to solve the "traffic jam" problem of multiple access.
• ARP searches for the target MAC address by "roaring". After the "roaring", it will be remembered for a period of time. This is called cache.
• The switch is capable of MAC address learning. After learning it, you can know who is where, without broadcasting.

2.2 Switch and VLAN: The office is too complicated, I want to go back to school, this section is summarized as follows:

• ·When the number of switches is increasing, loop problems will be encountered and broadcast packets will get lost. At this time, it is necessary to use STP to turn a graph with loops into a tree without loops through a "competition and sword" method to solve the loop problem.
• · Too many switches can cause isolation problems. A virtual local area network can be formed through VLAN to solve broadcasting and security issues.

2.3 ICMP and ping: Scouts who throw stones and ask directions. The content of this section is summarized as follows:

• · ICMP is equivalent to the scout in the online world. This section explains two types of ICMP messages, one is the active probe query message, and the other is the error message for abnormal reporting.
• Ping uses query messages, and Traceroute uses error messages.

2.4 The world is so big, I came up with a gateway: Ten European Countries Tour and Xuanzang's westward journey. This section is summarized as follows:

• ·If you leave the LAN, you need to go through the gateway.
• ·The router is a three-layer device with rules on how to find the next hop.
• ·The MAC header needs to change after passing through the router. If the I address does not change, it is equivalent to the "Tour of Ten Countries in Europe" without changing the passport. If the IP address changes, it is equivalent to "Xuan Zang Westbound" for changing the passport.

2.5 Routing protocol: "West-out gateway for no reason" "Dare to ask where is the way", this section summarizes as follows:

• Routing is divided into static routing and dynamic routing. Static routing can be configured with complex policy routing to control forwarding strategies.
• There are two mainstream protocols for dynamic routing, distance vector routing protocol and link state routing protocol. Corresponding to the two implementations of BGP and OSPF respectively.

Chapter 3 Important Transport Layer.

3.1 UDP: Although simple but can be customized, this section summarizes as follows:

• If you compare TCP to a mature social person, UDP is a simple-minded kid. TCP is complicated, UDP is simple. TCP maintains connections, UDP everyone believes. TCP knows how to advance and retreat, UDP froze, and went forward bravely.
• ·UDP is simple, but it has simple usage. It can be used where the environment is simple, multicast is required, and the application layer controls the transmission itself, such as DHCP, VXLAN, QUIC, etc.

3.2 TCP (Part 1): Although complicated, it is easy to use. This section is summarized as follows:

• · The TCP header is very complicated, but it mainly focuses on five aspects: sequence problems, packet loss problems, connection maintenance, flow control, and congestion control.
• The establishment of the connection requires three handshake, and the disconnection requires four waves of hands.

3.3 TCP (Part 2): Going west must be more enchanting, persevering and wise to eliminate hardships, summarized as follows:

• Sequence problems, packet loss problems, and flow control are all solved by sliding windows. The sliding windows are actually equivalent to the work memos of the leaders and subordinates. The assigned tasks must be numbered, and there will be feedback after they are done. You cannot send too many tasks. , It can't be too little.
• Congestion control is solved through the congestion window, which is equivalent to pouring water into the pipe. If it is faster, it will overflow, and if it is slow, it will waste bandwidth. You must cross the river by feeling the stones to find the optimal value.

3.4 socket: Talk is cheap, show me the code, this section is summarized as follows:

• You need to remember which functions both the client and the server need to call during the function call process of socket programs based on TCP and UDP.
• It is not easy to write a high-concurrency server that can support a large number of connections. It requires multiple processes and multiple threads, and epoll can solve the C10K problem.

Chapter 4 Common Application Layer.

4.1 HTTP: It turned out to be so troublesome to read the news. This section is summarized as follows:

• HTTP is very commonly used and very complicated. It is important to remember the methods GET, POST, PUT, DELETE, and important header fields.
• HTTP2.0 improves performance through techniques such as header compression, framing, binary encoding, and multiplexing.
• The QUIC protocol further improves its performance through UDP-based custom connection, retransmission, multiplexing, flow control and other mechanisms.

4.2 HTTPS: The process of ordering food was originally so complicated. This section is summarized as follows:

• Encryption is divided into symmetric encryption and asymmetric encryption. Symmetric encryption has high efficiency, but it cannot solve the problem of key transmission; asymmetric encryption can solve this problem, but it is inefficient.
• Asymmetric encryption requires certificates and authorities to verify the legitimacy of the public key.
• HTTPS is an HTTP that combines symmetric encryption and asymmetric encryption. It not only guarantees transmission safety, but also guarantees transmission efficiency.

4.3 Streaming Media Protocol: How to see handsome men and beautiful women in live broadcasts, this section summarizes as follows:

• The two major genres of coding have reached an agreement, both of which compress data through various algorithms about time and space.
• The compressed data will form a series of NALUs for the convenience of transmission, arranged in order of frames and slices.
• When the arranged NALU is transmitted over the network, it must be packaged according to the format of the RTMP packet, and the RTMP packet will be split into blocks for transmission.
• The video stream pushed to the streaming media server is transcoded and distributed, and can be pulled by the client through RTMP, then combined into NALU, decoded into a video format for playback.

4.4 P2P protocol: download movies, distributed protocol is fast, this section summarizes as follows:

• To download a file, you can use HTTP or FTP. Both protocols use a centralized download method, while P2P uses a different way of thinking and adopts a decentralized download method.
• P2P also has two downloading methods. One is to rely on the tracker server, that is, metadata is centralized, and file data is scattered; the other is based on a distributed hash algorithm, where metadata and file data are all scattered.

Chapter 5 Unfamiliar Data Center.

5.1 DNS: The address book of the network world, this section is summarized as follows:

• DNS is the address book of the network world, and addresses can be queried through domain names. Since DNS servers are organized in a tree structure, domain name lookup uses a recursive method and enhances performance through caching.
• The process of mapping between domain names and IP addresses gives applications an opportunity to perform load balancing based on domain names, which can achieve simple load balancing, or global load balancing based on addresses and operators.

5.2 HTTPDNS: The address book of the online world will also point the wrong way. This section needs to remember the following two important points:

• ·Traditional DNS servers have many problems, such as slow resolution and untimely updates. Because of caching and forwarding NAT problems, the client misunderstands where it is and the operator it belongs to, which affects the scheduling of traffic.
• The HTTPDNS server uses the client SDK and the server directly calls and resolves the DNS server through HTTP, bypassing the shortcomings of traditional DNS servers and realizing intelligent scheduling.

5.3 CDN: Have you ever picked up the courier from the canteen? In this section, you need to remember the following two important points:

• The distributed warehousing system of CDN and e-commerce system is divided into central nodes, regional nodes, and edge nodes, which cache data in the location closest to users.
• CDN is best at caching static data, in addition to caching streaming media data, you should pay attention to the use of anti-theft links. CDN also supports dynamic data caching, and there are two available modes: one is the fresh supermarket mode of edge computing, and the other is the cold chain transportation mode of link optimization.

5.4 Data Center: I am a developer, and I take the land to build a villa by myself. In this section, you need to remember the following 3 important points:

• The data center is divided into three levels. The server is connected to the access layer, then the aggregation layer, then the core layer, and the outermost border routers and security devices.
• All links in the data center must be highly available. Servers can be bound to network cards, switches can be stacked, Layer 3 devices can pass equal cost routing, and Layer 2 devices can achieve high availability through the TRILL protocol.
• With the development of cloud and big data, east-west traffic is more important than north-south traffic, and the leaf-spine network structure has evolved.

5.5 VPN: There are people in North Korea who are good officials. This section is summarized as follows:

• VPN can connect multiple data centers of an organization through tunnels, making the organization feel like it is in a data center, just like traveling by car through the Qiongzhou Strait.
• The software-based IPsec VPN can guarantee privacy, integrity, and authenticity. It is simple and cheap, but its performance is slightly worse.
• MPLS-VPN combines the advantages of the I forwarding mode and the ATM label forwarding mode, with better performance, but it needs to be purchased from the operator.

5.6 Mobile network: When I go to Barcelona, ​​I can’t access "Facebook" on my mobile phone. This section is summarized as follows:

• The development of mobile networks has evolved from 2G to 3G, and then to 4G, and its functions have gradually shifted from making phone calls to using the Internet.
• Please remember the structure of the 4G network, including eNodeB, MME, SGW, PGW, etc., which are divided into control plane protocol and data plane protocol. You can compare this structure and try to tell the flow of mobile Internet access.
• Even if you surf the Internet within the scope of a foreign operator, it must be controlled by a domestic operator, so you can't get on Facebook.

Chapter 6 Networks in Cloud Computing.

6.1 Cloud Network: The cost of owning land is high, and buying apartments is more flexible. This section is summarized as follows:

• The key technology of cloud computing is virtualization. Here we focus on the virtual network card to connect the inside and outside of the virtual machine by opening the TUN/TAP character device.
• The network in the cloud focuses on four aspects: sharing, isolation, interoperability, and flexibility. Among them, there are two common ways of sharing and intercommunication, namely bridging and NAT. Isolation can be done through VLAN.

6.2 Software-defined network: Community property management methods that share infrastructure, this section summarizes as follows:

• Using SDN to control the entire cloud network is the same as the community security managing the entire property from the main control room, separating the control plane and the data plane.
• Open vSwitch is an implementation of an open source virtual switch, which can make any modification to the network package that passes through itself, thus making the cloud control of the network very flexible.
• After introducing Open vSwitch into the cloud, the configuration can be simple and flexible, and the physical network and virtual network can be decoupled.

6.3 The security of the cloud network: Although it is not a local tyrant, it also needs basic protection. This section is summarized as follows:

• The common way of security policy in the cloud is to use the rules of iptables, please remember its 5 chains: PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING.
• There are 4 types of iptables tables: raw, mangle, nat, and filter. Among them, the security policy is mainly implemented in the filter table, and the conversion of virtual network and physical network addresses is mainly implemented in the nat table.

6.4 QoS in the cloud network: roommates download movies crazy, what should I do, this section summarizes as follows:

• Traffic control in the cloud is mainly carried out through queues. The queuing rules are divided into two categories: classless queuing rules and class-based queuing rules.
• In the cloud network Open vSwitch, HTB is mainly used to allocate the total bandwidth in a tree according to the configured ratio, and when one branch does not use traffic, it is loaned to another branch, thereby enhancing bandwidth utilization.

6.5 Isolation of networks in the cloud GRE, VXLAN: Although living in a community, privacy must be protected. This section is summarized as follows:

• To isolate the networks of different users and solve the problem of the limited number of VLANs, the overlay method is needed, and GRE and VXLAN are often used.
• GRE is a point-to-point tunnel mode, and VXLAN is a tunnel mode that supports multicast. They must be encapsulated and decapsulated at a certain tunnel port to realize interworking across physical machines.
• Open vSwitch can be used as a tunnel port to switch between virtual machine networks and physical machine networks by setting flow table rules.

Chapter 7 Networks in Container Technology.

7.1 Container Network: When you are free to come and go, don’t buy an apartment to share your rent. This section is summarized as follows:

• Containers are a more lightweight isolation method than virtual machines. They mainly isolate resources through namespace and cgroup technologies. Namespace is responsible for "seeing" isolation, and cgroup is responsible for "used" isolation.
• The way a container network connects to a physical network is very similar to that of a virtual machine. The mutual access of containers on a physical machine can be achieved through bridging. If you want to access the external network, the easiest way is through NAT.

7.2 Flannel of the container network: one acre of land per person. This section is summarized as follows:

• The NAT-based container network model has two problems under the microservice architecture. One is IP address overlap, and the other is port conflict. It is necessary to maintain cross-node connectivity through the overlay network.
• Flannel is one of the cross-node container network solutions. The Overlay solution it provides mainly has two methods, one is UDP in the user mode, and the other is VXLAN in the kernel mode, and the performance of VXLAN is better.

7.3 Calico of Container Network: In order to tell white lies efficiently, this section is summarized as follows:

• Calico recommends using a physical machine as a router. This mode has no virtualization overhead and has higher performance.
• Calico's main components include routing, iptables configuration component Felix, routing and broadcasting component BGP Speaker, and BGP route reflector in large-scale scenarios.
• In order to solve the problem of cross-network segment, Calico also has an IPIP mode, that is, a tunnel is made between two machines, and the two machines are located at both ends of the tunnel. In this way, the two machines that are not neighbors, because the tunnel becomes a phase. Neighbor's machine.

7.4 Overview of RPC: Far in the sky, close in front of you, this section summarizes as follows:

• Remote calls seem to be able to use socket programming, but it is actually very complicated. It is necessary to solve the protocol agreement problem, the transmission protocol problem and the service discovery problem.
• Bruce Jay Nelson's paper, the early ONC RPC framework, and the implementation of NFS, gave an exemplary implementation to solve these three problems, that is, the agreement stipulates that a public protocol description file is required and the stub program is generated through this file. RPC transmission generally requires A state machine requires another process to do service discovery at the same time.

Chapter 8 Microservice related agreements.

8.1 XML-based SOAP: Don't talk about the NBA, please talk about the American Professional Basketball League. This section is summarized as follows:

• The original binary RPC has many shortcomings: the format is strict, the modification is too complicated, and it is not object-oriented. So a text-based calling method-SOAP based on XML was born.
• The three elements of SOAP: WSDL for protocol agreement, HTTP for transmission protocol, and UDDL for service discovery.

8.2 RESTful interface protocol based on JSON: I don't care about the process, please give me the result. This section is summarized as follows.

• SOAP is too complex, and the design is action-oriented, so the concurrency is often not increased due to architectural issues.
• RESTful is not only an API, but also an architectural model, which mainly provides stateless services for resources, which is conducive to horizontal expansion to deal with high concurrency.

8.3 Binary RPC protocol: Let's call it NBA. It's always hard to say the full name. This section is summarized as follows:

• RESTful API has basically formed a de facto standard for calls outside the access layer and Controller layer, but with the internal
• There are more and more calls between services, and performance is becoming more and more important, so Dubbo's RPC framework has its own uses.
• Dubbo solves service discovery problems through the registration center, solves the problems agreed by the agreement through Hessian2 serialization, and solves network transmission problems through Netty.

In more complex microservice scenarios, Spring Cloud's RESTful method will also be considered when calling internally. The important thing is the dependency and management of the JAR package.

8.4 Cross-language RPC protocol: Before the exchange, the two parties exchange a glossary of professional terms. This section is summarized as follows:

• gRPC is a binary, high performance, cross-language, more flexible, and at the same time, it can be used for service governance.
• The only shortcoming of the gRPC framework is to write protocol files.
• gRPC uses Protocol Buffers for serialization, HTTP 2.0 for network transmission, and Envoy-based Service Mesh for service management.

Chapter 9 Network Protocol Knowledge Series.

9.1 Knowledge stringing: use the "double *" story to string together fragmented knowledge of network protocols (Part 1),

9.2 Knowledge stringing: use the "double *" story to string together the fragmented knowledge of the network protocol (middle),

9.3 Knowledge stringing: use the "double *" story to string together the fragmented knowledge of the network protocol (below),

9.4 Build a network experiment environment: teaching people to fish is worse than teaching people to fish.

This [Interesting Network Protocol] document has a total of 435 pages. Friends who need the full version can forward this article to follow the editor, and the private message editor [Technology] to get it! !

Of course, it’s far from enough to have documents alone. There are videos and matching courseware for learning and improvement. Try to understand the computer network. I believe there will be an extraordinary life! !

TCP/IP/Network IO learning video

TCP/IP network protocol

Network IO

There are also courseware sharing

IO courseware

TCP/IP courseware

TCP/IP/IO network communication video and courseware acquisition, forwarding and following editor, private message [technology] acquisition!

Okay, I’m sharing it here today. I hope everyone can study hard and improve the computer network. I also hope that this article will be liked by everyone! !