1.1. Container network and strategy
Kubernetes itself does not implement the container network itself, but uses the CNI standard to integrate various network plug-ins in a plug-in manner to realize inter-network communication within the cluster. Anyone can write a CNI plugin, as long as they implement the core interface operations defined in the CNI standard (ADD, add a container to the network; DEL, delete a container from the network; CHECK, check whether the container's network is as expected, etc.). CNI plug-ins usually focus on container-to-container network communication. The Services network service constructed by Kubernetes is still processed by kube-proxy. The Pod service at the backend of the Service is determined through the IPtables of the host, and the network packet is forwarded to the target Pod through the CNI plug-in, as follows As shown in the figure:
The CNI interface does not refer to HTTP or gRPC interfaces. The CNI interface refers to the call (exec) executable program to the executable program. The default CNI plug-in path of the Kubernetes node is /opt/cni/bin.
CNI describes the network configuration through a configuration file in JSON format. When the container network needs to be set, the container runtime is responsible for executing the CNI plug-in, and passing the configuration file information through the standard input (stdin) of the CNI plug-in, and through the standard output (stdout) Receive the execution result of the plugin. From the network plug-in function can be divided into five categories:
(1) Main plug-in, create a specific network device (bridge: bridge device, connect container and host; ipvlan: add ipvlan network card for the container; loopback: IO device; macvlan: create a MAC address for the container; ptp: create a pair of Veth Pair; vlan: assign a vlan device; host-device: move an existing device into the container).
(2) IPAM plug-in: responsible for assigning IP addresses (dhcp: the container initiates a request to the DHCP server to issue or recycle IP addresses to the Pod; host-local: uses a pre-configured IP address segment for allocation; static: assigns a static IP address to the container IPv4/IPv6 address, mainly used for debugging).
(3) META plug-in: plug-in for other functions (tuning: adjust network device parameters through sysctl; portmap: configure port mapping through iptables; bandwidth: use Token Bucket Filter to limit current; sbr: set source based routing for network card; firewall: pass iptables restricts the incoming and outgoing traffic of the container network).
(4) Windows plug-in: CNI plug-in (win-bridge and win-overlay network plug-in) specially used for Windows platform
(5) Third-party network plug-ins: There are many third-party open source network plug-ins. Each component has its own advantages and adaptable scenarios. It is difficult to form a unified standard component. Flannel, Calico, Cilium, and OVN network plug-ins are commonly used.
Most of the CNI plug-ins are designed to follow the principle of single functional responsibility. For example, the Bridge plug-in is responsible for the configuration of the bridge, the Firewall plug-in is responsible for the configuration of the firewall, and the Portmap plug-in is responsible for the configuration of port mapping. Therefore, when the network setup is complex, it is usually done by calling multiple plugins. CNI uses multiple plug-ins through chain tuning (NetworkConfigList), and combines multiple plug-ins to call them in sequence, such as the chain call of the Flannel network plug-in CNI plug-in mentioned in the previous article when configuring the POD network.
The bottom layer of Kubernetes implements isolation of underlying basic resources through Linux Cgroup and Namespace. Each namespace (namespace) has its own network stack, including interfaces, routing tables, sockets, and IPTABLE rules. An interface can only belong to one network namespace, so multiple containers need multiple interfaces. In general, hardware resource sharing is realized through virtualization technology, and virtualized devices are connected to real physical devices. Specifically, it is divided into three types: Implementations:
(1) Virtual bridge (Virtual bridge): Create a virtual network card pair (veth pair), one end is in the root namespace of the container and the other end host, and use Linux bridge (bridge) or OpenvSwitch (OVS) to connect the two NIC pairs in different namespaces. In this way, the network data packets sent from the container can enter the host network stack through the bridge, and the network data packets sent to the container can also enter the container through the bridge.
(2) Multiplexing (Multiplexing): Use an intermediate network device to expose multiple virtual network card interfaces. The container network card can be connected to this intermediate device, and the mac address/IP address is used to distinguish whether the packet should be forwarded to a specific container. equipment.
(3) Hardware switching: assign a virtual network card to each Pod, so that the connection relationship between Pods will become very clear, because it is almost the communication basis between physical machines. Most network cards today support the SR-IOV function, which virtualizes a single physical network card into multiple VF interfaces, and each VF interface has a separate virtual PCIe channel, and these virtual PCIe channels share the PCIe channel of the physical network card.
For the Kubernetes network, network policy is also a very important capability; the network policy capability is more dependent on the specific implementation of the CNI plug-in, such as the Flannel plug-in, which does not enforce the policy at all; but most plug-ins will enforce the policy for the ingress traffic of the Pod and the egress of the Pod The traffic setting policy and the Kubernetes network policy belong to the scope of the namespace.
1.2. Calico container network plug-in
Calico is Tigera's open source network and network security solution based on the Apache 2.0 protocol. It is suitable for scenarios such as containers, virtual machines, and physical machines. Calico supports mainstream platforms including Kubernetes, OpenShift, and OpenStack. In terms of Kubernetes cloud-native container network, Calico fully complies with the CNI standard. The simplicity of Flannel is the first choice for initial users, and Calico is another good choice for its performance and flexibility. At present, the two major plug-ins of Flannel and Calico account for more than 90% of the container network plug-ins. Compared with the Flannel plug-in, Calico has more comprehensive functions, not only providing the network connection between the host and the Pod, but also involving network security and management. Starting from Calico 3.x version, the default mode of Calico is adjusted from BGP to IPIP, a more efficient Overlay mode. Calico features are as follows:
1. Efficient visual management: Calico provides comprehensive visual management, including native Linux eBPF management, standard Linux network management, and Windows HNS management. Calico provides an easy-to-use and consistent management plane by abstracting and unifying functions such as basic network, network policy, and IP address.
2. Network security strategy: Calico provides a rich network strategy model, which can easily implement network traffic management. At the same time, combined with the built-in Wireguard encryption function, it can quickly realize data transmission between Pods. There is also a Calico policy engine that can execute the same policy model on the host network and service network to realize network data risk isolation between infrastructure and upper-layer services.
3. High performance and scalability: Calico uses cutting-edge eBPF technology and deeply tunes the network pipeline of the operating system kernel to improve network performance. Calico supports many network configurations, and Overlay can not be used in most scenarios to avoid the large overhead of packet encapsulation/decapsulation. At the same time, Calico complies with the cloud-native design pattern, and the bottom layer adopts standard network protocols, which has excellent scalability.
4. Large-scale production operation practice: Calico has large-scale production environment operation practice, including SaaS providers, financial service companies and manufacturers; in terms of public cloud, Amazon EKS, Azure AKS, Google GKE and IBM IKS are all integrated Out-of-the-box Calico network security capabilities.
1.3. Calico core components
Calico's flexible network modular architecture includes four aspects: CNI network plug-in, CNI IPAM plug-in, network mode, and network strategy:
(1) CNI network plug-in: The Calico CNI network plug-in connects the Pod to the layer-3 routing of the host network namespace through a pair of virtual Ethernet devices (vethpair), avoiding the layer-2 bridge in many other Kubernetes network solutions performance overhead.
(2) CNI IPAM plug-in: Calico CNI IPAM plug-in assigns IP addresses to Pods from one or more configurable IP address ranges, and dynamically assigns small blocks of IPs to each node as needed. Calico CNI IPAM uses IP address space more efficiently than other CNI IPAM plugins.
(3) Network mode: The network mode supported by Calico is divided into two types: Overlay and Non-overlay: Overlay mode, Calico provides VXLAN or IP-in-IP network mode, including restricted cross-subnet mode (cross-subnet). In Non-overlay mode, Calico provides a Non-overlay network running on top of L2 network or L3 network.
(4) Network policy: Calico's network policy execution engine realizes all the functions of Kubernetes network policy, and adds additional extension functions.
Calico's most preferred network setting is the Non-overlay network mode that uses BGP to peer with the physical network, so that the Pod IP can be routed outside the cluster. If it is not possible to connect BGP peering to the physical network, but the cluster is located in an independent L2 network, you can also run the Non-overlay mode, which only peers BGP between nodes in the cluster, and the lack of Pod IP routing outside the application cannot be used in Out-of-cluster routing. Or set to VXLAN or IP-in-IP in Overlay mode, and use the cross-subnet Overlay mode to optimize performance within the L2 subnet.
The above picture is a complete Calico network and strategy architecture diagram, including all required and optional components, including Calico API server, Felix, BIRD, confd, Dikastes, CNI plugin, Datastore plugin, IPAM plugin, kube-controllers, Typha , calicoctl:
lCalico API server: supports managing Calico resources through kubectl.
lFelix: runs as a daemon process on each node of the cluster, mainly providing four key capabilities: interface management (Interface management), programmatic routing (Route programming), programmatic permissions (ACL programming), status report (State reporting).
lBIRD: Obtain routes from Felix and distribute them to BGP peers on the network for inter-host routing. Like Felix, it runs on every node of the cluster. It mainly provides two key capabilities: route distribution (Route distribution), route mapping configuration (BGP route reflector configuration)
lConfd: An open source, lightweight configuration management tool that stores BGP configurations and global default values, monitors data changes and dynamically generates BIRD configuration files, which will trigger BIRD to reload configuration information.
lCNI plugin: Provides Calico network capabilities for Kubernetes clusters.
lIPAM plugin: It is one of the Calico CNI plugins, which uses Calico's IP pool resources to control the allocation of IP addresses to Pods in the cluster.
lkube-controllers: Kubernetes controllers, including Policy controller, Namespace controller, Serviceaccount controller, Workloadendpoint controller, Node controller.
lcalicoctl: A command-line interface for creating, reading, updating, and deleting Calico objects.
lDatastore plugin: It increases scale by reducing the impact of each node on data storage. It is one of the Calico CNI plugins.
lTypha: Scale by reducing the impact of each node on data storage. Runs as a daemon between the datastore and the Felix instance.
lDikastes: Enhances the network strategy of the Istio service mesh and operates as a sidecar proxy of Istio Envoy.
1.4. Summary
The Kubernetes container network is relatively complex, and it needs to determine the container network solution with the underlying infrastructure and upper-layer services. At the same time, many network plug-ins support multiple modes, which require a lot of basic knowledge of the network to understand clearly. To select a suitable CNI plug-in, it is necessary to comprehensively consider the topology of the underlying network, combined with the network functions required by the application, and the requirements of the network routing protocol. Calico is a relatively mature container network plug-in with rich functions and continuous optimization in performance. Therefore, the Calico network plug-in is planned into a series of articles, from the basics of the network, the principle of Calico architecture, to the actual operation of Calico, a total of eight articles, and a deep understanding of the Calico network plug-in, so as to thoroughly grasp the container network related knowledge.