Get to know the container network interface CNI

I wrote it first. When I wrote this over the weekend, I realized that I might have dug a big hole for myself. The entire Kubernetes gateway-related content will be very complicated and huge.

A deep dive into the Kubernetes network model and network communications
Get to know the container network interface CNI (this article)
Source code analysis: looking at the use of CNI from the perspective of kubelet and container runtime
Learn Kubernetes VXLAN network from Flannel
Cilium CNI and eBPF
...

See how far you can learn~

In the article "In-depth exploration of Kubernetes network model and network communication" , we introduced how the network namespace (network namespace) works in the Kubernetes network model, and analyzed the traffic transmission path between pods through examples. The entire transfer process requires the participation of various components to complete, and these components have the same life cycle as the pod, following the creation and destruction of the pod. The maintenance of the container is completed by the kubelet entrusted to the container runtime, and the network namespace of the container is entrusted to the network plug-in by the container runtime.

Create a network namespace for pods (containers)
Create interface
Create veth pair
Set up namespace networking
Set up static routes
Configure the Ethernet bridge
Assign IP address
Create NAT rules
...

In the previous article, we also mentioned that different network plug-ins have different implementations of the Kubernetes network model, mainly focusing on the implementation of inter-pod communication across nodes. Users can choose the appropriate network plug-in according to their needs, which is inseparable from CNI (container network interface). These network plug-ins all implement the CNI standard and can be well integrated with container orchestration systems and runtimes.

runtime-with-cni

What is CNI

CNI is a project under CNCF. In addition to providing the most important specifications , libraries for integrating CNI with applications , CLI for implementing CNI plug-ins cnitool, and reference plug-ins . At the time of publishing this article, the latest version is v1.1.2.

CNI only focuses on the network connection of the container and cleaning up/releasing the allocated resources when the container is destroyed. Because of this, even if the container develops rapidly, CNI can still be simple and widely supported .

CNI specification

The CNI specification covers the following parts:

Network configuration file format
Protocol for container runtimes to interact with network plugins
Plug-in execution process
Will delegate the execution process of other plug-ins
The execution result data type returned to the runtime

1. Network configuration format

Configuration examples in the specification are posted here. The specification defines the format of network configuration, including required fields, optional fields, and the functions of each field. The example uses defines dbneta network named and configures the plugins bridgeand tuning.

CNI plug-ins are generally divided into two types:

Interface plugin: Used to create network interfaces, such as the one in the example bridge.
Chained: used to adjust the created network interface, such as in the example tuning.

{
  "cniVersion": "1.0.0",
  "name": "dbnet",
  "plugins": [
    {
      "type": "bridge",
      // plugin specific parameters
      "bridge": "cni0",
      "keyA": ["some more", "plugin specific", "configuration"],
      
      "ipam": {
        "type": "host-local",
        // ipam specific
        "subnet": "10.1.0.0/16",
        "gateway": "10.1.0.1",
        "routes": [
            {"dst": "0.0.0.0/0"}
        ]
      },
      "dns": {
        "nameservers": [ "10.1.0.1" ]
      }
    },
    {
      "type": "tuning",
      "capabilities": {
        "mac": true
      },
      "sysctl": {
        "net.core.somaxconn": "500"
      }
    },
    {
        "type": "portmap",
        "capabilities": {"portMappings": true}
    }
  ]
}

2. Protocol for interaction between container runtime and network plug-in

CNI provides four distinct operations for container runtimes :

ADD - Add a container to the network, or modify the configuration
DEL - Remove the container from the network, or cancel modifications
CHECK - checks whether the container network is normal, and returns an error if there is a problem with the container's network
VERSION - displays the version of the plugin

A specification defines the input and output of an operation. The main core fields are:

CNI_COMMAND: One of the four operations above
CNI_CONTAINERID:Container ID
CNI_NETNS: The isolation domain of the container. If a network namespace is used, the value here is the address of the network namespace
CNI_IFNAME: The name of the interface to be created in the container, such aseth0
CNI_ARGS: Parameters passed when executing parameters
CNI_PATH: Chaozhao path of the plug-in executable file

3. Plug-in execution process

CNI turns the network configuration ADD, DELETEand CHECKoperations on the container into attachments .

The operation of container network configuration requires the joint operation of one or more plug-ins, so plug-ins have a certain order of execution. For example, in the previous example configuration, the interface must be created first before the interface can be tuned.

Taking ADDthe operation as an example, it is usually executed first interface plugin, and then executed chained plugin. The output of the previous plug- in PrevResultand the configuration of the next plug-in will be used as the input of the next plug-in . If it is the first plugin, the network configuration will be included as part of the input. The plug-in can use the previous plug-in's PrevResultas its own output, or it can PrevResultupdate it in conjunction with its own operations. The output of the last plug-in PrevResultis returned to the container runtime as the CNI execution result, and the container runtime saves the result and uses it as input for other operations .

DELETEThe execution ADDorder of is exactly the opposite to that of . You must first remove the configuration on the interface or release the assigned IP, and finally delete the container network interface. DELETEThe input of the operation is the result of the operation saved when the container is running ADD.

cni-plugin-execution-flow

In addition to defining the execution order of plugins in a single operation, CNI also specifies the parallel operation of operations, repeated operations, etc.

4. Plug-in delegation

There are some operations that, for whatever reason, cannot reasonably be implemented as a loosely linked plugin. Conversely, a CNI plugin may wish to delegate some functionality to another plugin. A common example is IP Address Management (IPAM), which mainly allocates/recycles IP addresses for container interfaces, manages routing, etc.

CNI defines a third type of plug-in - IPAM plug-in. The CNI plug-in can call the IPAM plug-in at the appropriate time, and the IPAM plug-in will return the execution result to the client. The IPAM plug-in will complete operations based on the specified protocol (such as dhcp), data in local files, or ipamfield information in the network configuration file: assign IP, set gateway, routing, etc.

"ipam": {
  "type": "host-local",
  // ipam specific
  "subnet": "10.1.0.0/16",
  "gateway": "10.1.0.1",
  "routes": [
      {"dst": "0.0.0.0/0"}
  ]
}

5. Execution results

Plugins can return one of three results, the format of which is defined by the specification.

Success: It will also contain PrevResultinformation, such as returned to the container runtime after ADDthe operation .PrevResult
Error: Contains necessary error message.
Version: This is VERSIONthe return result of the operation.

Library

The CNI library refers libcnito the integration of CNI and application programs, which defines CNI-related interfaces and configurations.

type CNI interface {  
   AddNetworkList(ctx context.Context, net *NetworkConfigList, rt *RuntimeConf) (types.Result, error)  
   CheckNetworkList(ctx context.Context, net *NetworkConfigList, rt *RuntimeConf) error  
   DelNetworkList(ctx context.Context, net *NetworkConfigList, rt *RuntimeConf) error  
   GetNetworkListCachedResult(net *NetworkConfigList, rt *RuntimeConf) (types.Result, error)  
   GetNetworkListCachedConfig(net *NetworkConfigList, rt *RuntimeConf) ([]byte, *RuntimeConf, error)  
  
   AddNetwork(ctx context.Context, net *NetworkConfig, rt *RuntimeConf) (types.Result, error)  
   CheckNetwork(ctx context.Context, net *NetworkConfig, rt *RuntimeConf) error  
   DelNetwork(ctx context.Context, net *NetworkConfig, rt *RuntimeConf) error  
   GetNetworkCachedResult(net *NetworkConfig, rt *RuntimeConf) (types.Result, error)  
   GetNetworkCachedConfig(net *NetworkConfig, rt *RuntimeConf) ([]byte, *RuntimeConf, error)  
  
   ValidateNetworkList(ctx context.Context, net *NetworkConfigList) ([]string, error)  
   ValidateNetwork(ctx context.Context, net *NetworkConfig) ([]string, error)  
}

Take the code for adding a network as an example:

func (c *CNIConfig) addNetwork(ctx context.Context, name, cniVersion string, net *NetworkConfig, prevResult types.Result, rt *RuntimeConf) (types.Result, error) {  
   ...
   return invoke.ExecPluginWithResult(ctx, pluginPath, newConf.Bytes, c.args("ADD", rt), c.exec)  
}

The execution logic is simply:

Find executable file
Load network configuration
Execute ADDaction
Result processing

Summarize

This article has learned the content of the CNI specification and the execution process of the network plug-in, and has a general understanding of the abstract network management interface of the CNI.

The next article will combine source code analysis to understand how kubelet, container runtime, and CNI network plug-ins interact.

reference

Articles are published uniformly on the public account云原生指北