Flannel is developed by coreOS and is used to solve the overlay network (overlay network) of docker cluster cross-host communication . Its main idea is: set aside a network segment in advance, each host uses a part of it, and then each container is assigned a different ip ; Let all containers think that everyone is in the same directly connected network, and the bottom layer encapsulates and forwards packets through UDP/VxLAN/Host-GW , etc.

Flannel is essentially an "overlay network" , that is, it packs TCP data in another network packet for routing, forwarding and communication. It currently supports udp, vxlan, host-gw, aws-vpc, gce and Alloc routing and other data forwarding methods, the default inter-node data communication method is UDP forwarding.

Its function is to allow Docker containers created by different node hosts in the cluster to have unique virtual IP addresses for the entire cluster.

The design purpose of Flannel is to re-plan the IP address usage rules for all nodes in the cluster , so that containers on different nodes can obtain IP addresses that belong to the same intranet and do not overlap, and allow containers on different nodes to directly pass Intranet IP communication.

1.1 Implementation principle of Flannel

On different nodes in the cluster, the Pods created have virtual IP addresses unique to the entire cluster.
Establish an overlay network (overlay network), through which the data packet is delivered to the target container intact. An overlay network decouples network services from the underlying infrastructure by encapsulating one packet within another. After forwarding the encapsulated packet to the endpoint, it is decapsulated.
Create a new virtual network card flannel0 to receive the data of the docker bridge, and package and forward the received data by maintaining the routing table (vxlan).
etcd ensures that the configuration seen by flanned on all nodes is consistent. At the same time, flanned on each node monitors data changes on etcd and perceives changes in nodes in the cluster in real time.

1.2 Data forwarding process

The container directly uses the ip of the target container to access, which is sent through eth0 inside the container by default.
The message is sent to vethXXX through veth pair.
vethXXX is directly connected to the virtual switch docker0, and the message is sent out through the virtual bridge docker0.
Look up the routing table, and the packets of the external container ip will be forwarded to the flannel0 virtual network card, which is a P2P virtual network card, and then the packets will be forwarded to flanneld listening on the other end.
flanneld maintains the routing table between each node through etcd, encapsulates the original message with UDP, and sends it out through the configured iface.
The packet finds the target host through the network between the hosts.
The message continues to go up to the transport layer, and is handed over to the flanneld program listening on port 8285 for processing.
The data is unpacked and sent to the flannel0 virtual NIC.
Look up the routing table and find that the packets corresponding to the container should be handed over to docker0.
docker0 finds the container connected to itself and sends the message to it.

Simple understanding:

The container 10.1.15.2/24 accesses 10.0.20.3/24, passes through the docker0 of the host, which is equivalent to the gateway of this container, and forwards it to the flannel0 virtual network card, which is processed by flanneld and encapsulates the ip of the source container and the ip of the destination container Into the internal ip, the gateway of the source physical network card and the gateway of the destination physical network card are encapsulated into an external ip, and its MAC address is encapsulated at the same time, the flanneld service of the source host encapsulates the original data content UDP and delivers it to the destination node according to its own routing table The flanneld service, the data is unpacked after arriving, and then enters the flannel0 virtual network card of the destination node, and then is forwarded to the docker0 virtual network card of the destination host, and passed to the container corresponding to the ip

Two: Flannel network overview

Overlay Network:
Overlay network, a virtual network technology model superimposed on the basic network, the hosts in the network are connected through virtual links,
similar to VPN tunnels, the principle is a logical network implemented on a physical network

VXLAN:
Encapsulate the source data packet into UDP, and use the IP/MAC of the basic network as the outer packet header for encapsulation, and then transmit it on the Ethernet. After reaching the destination, the tunnel will power off and unencapsulate and send the data to the target address

Flannel:
It is a kind of Overlay network. It also encapsulates the source data packet in another network packet for routing forwarding and communication. Currently, it supports data forwarding methods such as UDP, VXLAN, AWS VPN, and GCE routing.

features
hostgw	direct routing
vxlan	It is the way recommended by flannel. Network devices that need to communicate can support the vxlan protocol
udp	This method is very similar to vxlan, which wraps the ip layer network. Usually used in a debugging environment or in a network environment that does not support the vxlan protocol

2.1 Vxlan mode

2.1.1 Communication process

Pod communication process on different nodes:

The data in the pod is sent to the bridge cni0 according to the routing information of the pod
cni0 sends data to the tunnel device flannel.1 according to the node routing table
flannel.1 View the destination ip of the data packet, obtain the necessary information of the peer tunnel device from flanneld, and encapsulate the data packet
flannel.1 sends the packet to the peer device. The network card of the peer node receives the data packet, finds that the data packet is an overlay data packet, unpacks the outer layer, and sends the inner layer to the flannel.1 device
The Flannel.1 device checks the data packet, matches it according to the routing table, and sends the data to the cni0 device
cni0 matches the routing table and sends data to the bridge

2.1.2 Deployment

$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
 
# 配置 Pod CIDR 
$ vi kube-flannel.yml
  "Network": "10.244.0.0/16", 
  
# 多网卡时，可指定网卡
vi kube-flannel.yml
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=ens38    # 指定网卡
        
$ kubectl apply -f kube-flannel.yml
 
$ kubectl get pod -n kube-system
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-8qnnx   1/1     Running   0          10s
kube-flannel-ds-979lc   1/1     Running   0          16m
kube-flannel-ds-kgmgg   1/1     Running   0          16m

Network distribution on cluster nodes:

$ ip addr
6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether b6:95:2a:cd:01:c3 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.0/32 brd 10.244.0.0 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::b495:2aff:fecd:1c3/64 scope link
       valid_lft forever preferred_lft forever
7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 16:ac:e9:68:a4:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.1/24 brd 10.244.0.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::14ac:e9ff:fe68:a4c0/64 scope link
       valid_lft forever preferred_lft forever
 
$ ethtool -i cni0
driver: bridge
 
$ ethtoo -i flannel.1
driver: vxlan
 
$ ps -ef | grep flanneld
root       15300   15275  0 10:21 ?        00:00:19 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
 
$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.80.2    0.0.0.0         UG    0      0        0 ens33
10.244.0.0      10.244.0.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.1.0      10.244.1.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.2.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
192.168.80.0    0.0.0.0         255.255.255.0   U     0      0        0 ens33
 
$ brctl show
bridge name     bridge id               STP enabled     interfaces
cni0            8000.e2ee89678398       no              veth28b04daf
                                                        vethe6d4a6b8

cni0: bridge device, each time a pod is created, a pair of veth pairs will be created. One segment is eth0 in the pod, and the other end is a port in the cni0 bridge.

flannel.1: vxlan gateway device, unpacks and packs user vxlan packets. Different pod data traffic is sent from the overlay device to the peer end in the form of a tunnel. flannel.1 will not send an arp request to obtain the mac address of the target IP. Instead, the Linux kernel will send a "L3 Miss" event request to the flanneld program in user space. Find the mac address of the subnet flannel.1 device that can match this address, that is, the mac address of the flannel.1 device in the host where the target pod is located.

flanneld: run flanneld in each host as an agent, it will obtain a small network segment subnet from the network address space of the cluster for the host where it is located, and the IP addresses of all containers in the host will be allocated from it. At the same time, Flanneld monitors the K8s cluster database and provides network data information such as mac and ip necessary for flannel.1 devices to encapsulate data.

VXLAN: Virtual eXtensible Local Area Network, Virtual Extended Local Area Network. L2 over L4 (MAC-in-UDP) packet encapsulation mode is adopted to encapsulate Layer 2 packets with Layer 3 protocols, so as to realize the expansion of Layer 2 networks within the scope of Layer 3, and at the same time meet the large Layer 2 virtual migration of data centers and multi-tenant requirements.

flannel only uses some functions of vxlan, and VNI is fixed to 1. Container cross-network communication solution: If the hosts of the cluster are in the same subnet, they will be forwarded through routing; if they are not in the same subnet, they will be forwarded through tunnels.

2.1.3 Related configuration

$ cat /etc/cni/net.d/10-flannel.conflist
{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
 
$ cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
 
# Bridge CNI 插件
$ cat /var/lib/cni/flannel/462cf658ef71d558b36884dfb6d068e100a3209d36ba2602ad04dd9445e63684 | python3 -m json.tool
{
    "cniVersion": "0.3.1",
    "hairpinMode": true,
    "ipMasq": false,
    "ipam": {
        "routes": [
            {
                "dst": "10.244.0.0/16"
            }
        ],
        "subnet": "10.244.2.0/24",
        "type": "host-local"
    },
    "isDefaultGateway": true,
    "isGateway": true,
    "mtu": 1450,
    "name": "cbr0",
    "type": "bridge"
}

2.1.4 Uninstall

# 主节点
kubectl delete -f kube-flannel.yml
 
# 所有节点上
ip link set cni0 down
ip link set flannel.1 down
 
ip link delete cni0
ip link delete flannel.1
 
rm -rf /var/lib/cni/
rm -f /etc/cni/net.d/*

2.2 Host-GW mode

2.2.1 Communication process

host-gw adopts a purely static routing method, requiring all hosts to be in a local area network, and routing across local area networks cannot be performed. If cross-LAN routing is required, routing needs to be added on other devices, but this is beyond the capabilities of flannel. You can choose to use dynamic routing technology such as calico, and announce the local route through the broadcast route, so as to realize cross-LAN route learning.

All subnet and host information are stored in Etcd, and flanneld only needs to watch the changes of these data and update the routing table in real time.
Core: When an IP packet is encapsulated into a frame, it uses the MAC address on the "next hop" setting of the routing table, so that it can reach the destination host through the Layer 2 network.

2.2.2 Deployment

$ vi kube-flannel.yml
      "Backend": {
        "Type": "host-gw"
      }
 
$ kubectl apply -f kube-flannel.yml
 
$ kubectl get pod -n kube-system
NAMESPACE     NAME                    READY   STATUS    RESTARTS   AGE
kube-system   kube-flannel-ds-l2dg7   1/1     Running   0          7s
kube-system   kube-flannel-ds-tj2vg   1/1     Running   0          7s
kube-system   kube-flannel-ds-xxhfm   1/1     Running   0          7s

Network distribution on cluster nodes:

$ ip addr
7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 2a:00:05:23:3f:5e brd ff:ff:ff:ff:ff:ff
    inet 10.244.2.1/24 brd 10.244.2.255 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::2800:5ff:fe23:3f5e/64 scope link
       valid_lft forever preferred_lft forever
 
$ kubectl logs kube-flannel-ds-l2dg7 -n kube-system
I1227 12:09:56.991787       1 route_network.go:86] Subnet added: 10.244.2.0/24 via 192.168.80.240
I1227 12:09:56.992305       1 route_network.go:86] Subnet added: 10.244.0.0/24 via 192.168.80.241
 
$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.80.2    0.0.0.0         UG    0      0        0 ens33
10.244.0.0      192.168.80.241  255.255.255.0   UG    0      0        0 ens33
10.244.1.0      192.168.80.242  255.255.255.0   UG    0      0        0 ens33
10.244.2.0      0.0.0.0         255.255.255.0   U     0      0        0 cni0
192.168.80.0    0.0.0.0         255.255.255.0   U     0      0        0 ens33

2.2.3 Related configuration

$ cat /etc/cni/net.d/10-flannel.conflist
{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}
 
$ cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1500  # 路由方式下，MTU值使用默认值
FLANNEL_IPMASQ=true
 
# Bridge CNI 插件
$ cat /var/lib/cni/flannel/46c76c1d50d61494d6d95e0171667ec705bbcdcaeeafa859e25ac4749979bd76 | python3 -m json.tool
{
    "cniVersion": "0.3.1",
    "hairpinMode": true,
    "ipMasq": false,
    "ipam": {
        "ranges": [
            [
                {
                    "subnet": "10.244.2.0/24"
                }
            ]
        ],
        "routes": [
            {
                "dst": "10.244.0.0/16"
            }
        ],
        "type": "host-local"
    },
    "isDefaultGateway": true,
    "isGateway": true,
    "mtu": 1500,
    "name": "cbr0",
    "type": "bridge"
}

2.2.4 Uninstall

# 主节点
kubectl delete -f kube-flannel.yml
 
# 所有节点上
ip link set cni0 down
ip link delete cni0
 
rm -rf /var/lib/cni/
rm -f /etc/cni/net.d/*

2.3 The role of Flanneld

After receiving the EventAdded event, Flanneld configures various information reported by other hosts from etcd on the local machine, mainly divided into the following three types of information:

ARP: Correspondence between IP and MAC, three-layer forwarding
FDB: The corresponding relationship between MAC+VLAN and PORT, Layer 2 forwarding, even if the two devices are not in the same network segment or IP is not configured, as long as the link layer between the two devices is connected, data can be forwarded through the FDB table. Its function is to tell the device to go out from a certain port to a certain destination MAC
Routing Table: Packets to the destination address are sent out through the gateway

2.4 Mode Comparison

udp mode: use the device flannel.0 for packet unpacking, not natively supported by the kernel, large context switching, and very poor performance
vxlan mode: Use flannel.1 for packet unpacking, the kernel supports natively, and the performance loss is about 20%~30%
host-gw mode: no intermediate device such as flannel.1 is needed, the host machine is directly used as the next hop address of the subnet, and the performance loss is about 10%

Three: deployment

First, the two node nodes need to install the docker engine first

安装依赖包
yum install -y yum-utils device-mapper-persistent-data lvm2
 
设置阿里云镜像源
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
 
安装Docker-CE
vim /etc/selinux/config
SELINUX=disabled
 
yum install -y docker-ce
 
systemctl start docker.service
systemctl enable docker.service

mirror acceleration

Search container mirroring services

网络优化
 
echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf
sysctl -p
 
systemctl restart network
systemctl restart docker

master configuration

You need to use this command in a directory with a certificate
to write the assigned subnet segment to ETCD for use by flannel

[root@localhost ~]# cd k8s/etcd-cert/
 
[root@localhost ~]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.111.10:2379,https://192.168.111.30:2379,https://192.168.111.40:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'

拷贝到所有node节点（只需要部署在node节点即可）
[root@localhost k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz [email protected]:/root
[root@localhost k8s]# scp flannel-v0.10.0-linux-amd64.tar.gz [email protected]:/root

All node node operation decompression

[root@localhost ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz 
 
flanneld
mk-docker-opts.sh
README.md

k8s工作目录
[root@localhost ~]# mkdir /opt/kubernetes/{cfg,bin,ssl} -p
[root@localhost ~]# mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/

编写flannel启动脚本
[root@localhost ~]# vim flannel.sh
#!/bin/bash
 
ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}
 
cat <<EOF >/opt/kubernetes/cfg/flanneld
 
FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"
 
EOF
 
cat <<EOF >/usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
 
[Install]
WantedBy=multi-user.target
 
EOF
 
systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld

开启flannel网络功能
[root@localhost ~]# bash flannel.sh https://192.168.111.10:2379,https://192.168.111.30:2379,https://192.168.111.40:2379

配置docker连接flannel
[root@localhost ~]# vim /usr/lib/systemd/system/docker.service
 
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
 
 
systemctl daemon-reload
systemctl restart docker

验证下docker网络和flannel网络是否在同一个网段
ifconfig

Pull the image and enter the container to ping another container on a different network segment

docker run -it centos:7 /bin/bash
yum -y install net-tools
 
ping 172.17.39.1

If you get stuck when pinging here, you need to solve it in this way
All node nodes restart the service in order

systemctl restart flanneld.service
systemctl restart network
systemctl restart docker

Four: Failure Analysis

4.1 `kube-proxy 配置错误`

4.1.1 Phenomenon:

root@k8s-master:~# kubectl get pod -A -o wide -l app=flannel
NAMESPACE     NAME                    READY   STATUS             RESTARTS   AGE     IP              NODE         NOMINATED NODE   READINESS GATES
kube-system   kube-flannel-ds-5whpv   0/1     CrashLoopBackOff   5          6m53s   192.168.3.114   k8s-node01   <none>           <none>
kube-system   kube-flannel-ds-l7msr   1/1     Running            2          16d     192.168.3.113   k8s-master   <none>           <none>
kube-system   kube-flannel-ds-rvvhv   0/1     CrashLoopBackOff   10         33m     192.168.3.115   k8s-node02   <none>           <none>
root@k8s-master:~# kubectl logs  kube-flannel-ds-5whpv -n kube-system
I0211 02:04:21.358127       1 main.go:520] Determining IP address of default interface
I0211 02:04:21.359211       1 main.go:533] Using interface with name enp1s0 and address 192.168.3.118
I0211 02:04:21.359295       1 main.go:550] Defaulting external address to interface address (192.168.3.118)
W0211 02:04:21.359364       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E0211 02:04:51.456912       1 main.go:251] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-5whpv': Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-5whpv": dial tcp 10.96.0.1:443: i/o timeout
root@k8s-master:~# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   16d

4.1.2 Troubleshoot node k8s-node01:

root@k8s-node01:~# ip addr show kube-ipvs0
5: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
    link/ether a6:99:85:4e:ba:35 brd ff:ff:ff:ff:ff:ff
	inet 10.96.0.1/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
 
# 本地节点无法连接到kube-apiserver, 说明是kube-proxy 故障
root@k8s-node01:~# telnet 10.96.0.1 443
Trying 10.96.0.1...
telnet: Unable to connect to remote host: Connection timed out
 
# 检查 kube-proxy 日志，发现其配置有问题
root@k8s-node01:/var/log/kubernetes# vi kube-proxy.ERROR
...
E0211 09:13:56.135807    1842 node.go:161] Failed to retrieve node info: Get "https://192.168.3.113:6443/api/v1/nodes/k8s-node01": dial tcp 192.168.3.113:6443: connect: connection refused
 
root@k8s-node01:/var/log/kubernetes# vi kube-proxy.WARNING
...
: v1alpha1.KubeProxyConfiguration.IPTables: v1alpha1.KubeProxyIPTablesConfiguration.MasqueradeBit: ReadObject: found unknown field: masqueradeAl, error found in #10 byte of ...|queradeAl":"","masqu|..., bigger context ...|eOverride":"k8s-node01","iptables":{"masqueradeAl":"","masqueradeBit":14,"minSyncPeriod":"5s","syncP|...
 
# 修改 Kube-proxy 配置
vi /etc/kubernetes/kube-proxy-config.yml
...
iptables:
  masqueradeAll: true
 
# 重启 Kube-proxy
root@k8s-node01:/var/log/kubernetes# systemctl restart kube-proxy
 
# 再次检查，已正常
root@k8s-node01:/var/log/kubernetes# vi kube-proxy.INFO
...
I0211 10:25:30.297232    2754 service.go:421] Adding new service port "default/kubernetes:https" at 10.96.0.1:443/TCP
...
I0211 10:32:52.626926    3155 proxier.go:1034] Not syncing ipvs rules until Services and Endpoints have been received from master
 
# 重启flannel
root@k8s-master:~# kubectl delete pod kube-flannel-ds-5whpv -n kube-system
root@k8s-master:~# kubectl get pod -A -o wide -l app=flannel
NAMESPACE     NAME                    READY   STATUS             RESTARTS   AGE     IP              NODE         NOMINATED NODE   READINESS GATES
kube-system   kube-flannel-ds-lld4b   0/1     Running            0          6m53s   192.168.3.114   k8s-node01   <none>           <none>
kube-system   kube-flannel-ds-l7msr   1/1     Running            2          16d     192.168.3.113   k8s-master   <none>           <none>
kube-system   kube-flannel-ds-rvvhv   0/1     CrashLoopBackOff   10         33m     192.168.3.115   k8s-node02   <none>           <none>
 
# 确认OK
root@k8s-node01:/var/log/kubernetes# vi kube-proxy.INFO
I0211 02:36:07.555531       1 main.go:520] Determining IP address of default interface
I0211 02:36:07.556543       1 main.go:533] Using interface with name enp1s0 and address 192.168.3.118
I0211 02:36:07.556615       1 main.go:550] Defaulting external address to interface address (192.168.3.118)
W0211 02:36:07.556688       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0211 02:36:08.057730       1 kube.go:116] Waiting 10m0s for node controller to sync
I0211 02:36:08.057858       1 kube.go:299] Starting kube subnet manager
I0211 02:36:09.058115       1 kube.go:123] Node controller sync successful
I0211 02:36:09.058511       1 main.go:254] Created subnet manager: Kubernetes Subnet Manager - k8s-node01
I0211 02:36:09.058524       1 main.go:257] Installing signal handlers
I0211 02:36:09.152670       1 main.go:392] Found network config - Backend type: host-gw
I0211 02:36:09.254550       1 main.go:357] Current network or subnet (10.244.0.0/16, 10.244.0.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I0211 02:36:09.853007       1 iptables.go:172] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I0211 02:36:09.858096       1 iptables.go:172] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0211 02:36:09.952777       1 iptables.go:172] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I0211 02:36:09.955497       1 iptables.go:172] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j MASQUERADE --random-fully
I0211 02:36:09.962242       1 main.go:307] Setting up masking rules
I0211 02:36:09.964711       1 main.go:315] Changing default FORWARD chain policy to ACCEPT
I0211 02:36:09.965035       1 main.go:323] Wrote subnet file to /run/flannel/subnet.env
I0211 02:36:09.965050       1 main.go:327] Running backend.
I0211 02:36:09.965069       1 main.go:345] Waiting for all goroutines to exit
I0211 02:36:09.965099       1 route_network.go:53] Watching for new subnet leases
I0211 02:36:09.965579       1 route_network.go:86] Subnet added: 10.244.2.0/24 via 192.168.3.117
I0211 02:36:09.966182       1 route_network.go:86] Subnet added: 10.244.1.0/24 via 192.168.3.119
I0211 02:36:10.152723       1 iptables.go:148] Some iptables rules are missing; deleting and recreating rules
I0211 02:36:10.152782       1 iptables.go:172] Deleting iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0211 02:36:10.153844       1 iptables.go:148] Some iptables rules are missing; deleting and recreating rules
I0211 02:36:10.153886       1 iptables.go:172] Deleting iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0211 02:36:10.155194       1 iptables.go:172] Deleting iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0211 02:36:10.156970       1 iptables.go:172] Deleting iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0211 02:36:10.252675       1 iptables.go:172] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
I0211 02:36:10.255063       1 iptables.go:160] Adding iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0211 02:36:10.255399       1 iptables.go:172] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
I0211 02:36:10.353644       1 iptables.go:160] Adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0211 02:36:10.452443       1 iptables.go:160] Adding iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0211 02:36:10.456201       1 iptables.go:160] Adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0211 02:36:10.555203       1 iptables.go:160] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
I0211 02:36:10.655121       1 iptables.go:160] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully

4.2 In ipvs mode, the routing table is wrong

# 无法访问 kube-apiserver
$ kubectl logs -f kube-flannel-ds-ntwh4 -n kube-system
I1108 10:48:00.864770       1 main.go:520] Determining IP address of default interface
I1108 10:48:00.865795       1 main.go:533] Using interface with name ens33 and address 192.168.80.241
I1108 10:48:00.865827       1 main.go:550] Defaulting external address to interface address (192.168.80.241)
W1108 10:48:00.865861       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
E1108 10:48:30.960762       1 main.go:251] Failed to create SubnetManager: error retrieving pod spec for 'kube-system/kube-flannel-ds-ntwh4': Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/pods/kube-flannel-ds-ntwh4": dial tcp 10.96.0.1:443: i/o timeout
 
# 切换到相关主机，测试网络，无法连通
$ curl 10.96.0.1:443
 
# 查询网络
$ ip addr
3: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
    link/ether c6:14:f7:ad:b6:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.96.92.170/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.96.0.1/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
       
# 尝试删除 ip, 发现删除后，自动生成；进一步发现，该设备默认down，设置IP地址，不影响通信
$ ip addr delete 10.96.0.1/32 dev kube-ipvs0
 
# 查询路由表，发现路由表错误
$ ip route show table local
local 10.96.0.1 dev kube-ipvs0 proto kernel scope host src 10.96.0.1
 
# 删除路由表
$ ip route del table local local 10.96.0.1 dev kube-ipvs0 proto kernel scope host src 10.96.0.1

Kubernetes: (18) flannel network

One: What is Flannel

1.1 Implementation principle of Flannel

1.2 Data forwarding process

Two: Flannel network overview

2.1 Vxlan mode

2.1.1 Communication process

2.1.2 Deployment

2.1.3 Related configuration

2.1.4 Uninstall

2.2 Host-GW mode

2.2.1 Communication process

2.2.2 Deployment

2.2.3 Related configuration

2.2.4 Uninstall

2.3 The role of Flanneld

2.4 Mode Comparison

Three: deployment

Four: Failure Analysis

4.1 `kube-proxy 配置错误`

4.1.1 Phenomenon:

4.1.2 Troubleshoot node k8s-node01:

4.2 In ipvs mode, the routing table is wrong

Guess you like

Kubernetes: (18) flannel network

One: What is Flannel

1.1 Implementation principle of Flannel

1.2 Data forwarding process

Two: Flannel network overview

2.1 Vxlan mode

2.1.1 Communication process

2.1.2 Deployment

2.1.3 Related configuration

2.1.4 Uninstall

2.2 Host-GW mode

2.2.1 Communication process

2.2.2 Deployment

2.2.3 Related configuration

2.2.4 Uninstall

2.3 The role of Flanneld

2.4 Mode Comparison

Three: deployment

Four: Failure Analysis

4.1 kube-proxy 配置错误

4.1.1 Phenomenon:

4.1.2 Troubleshoot node k8s-node01:

4.2 In ipvs mode, the routing table is wrong

Guess you like

4.1 `kube-proxy 配置错误`