k8s manually deploy flannel (k8s is based on the cnm network plug-in)

Environmental preparation

Install etcd, and configure the key to store network information

Copy the etcd key file to each node including the master node (and authorize)

scp /opt/etcd/ssl/* [email protected]:/opt/etcd/ssl
chmod 755 /opt/etcd/ssl

Since flannel does not support etcd-v3, you need to enable etcd-v2 version

#[Clustering]
ETCD_ENABLE_V2="true"

systemctl restart etcd

Store flannel network information

vi flannel-config.json
{
    
    
  "Network": "10.2.0.0/16",
  "SubnetLen": 24,
  "Backend": {
    
    
    "Type": "vxlan"
  }
}
#Network 定义该网络的 IP 池为 10.2.0.0/16
#SubnetLen 指定每个主机分配到的 subnet 大小为 24 位，即10.2.X.0/24
#Backend 为 vxlan，即主机间通过 vxlan 通信

export ETCDCTL_API=2
#环境变量

etcdctl --endpoints=$ENDPOINTS --ca-file="/opt/etcd/ssl/ca.pem" --cert-file="/opt/etcd/ssl/server.pem" --key-file="/opt/etcd/ssl/server-key.pem" set /docker-flannel/network/config < flannel-config.json
#注意命令跟v3有区别

Download flannel, configure and run

wget https://github.com/coreos/flannel/releases/download/v0.13.1-rc1/flannel-v0.13.1-rc1-linux-amd64.tar.gz
tar -zxvf flannel-v0.13.1-rc1-linux-amd64.tar.gz
mv flanneld /usr/local/bin/
scp flannel [email protected]:/usr/local/bin/
#拷贝到其他节点和主节点

flanneld -etcd-endpoints="https://192.168.12.10:2379,https://192.168.12.11:2379,https://192.168.12.12:2379" -etcd-cafile="/opt/etcd/ssl/ca.pem" -etcd-certfile="/opt/etcd/ssl/server.pem" -etcd-keyfile="/opt/etcd/ssl/server-key.pem"  -etcd-prefix=/docker-flannel/network  -iface=eth0
#-etcd-endpoints 指定 etcd url
#-iface 指定主机间数据传输使用的 interface
#-etcd-prefix 指定 etcd 存放 flannel 网络配置信息的 key

Modify docker network

vi /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --bip=10.2.94.1/24 --mtu=1450
#flanneld运行的时候会生成一个文件在/run/flannel/subnet.env记录了网络信息
#--bip=/run/flannel/subnet.env里的DOCKER_OPT_BIP
#--mtu=/run/flannel/subnet.env里的DOCKER_OPT_MTU

systemctl daemon-reload
systemctl restart docker
#查看docker0接口地址发生了改变

Insert picture description here
flanneld is inconvenient to manage and needs to be changed to systemctl for management

vi /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/usr/local/bin/flanneld \
  -etcd-cafile=/opt/etcd/ssl/ca.pem \
  -etcd-certfile=/opt/etcd/ssl/server.pem \
  -etcd-keyfile=/opt/etcd/ssl/server-key.pem \
  -etcd-endpoints=https://192.168.12.10:2379,https://192.168.12.11:2379,https://192.168.12.12:2379 \
  -etcd-prefix=/docker-flannel/network \
  -iface=eth0 \
  -ip-masq
ExecStartPost=/usr/local/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target
WantedBy=docker.service
#创建systemctl服务文件，Service部分是跟命令启动类似的只要替换即可
#-ip-masq不开启flannel.1 接口的源nat，避免访问不是docker的真实IP
#ExecStartPost=/usr/local/flannel/mk-docker-opts.sh这个脚本文件在flannel下载解压后会有，指定到具体目录即可

Modify docker network

vi /usr/lib/systemd/system/docker.service
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock $DOCKER_NETWORK_OPTIONS
#flanneld运行的时候会生成一个文件在/run/flannel/subnet.env记录了网络信息，docker只要每次重启获取文件内里的信息

systemctl daemon-reload
systemctl restart docker

Insert picture description here
Observing the route, you can see that each host is a subnet, and you can access the other through the route. If it is in vxlan mode, tunnel transmission is carried out. Flannel also supports host-wg host gateway mode

The systemctl service file will not write, you can install flannel with yum, then save a copy of the systemctl service file and modify it after uninstalling.
Part of the systemctl service file reference: https://www.cnblogs.com/love19791125/p/11283633.html

Question: It can be seen that they are all errors, but in fact, there is no problem. My version is v0.13.1-rc1 (the command start also has an error prompt)
Insert picture description here
2021.02.02 update

Since the built k8s did not use the CNI plug-in, but instead used the CNM that came with docker, the node appeared for more than 24 hours during use, and the flannel address was changed after re-linking. It was shocked. This would cause the Pod network to fail, unless Rebuild the Pod

Checked flannel and found that there is a problem of renewing the lease. The renewal time is 24 hours. If it exceeds 24 hours, the FLANNEL_SUBNET parameter value under /run/flannel/subnet.env will be read for etcd registration. If an IP conflict occurs, it will be re-allocated. New subnet to node

1. Did I have an IP conflict?
Insert picture description here
After checking the log, I found that there was no error about IP address duplication, but I found that the local match was None when creating the subnet. It shouldn’t be. I have this file /run/flannel/subnet.env locally.

Insert picture description here
It’s strange, there is no FLANNEL_SUBNET field, and the problem is found, continue to check how the subnet.env file is generated

Insert picture description here Generated by a script. Check it and find that it is wrong.

Solution:

Do not add -f or -d and use the default location, check /run/flannel/subnet.env and /run/docker_opts.env

/run/flannel/subnet.env is used for environment variables read by flannel
/run/docker_opts.env is used for docker startup parameters

Insert picture description here Then change the environment variable file of the docker service, restart the docker service and the flannel service

Reference: https://coreos.com/flannel/docs/latest/reservations.html

k8s manually deploy flannel (k8s is based on the cnm network plug-in)

Guess you like