kubernetes搭建三、flannel网络

一、Flannel网络介绍

Overlay Network模式

覆盖网络，在基础网络上叠加的一种虚拟网络技术模式，该网络中的主机通过虚拟链路连接起来。

VXLAN

将源数据包封装到UDP中，并使用基础网络的IP/MAC作为外层报文头进行封装，然后在以太网上传输，到达目的地后由隧道端点解封装并将数据发送给目标地址。
在这里插入图片描述

Flannel

是Overlay网络的一种，也是将源数据包封装在另一种网络包里面进行路由转发和通信，目前已经支持UDP、VXLAN、AWS VPC和GCE路由等数据转发方式。

Flannel通过给每台宿主机分配一个子网的方式为容器提供虚拟网络，它基于Linux TUN/TAP，使用UDP封装IP包来创建overlay网络，并借助etcd维护网络的分配情况。

Flannel是CoreOS团队针对Kubernates设计的跨主机容器网络解决方案, 它可以使集群中不同节点上运行的docker容器都具有全集群唯一的虚拟IP地址。其模型为全部的宿主机使用一个network，然后在每个宿主机上从network中划分一个子网subnet。为宿主机上的容器创建网络时，从subnet中划分一个ip给容器。

数据从源容器中发出后，经由所在主机的docker0虚拟网卡转发到flannel0虚拟网卡，这是个P2P的虚拟网卡，flanneld服务监听在网卡的另外一端。
Flannel通过Etcd服务维护了一张节点间的路由表，详细记录了各节点子网网段。
源主机的flanneld服务将原本的数据内容UDP封装后根据自己的路由表投递给目的节点的flanneld服务，数据到达以后被解包，然后直接进入目的节点的flannel0虚拟网卡，然后被转发到目的主机的docker0虚拟网卡，最后就像本机容器通信一样的有docker0路由到达目标容器。

在这里插入图片描述

二、Flannel网络搭建

1、Falnnel要用etcd存储自身一个子网信息，所以要保证能成功连接Etcd。写入分配的子网段到etcd，供flanneld使用，把flannel网络信息添加到etcd，在哪个节点都行，因为是写入etcd集群
使用set设置子网段为172.17.0.0/16

/opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem  \
--key-file=/opt/kubernetes/ssl/server-key.pem \
--endpoints="https://10.0.0.201:2379,https://10.0.0.202:2379,https://10.0.0.203:2379" \ 
set /coreos.com/network/config '{"Network":"172.17.0.0/16","Backend":{"Type":"vxlan"}}'

2、使用get查看子网段

[root@k8s-master-101 UI]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem  \
--cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem \
--endpoints="https://10.0.0.101:2379,https://10.0.0.102:2379,https://10.0.0.103:2379" \
get /coreos.com/network/config
{"Network":"172.17.0.0/16","Backend":{"Type":"vxlan"}}

3、在master节点上下载flannel安装包，然后发送到node节点上

wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
scp flanneld mk-docker-opts.sh [email protected]:/opt/kubernetes/bin/
scp flanneld mk-docker-opts.sh [email protected]:/opt/kubernetes/bin/

下面的操作在node上进行，两台都一样
4、在node上设置flannel变量配置文件
vim /opt/kubernetes/cfg/flanneld

[root@k8s-node1-102 cfg]# cat /opt/kubernetes/cfg/flanneld

FLANNEL_OPTIONS="--etcd-endpoints=https://10.0.0.101:2379,https://10.0.0.102:2379,https://10.0.0.103:2379 \
-etcd-cafile=/opt/kubernetes/ssl/ca.pem -etcd-certfile=/opt/kubernetes/ssl/server.pem \
-etcd-keyfile=/opt/kubernetes/ssl/server-key.pem"

5、设置flannel的服务配置文件
vim /usr/lib/systemd/system/flanneld.service

[root@k8s-node1-102 cfg]# cat /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

[Install]
WantedBy=multi-user.target

6、启动服务
systemctl start flanneld
7、ifconfig可以看到有docker0的地址已经变成172.17.50.1，有一个flannel1的网络

[root@k8s-node1-102 ~]# ifconfig
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.50.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::4cfa:eeff:fe03:83bf  prefixlen 64  scopeid 0x20<link>
        ether 4e:fa:ee:03:83:bf  txqueuelen 0  (Ethernet)
        RX packets 3  bytes 269 (269.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 253 (253.0 B)
        TX errors 0  dropped 5439 overruns 0  carrier 0  collisions 0

8、查看生成子网的变量文件

[root@k8s-node1-102 ~]# cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=172.17.50.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.50.1/24 --ip-masq=false --mtu=1450"

9、修改docker的服务配置文件，让docker应用flanneld网络

mv /usr/lib/systemd/system/docker.service /usr/lib/systemd/system/docker.service.bak
vim /usr/lib/systemd/system/docker.service

[root@k8s-node1-102 ~]# cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd  $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

10、重启服务
systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld
systemctl restart docker

然后在另一台node上相同操作安装flannel网络

11、而在没有装flannel的master节点上，可以看到docker0的地址默认为172.17.0.1，如果没有设置flannel网络的话，每个node节点上的docker地址都是独立的而且容器之间无法通信，分配的地址还可能有冲突，所以配置flannel网络让docker分配的容器具有全局唯一的地址并且让不同宿主机上的容器能相互通信。

[root@k8s-master-101 ~]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:f2:ad:3f:25  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

三、查看相关信息

1、node1 docker和flannel地址

[root@k8s-node1-102 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.50.1  netmask 255.255.255.0  broadcast 172.17.50.255
        inet6 fe80::42:d8ff:fedc:f1f6  prefixlen 64  scopeid 0x20<link>
        ether 02:42:d8:dc:f1:f6  txqueuelen 0  (Ethernet)
        RX packets 12  bytes 745 (745.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5686  bytes 666915 (651.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.50.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::4cfa:eeff:fe03:83bf  prefixlen 64  scopeid 0x20<link>
        ether 4e:fa:ee:03:83:bf  txqueuelen 0  (Ethernet)
        RX packets 3  bytes 269 (269.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 253 (253.0 B)
        TX errors 0  dropped 5730 overruns 0  carrier 0  collisions 0

2、node2 docker和flannel地址

[root@k8s-node2-103 ~]# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.71.1  netmask 255.255.255.0  broadcast 172.17.71.255
        inet6 fe80::42:33ff:fe8e:e80f  prefixlen 64  scopeid 0x20<link>
        ether 02:42:33:8e:e8:0f  txqueuelen 0  (Ethernet)
        RX packets 100046  bytes 14918590 (14.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 113117  bytes 21596123 (20.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.71.0  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::50bf:ddff:fe8d:d4f3  prefixlen 64  scopeid 0x20<link>
        ether 52:bf:dd:8d:d4:f3  txqueuelen 0  (Ethernet)
        RX packets 3  bytes 253 (253.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 269 (269.0 B)
        TX errors 0  dropped 9061 overruns 0  carrier 0  collisions 0

3、互相ping 一下能通

[root@k8s-node1-102 ~]# ping 172.17.71.1
PING 172.17.71.1 (172.17.71.1) 56(84) bytes of data.
64 bytes from 172.17.71.1: icmp_seq=1 ttl=64 time=0.503 ms
64 bytes from 172.17.71.1: icmp_seq=2 ttl=64 time=0.456 ms
64 bytes from 172.17.71.1: icmp_seq=3 ttl=64 time=0.305 ms
64 bytes from 172.17.71.1: icmp_seq=4 ttl=64 time=0.471 ms

4、可以用ls查看/coreos.com/network下生成的配置文件

[root@k8s-master-101 UI]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem \
--cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem \
--endpoints="https://10.0.0.101:2379,https://10.0.0.102:2379,https://10.0.0.103:2379" \
ls /coreos.com/network/

/coreos.com/network/config
/coreos.com/network/subnets

5、ls查看子网

[root@k8s-master-101 UI]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem \
--cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem \
--endpoints="https://10.0.0.101:2379,https://10.0.0.102:2379,https://10.0.0.103:2379"  \
ls /coreos.com/network/subnets

/coreos.com/network/subnets/172.17.50.0-24
/coreos.com/network/subnets/172.17.71.0-24

6、get获取这个网段的容器在哪个节点上

[root@k8s-master-101 UI]# /opt/kubernetes/bin/etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem \
--cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem \
--endpoints="https://10.0.0.101:2379,https://10.0.0.102:2379,https://10.0.0.103:2379" \
get /coreos.com/network/subnets/172.17.50.0-24

{"PublicIP":"10.0.0.102","BackendType":"vxlan","BackendData":{"VtepMAC":"4e:fa:ee:03:83:bf"}}