Flannel是如何工作的
https://my.oschina.net/jxcdwangtao/blog/1624486
Author: [email protected]
概述
最近我们的TaaS平台遇到很多的网络问题，事实证明“contiv + ovs + vlan”的方案并不适合TaaS这种大规模高并发的场景，填不完的坑，当然DevOps场景下是没什么问题的。时间紧迫，只能使用“Flannel + host-gw”这个简单、稳定的网络方案搭建一个小规模的集群来作为紧急备选方案。趁这个机会，也学习一下前两年因性能差，广为诟病而一直不敢碰的Flannel如今是怎么个样子。经过春节半个月的稳定测试、压力测试证明确实很稳定。当然，calico(bgp)才是我们后续的主要网络方案。
Flannel支持多种Backend协议，但是不支持运行时修改Backend。官方推荐使用以下Backend：
- VXLAN，性能损耗大概在20~30%；
- host-gw, 性能损耗大概10%，要求Host之间二层直连，因此只适用于小集群；
- UDP, 建议只用于debug，因为性能烂到家了，如果网卡支持 enable udp offload，直接由网卡进行拆包解包，性能还是很棒的。
- AliVPC。
实验性的Backend，不建议上生产：
- Alloc
- AWS VPC
- GCE
- IPIP
- IPSec
Flannel的配置
Flannel在官方配置可以在https://github.com/coreos/flannel/blob/master/Documentation/configuration.md找到,但是注意文档中的配置不是最新的，是不完整的。
通过命令行配置
目前最新版的Flannel v0.10.0的命令行配置及说明如下：
Usage: /opt/bin/flanneld [OPTION]...
-etcd-cafile string
SSL Certificate Authority file used to secure etcd communication
-etcd-certfile string
SSL certification file used to secure etcd communication
-etcd-endpoints string
a comma-delimited list of etcd endpoints (default "http://127.0.0.1:4001,http://127.0.0.1:2379")
-etcd-keyfile string
SSL key file used to secure etcd communication
-etcd-password string
password for BasicAuth to etcd
-etcd-prefix string
etcd prefix (default "/coreos.com/network")
-etcd-username string
username for BasicAuth to etcd
-healthz-ip string
the IP address for healthz server to listen (default "0.0.0.0")
-healthz-port int
the port for healthz server to listen(0 to disable)
-iface value
interface to use (IP or name) for inter-host communication. Can be specified multiple times to check each option in order. Returns the first match found. -iface-regex value regex expression to match the first interface to use (IP or name) for inter-host communication. Can be specified multiple times to check each regex in order. Returns the first match found. Regexes are checked after specific interfaces specified by the iface option have already been checked. -ip-masq setup IP masquerade rule for traffic destined outside of overlay network -kube-api-url string Kubernetes API server URL. Does not need to be specified if flannel is running in a pod. -kube-subnet-mgr contact the Kubernetes API for subnet assignment instead of etcd. -kubeconfig-file string kubeconfig file location. Does not need to be specified if flannel is running in a pod. -log_backtrace_at value when logging hits line file:N, emit a stack trace -public-ip string IP accessible by other nodes for inter-host communication -subnet-file string filename where env variables (subnet, MTU, ... ) will be written to (default "/run/flannel/subnet.env") -subnet-lease-renew-margin int subnet lease renewal margin, in minutes, ranging from 1 to 1439 (default 60) -v value log level for V logs -version print version and exit -vmodule value comma-separated list of pattern=N settings for file-filtered logging 需要说明如下：
-
我们是通过
-kube-subnet-mgr配置Flannel从Kubernetes APIServer中读取对应的ConfigMap来获取配置的。-kubeconfig-file, -kube-api-url我们也没有配置，因为我们是使用DaemonSet通过Pod来部署的Flannel，所以Flannel与Kubernetes APIServer是通过ServiceAccount来认证通信的。 -
另外一种方式是直接从etcd中读取Flannel配置，需要配置对应的
-etcd开头的Flag。 -
-subnet-file默认为/run/flannel/subnet.env,一般无需改动。Flannel会将本机的subnet信息对应的环境变量注入到该文件中，Flannel真正是从这里获取subnet信息的,比如：FLANNEL_NETWORK=10.244.0.0/16 FLANNEL_SUBNET=10.244.26.1/24 FLANNEL_MTU=1500 FLANNEL_IPMASQ=true -
-subnet-lease-renew-margin表示etcd租约到期前多少时间就可以重新自动续约，默认是1h。因为ttl时间是24h，所以这项配置自然不允许超过24h，即[1, 1439] min.
通过环境变量配置
上面的命令行配置项，都可以通过改成大写，下划线变中划线，再加上FLANNELD_前缀转成对应的环境变量的形式来设置。
比如--etcd-endpoints=http://10.0.0.2:2379对应的环境变量为FLANNELD_ETCD_ENDPOINTS=http://10.0.0.2:2379。
部署Flannel
通过Kubernetes DaemonSet部署Flannel，这一点毫无争议。同时创建对应的ClusterRole，ClusterRoleBinding，ServiceAccount，ConfigMap。完整的Yaml描述文件可参考如下：
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: flannel rules: - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "" resources: - nodes verbs: - list - watch - apiGroups: - "" resources: - nodes/status verbs: - patch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node k8s-app: flannel data: cni-conf.json: | { "name": "cbr0", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } } ] } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "host-gw" } } --- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: kube-flannel namespace: kube-system labels: tier: node k8s-app: flannel spec: template: metadata: labels: tier: node k8s-app: flannel spec: imagePullSecrets: - name: harborsecret serviceAccountName: flannel containers: - name: kube-flannel image: registry.vivo.xyz:4443/coreos/flannel:v0.10.0-amd64 command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr"] securityContext: privileged: true env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP volumeMounts: - name: run mountPath: /run - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ - name: install-cni image: registry.vivo.xyz:4443/coreos/flannel-cni:v0.3.0 command: ["/install-cni.sh"] #command: ["sleep","10000"] env: # The CNI network config to install on each node. - name: CNI_NETWORK_CONFIG valueFrom: configMapKeyRef: name: kube-flannel-cfg key: cni-conf.json volumeMounts: #- name: cni # mountPath: /etc/cni/net.d - name: cni mountPath: /host/etc/cni/net.d - name: host-cni-bin mountPath: /host/opt/cni/bin/ hostNetwork: true tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule volumes: - name: run hostPath: path: /run #- name: cni # hostPath: # path: /etc/kubernetes/cni/net.d - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg - name: