For more in-depth articles on Kubernetes, please visit my oschina/WaltonWang blog homepage.
Kubernetes Master HA Architecture Diagram
Configuration and Description
-
All components can be started and managed by kubelet static pod. The kubelet static pod mechanism ensures the high availability of each component on the host. Note that kubelet needs to add configuration
--allow-privileged=true; -
The high availability of the kubelet that manages the static pod is responsible for systemd;
-
Of course, you can also deploy these components directly through processes, and systemd manages these processes directly; (We chose this method to reduce complexity.)
-
In the above figure, etcd and Master are deployed together, and three etcds are deployed on three Master nodes respectively, and these three etcds form a cluster; (Of course, if conditions permit, it is recommended to deploy etcd clusters and Master nodes separately.)
-
The apiserver, controller-manager, and scheduler in each Master use hostNetwork, and the controller-manager and scheduler connect to the apiserver of this node through localhost, and will not connect to the apiservers of the other two Master nodes;
-
External rest-client, kubectl, kubelet, kube-proxy, etc. all pass the TLS certificate, and do TLS termination on the LB node. When the LB comes out, the http request is sent to the corresponding apiserver instance through the LB policy (RR);
-
The access from apiserver to kubelet server and kube-proxy server is also similar. Https goes to LB for TLS termination, and then http request goes out to the kubelet/kube-proxy server of the corresponding node;
-
The HA of the apiserver is guaranteed by the classic haproxy + keepalived, and the cluster exposes the VIP to the outside world;
-
The HA of controller-manager and scheduler uses the leader election function (--leader-elect=true) provided by itself, so that only one of the three controller-manager and scheduler is the leader, and the leader is in a normal working state. When the leader fails, it will Re-elect a new leader to replace and continue to work;
-
Therefore, in this HA scheme, haproxy+keepalived is used to do LB and HA of apiserver, controller-manager and scheduler achieve HA through their own leader election, etcd ensures the consistency of etcd cluster data through the raft protocol to achieve HA;
-
The configuration of keepalived can refer to the following:
vrrp_script check_script { script "/etc/keepalived/check_haproxy.py http://caicloud:[email protected]/haproxy?stats" interval 5 # check every 5 seconds weight 5 fall 2 # require 2 fail for KO rise 1 # require 1 successes for OK } vrrp_instance VI_01 { state MASTER (BACKUP) interface eth1 track_interface { eth1 } vrrp_garp_master_repeat 5 vrrp_garp_master_refresh 10 virtual_router_id 51 priority 100 (97) advert_int 1 authentication { auth_type PASS auth_pass username } virtual_ipaddress { 192.168.205.254 dev eth1 label eth1:vip } track_script { check_script } notify "etc/keepalived/notify_state.sh" } -
The configuration of haproxy can be referred to as follows:
global log 127.0.0.1 local0 maxconn 32768 pidfile /run/haproxy.pid # turn on stats unix socket stats socket /run/haproxy.stats tune.ssl.default-dh-param 2048 default log global mode http option httplog option dontlognull retries 3 timeout connect 5000ms timeout client 50000ms timeout server 50000ms timeout check 50000ms timeout queue 50000ms frontend frontend-apisver-http bind *:8080 option forwardfor acl local_net src 192.168.205.0/24 http-request allow if local_net http-request deny default_backend backend-apiserver-http frontedn frontend-apiserver-https # haproxy enable ssl bind *:443 ssl crt /etc/kubernetes/master-lb.pem option forwardfor default_backend backend-apiserver-http backend backend-apiserver-http balance roundrobin option forward-for server master-1 192.168.205.11:8080 check server master-2 192.168.205.12:8080 check server master-3 192.168.205.13:8080 check listen admin_stats bind 0.0.0.0:80 log global mode http maxconn 10 stats enable #Hide HAPRoxy version, a necessity for any public-facing site stats hide-version stats refresh 30s stats show-node stats realm Haproxy\ Statistics stats auth caicloud:caicloud stats uri /haproxy?stats -
The node where the LB is located, pay attention to ensure that the ip_vs model is loaded, and ip_forward and ip_nonlocal_bind are enabled;
# make sure ip_vs kernel model is loaded modprobe ip_vs modprobe ip_vs_rr modprobe ip_vs_wrr # enable ip_forward and ip_nonlocal_bind echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.conf echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf -
If you deploy K8S components through pods, you can refer to the official Yaml:
-
apiserver
apiVersion: v1 kind: Pod metadata: name: kube-apiserver spec: hostNetwork: true containers: - name: kube-apiserver image: gcr.io/google_containers/kube-apiserver:9680e782e08a1a1c94c656190011bd02 command: - /bin/sh - -c - /usr/local/bin/kube-apiserver --address=127.0.0.1 --etcd-servers=http://127.0.0.1:4001 --cloud-provider=gce --admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota --service-cluster-ip-range=10.0.0.0/16 --client-ca-file=/srv/kubernetes/ca.crt --basic-auth-file=/srv/kubernetes/basic_auth.csv --cluster-name=e2e-test-bburns --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key --secure-port=443 --token-auth-file=/srv/kubernetes/known_tokens.csv --v=2 --allow-privileged=False 1>>/var/log/kube-apiserver.log 2>&1 ports: - containerPort: 443 hostPort: 443 name: https - containerPort: 7080 hostPort: 7080 name: http - containerPort: 8080 hostPort: 8080 name: local volumeMounts: - mountPath: /srv/kubernetes name: srvkube readOnly: true - mountPath: /var/log/kube-apiserver.log name: logfile - mountPath: /etc/ssl name: etcssl readOnly: true - mountPath: /usr/share/ssl name: usrsharessl readOnly: true - mountPath: /var/ssl name: varssl readOnly: true - mountPath: /usr/ssl name: usrssl readOnly: true - mountPath: /usr/lib/ssl name: usrlibssl readOnly: true - mountPath: /usr/local/openssl name: usrlocalopenssl readOnly: true - mountPath: /etc/openssl name: etcopenssl readOnly: true - mountPath: /etc/pki/tls name: etcpkitls readOnly: true volumes: - hostPath: path: /srv/kubernetes name: srvkube - hostPath: path: /var/log/kube-apiserver.log name: logfile - hostPath: path: /etc/ssl name: etcssl - hostPath: path: /usr/share/ssl name: usrsharessl - hostPath: path: /var/ssl name: varssl - hostPath: path: /usr/ssl name: usrssl - hostPath: path: /usr/lib/ssl name: usrlibssl - hostPath: path: /usr/local/openssl name: usrlocalopenssl - hostPath: path: /etc/openssl name: etcopenssl - hostPath: path: /etc/pki/tls name: etcpkitls- controller-manager
apiVersion: v1 kind: Pod metadata: name: kube-controller-manager spec: containers: - command: - /bin/sh - -c - /usr/local/bin/kube-controller-manager --master=127.0.0.1:8080 --cluster-name=e2e-test-bburns --cluster-cidr=10.245.0.0/16 --allocate-node-cidrs=true --cloud-provider=gce --service-account-private-key-file=/srv/kubernetes/server.key --v=2 --leader-elect=true 1>>/var/log/kube-controller-manager.log 2>&1 image: gcr.io/google_containers/kube-controller-manager:fda24638d51a48baa13c35337fcd4793 livenessProbe: httpGet: path: /healthz port: 10252 initialDelaySeconds: 15 timeoutSeconds: 1 name: kube-controller-manager volumeMounts: - mountPath: /srv/kubernetes name: srvkube readOnly: true - mountPath: /var/log/kube-controller-manager.log name: logfile - mountPath: /etc/ssl name: etcssl readOnly: true - mountPath: /usr/share/ssl name: usrsharessl readOnly: true - mountPath: /var/ssl name: varssl readOnly: true - mountPath: /usr/ssl name: usrssl readOnly: true - mountPath: /usr/lib/ssl name: usrlibssl readOnly: true - mountPath: /usr/local/openssl name: usrlocalopenssl readOnly: true - mountPath: /etc/openssl name: etcopenssl readOnly: true - mountPath: /etc/pki/tls name: etcpkitls readOnly: true hostNetwork: true volumes: - hostPath: path: /srv/kubernetes name: srvkube - hostPath: path: /var/log/kube-controller-manager.log name: logfile - hostPath: path: /etc/ssl name: etcssl - hostPath: path: /usr/share/ssl name: usrsharessl - hostPath: path: /var/ssl name: varssl - hostPath: path: /usr/ssl name: usrssl - hostPath: path: /usr/lib/ssl name: usrlibssl - hostPath: path: /usr/local/openssl name: usrlocalopenssl - hostPath: path: /etc/openssl name: etcopenssl - hostPath: path: /etc/pki/tls name: etcpkitls- scheduler
apiVersion: v1 kind: Pod metadata: name: kube-scheduler spec: hostNetwork: true containers: - name: kube-scheduler image: gcr.io/google_containers/kube-scheduler:34d0b8f8b31e27937327961528739bc9 command: - /bin/sh - -c - /usr/local/bin/kube-scheduler --master=127.0.0.1:8080 --v=2 --leader-elect=true 1>>/var/log/kube-scheduler.log 2>&1 livenessProbe: httpGet: path: /healthz port: 10251 initialDelaySeconds: 15 timeoutSeconds: 1 volumeMounts: - mountPath: /var/log/kube-scheduler.log name: logfile - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: default-token-s8ejd readOnly: true volumes: - hostPath: path: /var/log/kube-scheduler.log name: logfile- etcd
apiVersion: v1 kind: Pod metadata: name: etcd-server spec: hostNetwork: true containers: - image: gcr.io/google_containers/etcd:2.0.9 name: etcd-container command: - /usr/local/bin/etcd - --name - ${NODE_NAME} - --initial-advertise-peer-urls - http://${NODE_IP}:2380 - --listen-peer-urls - http://${NODE_IP}:2380 - --advertise-client-urls - http://${NODE_IP}:4001 - --listen-client-urls - http://127.0.0.1:4001 - --data-dir - /var/etcd/data - --discovery - ${DISCOVERY_TOKEN} ports: - containerPort: 2380 hostPort: 2380 name: serverport - containerPort: 4001 hostPort: 4001 name: clientport volumeMounts: - mountPath: /var/etcd name: varetcd - mountPath: /etc/ssl name: etcssl readOnly: true - mountPath: /usr/share/ssl name: usrsharessl readOnly: true - mountPath: /var/ssl name: varssl readOnly: true - mountPath: /usr/ssl name: usrssl readOnly: true - mountPath: /usr/lib/ssl name: usrlibssl readOnly: true - mountPath: /usr/local/openssl name: usrlocalopenssl readOnly: true - mountPath: /etc/openssl name: etcopenssl readOnly: true - mountPath: /etc/pki/tls name: etcpkitls readOnly: true volumes: - hostPath: path: /var/etcd/data name: varetcd - hostPath: path: /etc/ssl name: etcssl - hostPath: path: /usr/share/ssl name: usrsharessl - hostPath: path: /var/ssl name: varssl - hostPath: path: /usr/ssl name: usrssl - hostPath: path: /usr/lib/ssl name: usrlibssl - hostPath: path: /usr/local/openssl name: usrlocalopenssl - hostPath: path: /etc/openssl name: etcopenssl - hostPath: path: /etc/pki/tls name: etcpkitls