https://www.cnblogs.com/dhcn/p/12572904.html
使用Kubeadm + HAProxy + Keepalived部署高可用Kubernetes集群
这两天kubernetes爆出第一个特权升级高危漏洞，波及非常广泛，且没有有效的补丁可以修改此漏洞，只能将kubernetes升级。说巧不巧，与此同时，kubernetes v1.13.0版本发布，kubeadm也升级为GA版本，简单阅读了一下更新日志，先部署一套练练手～
1、环境说明
本次高可用集群基本参照官网步骤进行部署，官网给出了两种拓扑结构：堆叠control plane node和external etcd node，本文基于第一种拓扑结构进行部署，使用Keepalived + HAProxy搭建高可用Load balancer，完整的拓扑图如下：
单个mastre节点将部署keepalived、haproxy、etcd、apiserver、controller-manager、schedule六种服务，load balancer集群和etcd集群仅用来为kubernetes集群集群服务，不对外营业。如果必要，可以将load balancer或者etcd单独部署，为kubernetes集群提供服务的同时，也可以为其他有需要的系统提供服务，比如下面这样的拓扑结构：
这种拓扑结构也对应external etcd node～
本文仅部署master节点，使用kubeadm部署worker节点非常简单，不在赘述，环境清单：
[root@master-0 kubernetes]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master-0 Ready master 4h2m v1.13.0 172.16.7.11 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0
master-1 Ready master 3h53m v1.13.0 172.16.7.12 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0
master-2 Ready master 3h52m v1.13.0 172.16.7.13 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0
1
2
3
4
5
镜像清单：
[root@master-0 kubernetes]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.13.0 8fa56d18961f 34 hours ago 80.2MB
k8s.gcr.io/kube-scheduler v1.13.0 9508b7d8008d 34 hours ago 79.6MB
k8s.gcr.io/kube-apiserver v1.13.0 f1ff9b7e3d6e 34 hours ago 181MB
k8s.gcr.io/kube-controller-manager v1.13.0 d82530ead066 34 hours ago 146MB
quay.io/calico/node v3.3.1 427a0694c75c 3 weeks ago 75.3MB
quay.io/calico/cni v3.3.1 fa6f35a1c14d 3 weeks ago 75.4MB
k8s.gcr.io/coredns 1.2.6 f59dcacceff4 4 weeks ago 40MB
k8s.gcr.io/etcd 3.2.24 3cab8e1b9802 2 months ago 220MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 11 months ago 742kB
1
2
3
4
5
6
7
8
9
10
11
主要软件清单：
keepalived-1.3.5-6.el7.x86_64.rpm
haproxy-1.5.18-8.el7.x86_64.rpm
docker-ce-18.09.0-3.el7.x86_64.rpm
kubeadm-1.13.0-0.x86_64.rpm
kubectl-1.13.0-0.x86_64.rpm
kubelet-1.13.0-0.x86_64.rpm
网盘下载地址：https://pan.baidu.com/s/1PwS5kiUd9G-oY2i9TgJVyA（包括rpm依赖）
2、部署步骤
2.1、部署keepalived
此处的keeplived的主要作用是为haproxy提供vip（172.16.7.10），在三个haproxy实例之间提供主备，降低当其中一个haproxy失效的时对服务的影响。
系统配置
[root@master-0 ~]# cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_forward = 1
EOF
[root@master-0 ~]# sysctl -p
net.ipv4.ip_forward = 1
1
2
3
4
5
In order for the Keepalived service to forward network packets properly to the real servers, each router node must have IP forwarding turned on in the kernel.
安装keepalived
[root@master-0 ~]# yum install -y keepalived
1
配置keepalived
[root@master-2 ~]# cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script check_haproxy {
script "killall -0 haproxy"
interval 3
weight -2
fall 10
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 250
advert_int 1
authentication {
auth_type PASS
auth_pass 35f18af7190d51c9f7f78f37300a0cbd
}
virtual_ipaddress {
172.16.7.10
}
track_script {
check_haproxy
}
}
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
killall -0 根据进程名称检测进程是否存活
master-0节点为***MASTER***，其余节点为***BACKUP***
priority各个几点到优先级相差50，范围：0～250（非强制要求）
启动并检测服务
[root@master-0 ~]# systemctl enable keepalived.service
[root@master-0 ~]# systemctl start keepalived.service
[root@master-0 ~]# systemctl status keepalived.service
[root@master-0 ~]# ip address show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:50:56:25:99:6e brd ff:ff:ff:ff:ff:ff
inet 172.16.7.11/24 brd 172.16.7.255 scope global noprefixroute ens33
valid_lft forever preferred_lft forever
inet 172.16.7.10/32 scope global ens33
valid_lft forever preferred_lft forever
1
2
3
4
5
6
7
8
9
10
2.2、部署haproxy
此处的haproxy为apiserver提供反向代理，haproxy将所有请求轮询转发到每个master节点上。相对于仅仅使用keepalived主备模式仅单个master节点承载流量，这种方式更加合理、健壮。
系统配置
[root@master-0 ~]# cat >> /etc/sysctl.conf << EOF
net.ipv4.ip_nonlocal_bind = 1
EOF
[root@master-0 ~]# sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
1
2
3
4
5
6
Load balancing in HAProxy and Keepalived at the same time also requires the ability to bind to an IP address that are nonlocal, meaning that it is not assigned to a device on the local system. This allows a running load balancer instance to bind to an IP that is not local for failover.
安装haproxy
[root@master-0 ~]# yum install -y haproxy
1
配置haproxy
[root@master-0 ~]# cat > /etc/haproxy/haproxy.cfg << EOF
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes-apiserver
mode tcp
bind *:16443
option tcplog
default_backend kubernetes-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
mode tcp
balance roundrobin
server master-0 172.16.7.11:6443 check
server master-1 172.16.7.12:6443 check
server master-2 172.16.7.13:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
bind *:1080
stats auth admin:awesomePassword
stats refresh 5s
stats realm HAProxy\ Statistics
stats uri /admin?stats
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
所以master节点上的配置完全相同～
启动并检测服务
[root@master-0 ~]# systemctl enable haproxy.service
[root@master-0 ~]# systemctl start haproxy.service
[root@master-0 ~]# systemctl status haproxy.service
[root@master-0 ~]# ss -lnt | grep -E "16443|1080"
LISTEN 0 128 *:1080 *:*
LISTEN 0 128 *:16443 *:*
1
2
3
4
5
6
2.3、安装kubeadm、kubectl、kubelet、docker
系统配置
[root@master-0 ~]# systemctl disable firewalld.service
[root@master-0 ~]# systemctl stop firewalld.service
[root@master-0 ~]# systemctl status firewalld.service
[root@master-0 ~]# sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
[root@master-0 ~]# setenforce 0
[root@master-0 ~]# sed -i 's/\(.*swap.*\)/# \1/g' /etc/fstab
[root@master-0 ~]# swapoff -a
[root@master-0 ~]# cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
[root@master-0 ~]# sysctl --system
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
安装软件
### 设置docker-ce的yum源 ###
[root@master-0 ~]# cat <<EOF > /etc/yum.repos.d/docker-ce.repo
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/centos/7/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg
EOF
### 安装docker-ce ###
[root@master-0 ~]# yum install -y docker-ce --disableexcludes=docker-ce-stable
### 设置kubernetes的yum源 ###
[root@master-0 ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF
### 安装 ###
[root@master-0 ~]# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
设置开机启动
[root@master-0 ~]# systemctl enable docker.service
[root@master-0 ~]# systemctl start docker.service
[root@master-0 ~]# systemctl status docker.service
[root@master-0 ~]# systemctl enable kubelet.service
1
2
3
4
5
2.4、部署master-0
编辑hosts文件
[root@master-0 ~]# cat >> /etc/hosts << EOF
172.16.7.10 cluster.kube.com
172.16.7.11 master-0
172.16.7.12 master-1
172.16.7.13 master-2
EOF
1
2
3
4
5
6
7
编辑kubeadm配置文件
[root@master-0 ~]# cat > kubeadm-config.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.13.0
apiServer:
certSANs:
- "cluster.kube.com"
controlPlaneEndpoint: "cluster.kube.com:16443"
networking:
podSubnet: "192.168.0.0/16"
EOF
1
2
3
4
5
6
7
8
9
10
11
CNI使用Calico，设置podSubnet: “192.168.0.0/16”
初始化第一个master节点
[root@master-0 ~]# kubeadm init --config kubeadm-config.yaml
[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master-0 localhost] and IPs [172.16.7.11 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master-0 localhost] and IPs [172.16.7.11 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master-0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local cluster.kube.com cluster.kube.com] and IPs [10.96.0.1 172.16.7.11]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.503612 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master-0" as an annotation
[mark-control-plane] Marking the node master-0 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master-0 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 29kttk.v6a5ts4021pkc4zr
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join cluster.kube.com:16443 --token 29kttk.v6a5ts4021pkc4zr --discovery-token-ca-cert-hash sha256:d0bdfd403cd9822177f66c6f3fa28735a67c0747f52511f253ba82843b70f99a
[root@master-0 ~]# mkdir -p $HOME/.kube
[root@master-0 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@master-0 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
安装网络插件
[root@master-0 ~]# kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
[root@master-0 ~]# kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml
configmap/calico-config created
service/calico-typha created
deployment.apps/calico-typha created
poddisruptionbudget.policy/calico-typha created
daemonset.extensions/calico-node created
serviceaccount/calico-node created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
复制相关文件到其他master节点上
[root@master-0 ~]# ssh root@master-1 mkdir -p /etc/kubernetes/pki/etcd
[root@master-0 ~]# scp /etc/kubernetes/admin.conf root@master-1:/etc/kubernetes
[root@master-0 ~]# scp /etc/kubernetes/pki/{ca.*,sa.*,front-proxy-ca.*} root@master-1:/etc/kubernetes/pki
[root@master-0 ~]# scp /etc/kubernetes/pki/etcd/ca.* root@master-1:/etc/kubernetes/pki/etcd
1
2
3
4
2.5、部署master-other
[root@master-1 ~]# kubeadm join cluster.kube.com:16443 --token 29kttk.v6a5ts4021pkc4zr --discovery-token-ca-cert-hash sha256:d0bdfd403cd9822177f66c6f3fa28735a67c0747f52511f253ba82843b70f99a --experimental-control-plane
[preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[discovery] Trying to connect to API Server "cluster.kube.com:16443"
[discovery] Created cluster-info discovery client, requesting info from "https://cluster.kube.com:16443"
[discovery] Requesting info from "https://cluster.kube.com:16443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "cluster.kube.com:16443"
[discovery] Successfully established connection with API Server "cluster.kube.com:16443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[join] Running pre-flight checks before initializing the new control plane instance
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master-1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local cluster.kube.com cluster.kube.com] and IPs [10.96.0.1 172.16.7.12]
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master-1 localhost] and IPs [172.16.7.12 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master-1 localhost] and IPs [172.16.7.12 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Using existing up-to-date kubeconfig file: "/etc/kubernetes/admin.conf"
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Checking Etcd cluster health
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master-1" as an annotation
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node master-1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master-1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
This node has joined the cluster and a new control plane instance was created:
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Master label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
3、检查测试
查看kubernetes集群状态
[root@master-0 etcd]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master-0 Ready master 11m v1.13.0 172.16.7.11 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0
master-1 Ready master 2m16s v1.13.0 172.16.7.12 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0
master-2 Ready master 45s v1.13.0 172.16.7.13 <none> CentOS Linux 7 (Core) 3.10.0-862.el7.x86_64 docker://18.9.0
[root@master-0 etcd]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
[root@master-0 etcd]# kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-6jgcr 11m system:node:master-0 Approved,Issued
node-csr-8wOc8SoFxlkm8Ep5QEr9erUexnYJ-lrilft4QUJ7RfI 2m26s system:bootstrap:29kttk Approved,Issued
node-csr-l-ijTesiUBRFmaAXIHYj3J4vYEA7qbJLolIsKv5FZuM 55s system:bootstrap:29kttk Approved,Issued
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
查看etcd集群状态
[root@master-0 ~]# kubectl exec -ti -n kube-system etcd-master-0 sh
/ # export ETCDCTL_API=3
/ # etcdctl --endpoints=https://[127.0.0.1]:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/ku
bernetes/pki/etcd/healthcheck-client.key member list
25e9e768b2abcbb6, started, master-2, https://172.16.7.13:2380, https://172.16.7.13:2379
28e1924984642da1, started, master-1, https://172.16.7.12:2380, https://172.16.7.12:2379
e19f37b34b9497ee, started, master-0, https://172.16.7.11:2380, https://172.16.7.11:2379
1
2
3
4
5
6
7
————————————————
版权声明：本文为CSDN博主「迷途的攻城狮（798570156）」的原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/chenleiking/article/details/84841394