实验环境
主机名 IP地址 安装软件
Master01：14.0.0.50 kube-apiserver kube-controller-manager kube-scheduler etcd
Node01： 14.0.0.60 kubelet kube-proxy docker flannel etcd
Node02： 14.0.0.70 kubelet kube-proxy docker flannel etcd
实验过程
1.Etcd集群部署
#初始化环境，准备制作证书
[root@localhost ~]# mkdir k8s
[root@localhost ~]# cd k8s/
[root@localhost k8s]# ls #从宿主机上传进来
etcd-cert.sh etcd.sh
[root@localhost k8s]# mkdir etcd-cert
[root@localhost k8s]# mv etcd-cert.sh etcd-cert/
1.下载证书制作工具
[root@localhost k8s]# vim cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
2.执行脚本下载 cfssl官方包
[root@localhost k8s]# bash cfssl.sh
[root@localhost k8s]# ls /usr/local/bin/
cfssl cfssl-certinfo cfssljson
#cfssl：生成证书工具
#cfssljson：通过传入json文件生成证书
#cfssl-certinfo：查看证书信息
3.开始制作证书
（1）创建ca配置文件
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
（2）创建ca证书签名请求
cat > ca-csr.json <<EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
（3）生成证书
ca-key.pem：根证书的私钥
ca.pem：ca根证书文件
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
（4）指定 etcd三个节点之间的通信验证
cat > server-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"14.0.0.50",
"14.0.0.60",
"14.0.0.70"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
（5）生成 ETCD证书 server-key.pem server.pem
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
#证书生成完毕，开始配置etcd软件
#ETCD 二进制包地址
https://github.com/etcd-io/etcd/releases
将etcd-v3.3.10-linux-amd64.tar.gz上传到/root/k8s目录下
[root@localhost k8s]# ls
cfssl.sh etcd.sh
etcd-cert etcd-v3.3.10-linux-amd64.tar.gz
（1）将etcd软件包进行解压
[root@localhost k8s]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz
[root@localhost k8s]# ls etcd-v3.3.10-linux-amd64
Documentation etcd etcdctl README-etcdctl.md README.md READMEv2-etcdctl.md
（2）创建etcd专门的配置文件，命令文件和证书存放目录
[root@localhost k8s]# mkdir -p /opt/etcd/{
cfg,bin,ssl}
（3）将etcd软件包中的命令文件移动到/opt/etcd/bin目录下
[root@localhost k8s]# mv etcd-v3.3.10-linux-amd64/etcd etcd-v3.3.10-linux-amd64/etcdctl /opt/etcd/bin/
（4）拷贝证书到/opt/etcd/ssl目录下
[root@localhost k8s]# cp etcd-cert/*.pem /opt/etcd/ssl/
#启动etcd.sh脚本生成配置文件和服务启动脚本；并启动etcd服务
[root@localhost k8s]# bash etcd.sh etcd01 14.0.0.50 etcd02=https://14.0.0.60:2380,etcd03=https://14.0.0.70:2380 #这时会等待其他节点加入
#使用另外一个会话打开，发现 etcd进程已经开启
[root@localhost ~]# ps -ef | grep etcd
#使用scp拷贝配置文件，命令文件，证书文件到两个node节点
[root@localhost k8s]# scp -r /opt/etcd/ root@14.0.0.60:/opt/
[root@localhost k8s]# scp -r /opt/etcd/ root@14.0.0.70:/opt
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@14.0.0.60:/usr/lib/systemd/system/
[root@localhost k8s]# scp /usr/lib/systemd/system/etcd.service root@14.0.0.70:/usr/lib/systemd/system/
#进入node01节点修改（node02节点同理）
[root@localhost ~]# vim /opt/etcd/cfg/etcd
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://14.0.0.60:2380" #群集内部通信端口
ETCD_LISTEN_CLIENT_URLS="https://14.0.0.60:2379" #对外提供服务端口
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://14.0.0.60:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://14.0.0.60:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://14.0.0.50:2380,etcd02=https://14.0.0.60:2380,etcd03=https://14.0.0.70:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#启动etcd
[root@localhost ssl]# systemctl start etcd
[root@localhost ssl]# systemctl status etcd
#使用etcdctl命令检查群集健康状态
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379" cluster-health
member 3eae9a550e2e3ec is healthy: got healthy result from https://192.168.195.151:2379
member 26cd4dcf17bc5cbd is healthy: got healthy result from https://192.168.195.150:2379
member 2fcd2df8a9411750 is healthy: got healthy result from https://192.168.195.149:2379
cluster is healthy #说明群集健康
2.docker引擎部署
所有node节点部署docker-ce引擎
部署docker-ce社区版可参考我之前的博客：
https://blog.csdn.net/chengu04/article/details/108723407
3.flannel网络部署
所有node节点部署flannel组件
（1）写入分配的子网段到etcd中，供flannel使用（注意：要在/opt/etcd/ssl目录下）
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
{
"Network": "172.17.0.0/16", "Backend": {
"Type": "vxlan"}}
（2）在另一个节点查看写入信息
[root@localhost etcd-cert]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379" get /coreos.com/network/config
{
"Network": "172.17.0.0/16", "Backend": {
"Type": "vxlan"}}
（3）在两个node节点上部署flannel软件
[root@localhost ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
flanneld
mk-docker-opts.sh
README.md
#创建k8s工作目录，将软件包中的命令文件移动到/opt/kubernetes/bin/目录下
[root@localhost ~]# mkdir -p /opt/kubernetes/{
cfg,bin,ssl}
[root@localhost ~]# mv mk-docker-opts.sh flanneld /opt/kubernetes/bin/
#通过脚本配置flannel
[root@localhost ~]# vim flannel.sh
#!/bin/bash
ETCD_ENDPOINTS=${
1:-"http://127.0.0.1:2379"}
cat <<EOF >/opt/kubernetes/cfg/flanneld #生成配置文件
FLANNEL_OPTIONS="-- etcd-endpoints=${ETCD_ENDPOINTS} \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF
cat <<EOF >/usr/lib/systemd/system/flanneld.service #生成服务启动脚本
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld -- ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable flanneld
systemctl restart flanneld
#开始flannel网络功能
[root@localhost ~]# bash flannel.sh https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
#修改docker的服务启动文件指定子网段
[root@localhost ~]# vim /usr/lib/systemd/system/docker.service
[Service]
Type=notify
#the default is not to use systemd for cgroups because the delegate issues still
#exists and systemd currently does not support the cgroup feature set required
#for containers run by docker
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS -H fd:// -- containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
[root@localhost ~]# cat /run/flannel/subnet.env #查看flannel为该节点分配的子网段
DOCKER_OPT_BIP="--bip=172.17.42.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
##说明：bip指定启动时的子网
DOCKER_NETWORK_OPTIONS=" --bip=172.17.42.1/24 -- ip-masq=false -- mtu=1450"
#重启docker服务
[root@localhost ~]# systemctl daemon-reload
[root@localhost ~]# systemctl restart docker
#查看flannel网络
[root@localhost ~]# ifconfig
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.42.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::fc7c:e1ff:fe1d:224 prefixlen 64 scopeid 0x20<link>
ether fe:7c:e1:1d:02:24 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 26 overruns 0 carrier 0 collisions 0
#在两个node节点中都运行一个容器，测试能否ping通对方，能ping通则flannel部署成功
[root@localhost ~]# docker run -it centos:7 /bin/bash
[root@5f9a65565b53 /]# yum install net-tools -y
[root@5f9a65565b53 /]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.42.2 netmask 255.255.255.0 broadcast 172.17.84.255
ether 02:42:ac:11:54:02 txqueuelen 0 (Ethernet)
....省略内容
4.部署master节点
#在 master上操作，api-server生成证书
上传master.zip到/root/k8s目录下
[root@localhost k8s]# unzip master.zip
[root@localhost k8s]# mkdir /opt/kubernetes/{
cfg,bin,ssl} -p
[root@localhost k8s]# mkdir k8s-cert
[root@localhost k8s]# cd k8s-cert/
[root@localhost k8s-cert]# ls #上传到/root/k8s/k8s-cert/目录下
k8s-cert.sh #利用脚本生成k8s证书
#脚本内容如下
cat > ca-config.json <<EOF #ca证书配置文件
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
cat > ca-csr.json <<EOF #ca证书签名文件
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
#生成ca.pem和ca-key.pem（CA认证机构）
#-----------------------
cat > server-csr.json <<EOF
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"14.0.0.50", #master01
"14.0.0.80", #master02
"14.0.0.88", #VIP
"14.0.0.90", #反向代理（master）
"14.0.0.100", #反向代理（backup）
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
#生成kube-apiserver的tls认证证书和认证私钥server.pem和server-key.pem
#-----------------------
cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
#kubectl的TLS认证证书和认证私钥，具有admin权限，admin.pem和admin-key.pem
#-----------------------
cat > kube-proxy-csr.json <<EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
#生成kube-proxy-key.pem和kube-proxy.pem
#生成证书如下
[root@localhost k8s-cert]# ls *pem
admin-key.pem ca-key.pem kube-proxy-key.pem server-key.pem
admin.pem ca.pem kube-proxy.pem server.pem
#将ca证书和kube-apiserver证书复制到/opt/kubernetes/ssl/目录下
[root@localhost k8s-cert]# cp ca*pem server*pem /opt/kubernetes/ssl/
[root@localhost k8s-cert]# cd ..
#解压 kubernetes压缩包
[root@localhost k8s]# tar zxvf kubernetes-server-linux-amd64.tar.gz
[root@localhost k8s]# cd /root/k8s/kubernetes/server/bin
#复制软件包中bin目录下的关键命令文件
[root@localhost bin]# cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/
[root@localhost k8s]# cd /root/k8s
#使用 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 可以随机生成序列号
[root@localhost k8s]# vim /opt/kubernetes/cfg/token.csv
0fb61c46f8991b718eb38d27b605b008,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
填写内容：序列号， 用户名，uid，用户组
（该文件为一个用户的描述文件，基本格式为 Token,用户名,UID,用户组；这个文件在 apiserver 启动时被 apiserver 加载，然后就相当于在集群内创建了一个这个用户；接下来就可以用 RBAC 给他授权。）
#二进制文件，token，证书都准备好，开启 apiserver
[root@localhost k8s]# bash apiserver.sh 14.0.0.50 https://14.0.0.50:2379,https://14.0.0.60:2379,https://14.0.0.70:2379
#检查进程是否启动成功
[root@localhost k8s]# ps aux | grep kube
#监听的https端口
[root@localhost k8s]# netstat -ntap | grep 6443
tcp 0 0 192.168.195.149:6443 0.0.0.0:* LISTEN 46459/kube-apiserve
tcp 0 0 192.168.195.149:6443 192.168.195.149:36806 ESTABLISHED 46459/kube-apiserve
tcp 0 0 192.168.195.149:36806 192.168.195.149:6443 ESTABLISHED 46459/kube-apiserve
#监听的http端口
[root@localhost k8s]# netstat -ntap | grep 8080
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN
46459/kube-apiserve
#启动scheduler服务
[root@localhost k8s]# ./scheduler.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
[root@localhost k8s]# ps aux | grep ku
[root@localhost k8s]# chmod +x controller-manager.sh
#启动controller-manager服务
[root@localhost k8s]# ./controller-manager.sh 127.0.0.1
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
#查看master节点状态
[root@localhost k8s]# /opt/kubernetes/bin/kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy {
"health":"true"}
etcd-1 Healthy {
"health":"true"}
etcd-0 Healthy {
"health":"true"}
5.部署node节点
#把master01节点上的kubelet、kube-proxy拷贝到node节点上去
[root@localhost bin]# scp kubelet kube-proxy root@14.0.0.60:/opt/kubernetes/bin/
[root@localhost bin]# scp kubelet kube-proxy root@14.0.0.70:/opt/kubernetes/bin/
#nod01节点操作（上传node.zip到/root目录下）
[root@localhost ~]# ls
anaconda-ks.cfg flannel-v0.10.0-linux-amd64.tar.gz node.zip 公共 视频 文档 音乐
flannel.sh initial-setup-ks.cfg README.md 模板 图片 下载 桌 面
//解压 node.zip，获得 kubelet.sh；proxy.sh脚本
[root@localhost ~]# unzip node.zip
#在master上操作
[root@localhost k8s]# mkdir kubeconfig
[root@localhost k8s]# cd kubeconfig/
//将kubeconfig.sh文件进行重命名
[root@localhost kubeconfig]# mv kubeconfig.sh kubeconfig
//获取tokenID信息
[root@localhost ~]# cat /opt/kubernetes/cfg/token.csv
6351d652249951f79c33acdab329e4c4,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
[root@localhost kubeconfig]# vim kubeconfig
#修改tokenID
kubectl config set-credentials kubelet-bootstrap \
-- token=6351d652249951f79c33acdab329e4c4 \
-- kubeconfig=bootstrap.kubeconfig
//设置环境变量 （可以写入到 /etc/profile中）
[root@localhost kubeconfig]# echo export PATH=$PATH:/opt/kubernetes/bin/ >> /etc/profile
[root@localhost kubeconfig]# source /etc/profile
[root@localhost kubeconfig]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-1 Healthy {
"health":"true"}
etcd-0 Healthy {
"health":"true"}
etcd-2 Healthy {
"health":"true"}
//生成配置文件
[root@localhost kubeconfig]# bash kubeconfig 14.0.0.60 /root/k8s/k8s-cert/
Cluster "kubernetes" set.
User "kubelet-bootstrap" set.
Context "default" created.
Switched to context "default".
Cluster "kubernetes" set.
User "kube-proxy" set.
Context "default" created.
[root@localhost kubeconfig]# ls
bootstrap.kubeconfig kubeconfig kube-proxy.kubeconfig
//拷贝配置文件到 拷贝配置文件到 node节点
[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@14.0.0.60:/opt/kubernetes/cfg/
[root@localhost kubeconfig]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@14.0.0.70:/opt/kubernetes/cfg/
//首次启动时kubelet如何连接apiserver？
使用预设用户kubelet-bootstrap，但是需要群集角色绑定，将预设用户 kubelet-bootstrap与内置的 ClusterRole system:node-bootstrapper 绑定到一起，才能将权限用于连接 apiserver请求
[root@localhost kubeconfig]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created
#在node01节点上操作
[root@localhost ~]# bash kubelet.sh 14.0.0.60
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
//检查kubelet服务启动
[root@localhost ~]# ps aux | grep kube
root 106845 1.4 1.1 371744 44780 ? Ssl 00:34 0:01 /opt/kubernetes/bin/kubelet -- logtostderr=true -- v=4 -- hostname-override=192.168.195.150 -- kubeconfig=/opt/kubernetes/cfg/kubelet.kubconfig -- bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig -- config=/opt/kubernetes/cfgkubelet.config -- cert-dir=/opt/kubernetes/ssl -- pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
root 106876 0.0 0.0 112676 984 pts/0 S+ 00:35 0:00 grep -- color=auto kube
#master01上操作（手动签发）
//检查node01节点的请求
[root@localhost kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A 4m27s kubelet-bootstrap Pending（等待集群给该节点颁发证书）
[root@localhost kubeconfig]# kubectl certificate approve
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A
certificatesigningrequest.certificates.k8s.io/node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A approved
[root@localhost kubeconfig]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-NOI-9vufTLIqJgMWq4fHPNPHKbjCXlDGHptj7FqTa8A 8m56s kubelet-bootstrap Approved,Issued（Approved代表已经被允许加入群集）
//查看群集节点，成功加入 node01节点
[root@localhost kubeconfig]# kubectl get node
NAME STATUS ROLES AGE VERSION
14.0.0.50 Ready <none> 118s v1.12.3
#在 node01节点操作，启动 proxy服务
[root@localhost ~]# bash proxy.sh 14.0.0.60
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
[root@localhost ~]# systemctl status kube-proxy.service
● kube-proxy.service - Kubernetes Proxy
Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2020-02-02 00:47:29 CST; 11s ago
Main PID: 108006 (kube-proxy)
Memory: 7.5M
CGroup: /system.slice/kube-proxy.service
‣ 108006 /opt/kubernetes/bin/kube-proxy -- logtostderr=true -- v=4 -- hostname-override=1...
node02节点部署
#在node01节点操作
//把现成的 /opt/kubernetes目录复制到其他节点进行修改即可
[root@localhost ~]# scp -r /opt/kubernetes/ root@14.0.0.70:/opt/
//把 kubelet，kube-proxy的 service文件拷贝到 node2中
[root@localhost ~]# scp /usr/lib/systemd/system/{
kubelet,kube-proxy}.service root@14.0.0.70:/usr/lib/systemd/system/
#在node02节点上操作，进行修改
//首先删除复制过来的证书，等会 node02会自行申请证书
[root@localhost ~]# cd /opt/kubernetes/ssl/
[root@localhost ssl]# rm -rf *
//修改配置文件kubelet kubelet.config kube-proxy（三个配置文件）
[root@localhost ssl]# cd ../cfg/
[root@localhost cfg]# vim kubelet
KUBELET_OPTS="-- logtostderr=true \
-- v=4 \
-- hostname-override=14.0.0.70 \
-- kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
-- bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
-- config=/opt/kubernetes/cfg/kubelet.config \
-- cert-dir=/opt/kubernetes/ssl \
-- pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
[root@localhost cfg]# vim kubelet.config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 14.0.0.70
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true
~
[root@localhost cfg]# vim kube-proxy
KUBE_PROXY_OPTS="-- logtostderr=true \
-- v=4 \
-- hostname-override=14.0.0.70 \
-- cluster-cidr=10.0.0.0/24 \
-- proxy-mode=ipvs \
-- kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
//启动服务
[root@localhost cfg]# systemctl start kubelet.service
[root@localhost cfg]# systemctl enable kubelet.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@localhost cfg]# systemctl start kube-proxy.service
[root@localhost cfg]# systemctl enable kube-proxy.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
//在master上操作查看请求
[root@localhost k8s]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU 15s kubelet-bootstrap Pending
//授权许可加入群集
[root@localhost k8s]# kubectl certificate approve node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU
certificatesigningrequest.certificates.k8s.io/node-csr-OaH9HpIKh6AKlfdjEKm4C6aJ0UT_1YxNaa70yEAxnsU approved
//查看群集中的节点
[root@localhost k8s]# kubectl get node #两个node节点均显示Ready代表都已成功加入
NAME STATUS ROLES AGE VERSION
14.0.0.60 Ready <none> 21h v1.12.3
14.0.0.70 Ready <none> 37s v1.12.3
实验故障及如何处理
如果实验最后查看群集中的节点时，发现有node处于noready状态，如何处理？
1.先检查网络，查看各个节点间网络是否互通；
2.网络没问题再检查kubelet，检查kubeconfig中的三个配置（apiserver、tokenID、证书）是否正确；
3.使用公网上的主机访问node中的业务，如果内外网都无法访问服务器说明该node服务器可能已经宕机了，需要联系现场运维工程师查看。