1. 基本说明
本文章将演示CentOS 8二进制方式安装高可用k8s 1.16.x，相对于其他版本，二进制安装方式并无太大区别。CentOS 8相对于CentOS 7操作更加方便，比如一些服务的关闭，无需修改配置文件即可永久生效，CentOS 8默认安装的内核版本是4.18，所以在安装k8s的过程中也无需在进行内核升级，系统环境也可按需升级，如果下载的是最新版的CentOS 8，系统升级也可省略。
2. 基本环境配置
主机信息
192.168.1.19 k8s-master01 192.168.1.18 k8s-master02 192.168.1.20 k8s-master03 192.168.1.88 k8s-master-lb 192.168.1.21 k8s-node01 192.168.1.22 k8s-node02
系统环境
[root@k8s-master01 ~]# cat /etc/redhat-release CentOS Linux release 8.0.1905 (Core) [root@k8s-master01 ~]# uname -a Linux k8s-master01 4.18.0-80.el8.x86_64 #1 SMP Tue Jun 4 09:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
配置所有节点hosts文件
[root@k8s-master01 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.19 k8s-master01 192.168.1.18 k8s-master02 192.168.1.20 k8s-master03 192.168.1.88 k8s-master-lb 192.168.1.21 k8s-node01 192.168.1.22 k8s-node02
所有节点关闭firewalld 、dnsmasq、NetworkManager、selinux
systemctl disable --now firewalld systemctl disable --now dnsmasq
setenforce 0
所有节点关闭swap分区
[root@k8s-master01 ~]# swapoff -a && sysctl -w vm.swappiness=0 vm.swappiness = 0
所有节点同步时间
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime echo 'Asia/Shanghai' >/etc/timezone ntpdate time2.aliyun.com
Master01节点生成ssh key
[root@k8s-master01 ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:6uz2kI+jcMJIUQWKqRcDRbvpVxhCW3Tmqn0NKS+lT3U root@k8s-master01 The key's randomart image is: +---[RSA 2048]----+ |.o++=.o | |.+o+ + | |oo* . . | |. .* + . | |..+ + = S E | |.= o * * . | |. * * B . | | = Bo+ | | .+*oo | +----[SHA256]-----+
Master01配置免密码登录其他节点
[root@k8s-master01 ~]# for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
所有节点安装基本工具
yum install wget jq psmisc vim net-tools yum-utils device-mapper-persistent-data lvm2 git -y
Master01下载安装文件
[root@k8s-master01 ~]# git clone https://github.com/dotbalo/k8s-ha-install.git Cloning into 'k8s-ha-install'... remote: Enumerating objects: 12, done. remote: Counting objects: 100% (12/12), done. remote: Compressing objects: 100% (11/11), done. remote: Total 461 (delta 2), reused 5 (delta 1), pack-reused 449 Receiving objects: 100% (461/461), 19.52 MiB | 4.04 MiB/s, done. Resolving deltas: 100% (163/163), done.
切换到1.16.x分支
git checkout manual-installation-v1.16.x
3. 基本组件安装
配置Docker yum源
[root@k8s-master01 k8s-ha-install]# curl https://download.docker.com/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker-ce.repo % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 2424 100 2424 0 0 3639 0 --:--:-- --:--:-- --:--:-- 3645 [root@k8s-master01 k8s-ha-install]# yum makecache CentOS-8 - AppStream 559 kB/s | 6.3 MB 00:11 CentOS-8 - Base 454 kB/s | 7.9 MB 00:17 CentOS-8 - Extras 592 B/s | 2.1 kB 00:03 Docker CE Stable - x86_64 5.8 kB/s | 20 kB 00:03 Last metadata expiration check: 0:00:01 ago on Sat 02 Nov 2019 02:46:29 PM CST. Metadata cache created.
所有节点安装新版containerd
[root@k8s-master01 k8s-ha-install]# wget https://download.docker.com/linux/centos/7/x86_64/edge/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm --2019-11-02 15:00:20-- https://download.docker.com/linux/centos/7/x86_64/edge/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm Resolving download.docker.com (download.docker.com)... 13.225.103.32, 13.225.103.65, 13.225.103.10, ... Connecting to download.docker.com (download.docker.com)|13.225.103.32|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 27119348 (26M) [binary/octet-stream] Saving to: ‘containerd.io-1.2.6-3.3.el7.x86_64.rpm’ containerd.io-1.2.6-3.3.el7.x86_64.rpm 100%[===================================================================================================================>] 25.86M 1.55MB/s in 30s 2019-11-02 15:00:51 (887 KB/s) - ‘containerd.io-1.2.6-3.3.el7.x86_64.rpm’ saved [27119348/27119348] [root@k8s-master01 k8s-ha-install]# yum -y install containerd.io-1.2.6-3.3.el7.x86_64.rpm Last metadata expiration check: 0:14:35 ago on Sat 02 Nov 2019 02:46:29 PM CST.
所有节点安装最新版Docker
[root@k8s-master01 k8s-ha-install]# yum install docker-ce -y
所有节点开启Docker并设置开机自启动
[root@k8s-master01 k8s-ha-install]# systemctl enable --now docker Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /usr/lib/systemd/system/docker.service. [root@k8s-master01 k8s-ha-install]# docker version Client: Docker Engine - Community Version: 19.03.4 API version: 1.40 Go version: go1.12.10 Git commit: 9013bf583a Built: Fri Oct 18 15:52:22 2019 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 19.03.4 API version: 1.40 (minimum version 1.12) Go version: go1.12.10 Git commit: 9013bf583a Built: Fri Oct 18 15:50:54 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.2.6 GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb runc: Version: 1.0.0-rc8 GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f docker-init: Version: 0.18.0 GitCommit: fec3683
4. k8s组件安装
下载kubernetes 1.16.x安装包
https://dl.k8s.io/v1.16.2/kubernetes-server-linux-amd64.tar.gz
下载etcd 3.3.15安装包
[root@k8s-master01 ~]# wget https://github.com/etcd-io/etcd/releases/download/v3.3.15/etcd-v3.3.15-linux-amd64.tar.gz
解压kubernetes安装文件
[root@k8s-master01 ~]# tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}
解压etcd安装文件
[root@k8s-master01 ~]# tar -zxvf etcd-v3.3.15-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.3.15-linux-amd64/etcd{,ctl}
版本查看
[root@k8s-master01 ~]# etcd --version etcd Version: 3.3.15 Git SHA: 94745a4ee Go Version: go1.12.9 Go OS/Arch: linux/amd64 [root@k8s-master01 ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} The connection to the server localhost:8080 was refused - did you specify the right host or port?
将组件发送到其他节点
MasterNodes='k8s-master02 k8s-master03' WorkNodes='k8s-node01 k8s-node02' for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done for NODE in $WorkNodes; do scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done
CNI安装，下载CNI组件
wget https://github.com/containernetworking/plugins/releases/download/v0.7.5/cni-plugins-amd64-v0.7.5.tgz
所有节点创建/opt/cni/bin目录
mkdir -p /opt/cni/bin
解压cni并发送至其他节点
tar -zxf cni-plugins-amd64-v0.7.5.tgz -C /opt/cni/bin for NODE in $MasterNodes; do ssh $NODE 'mkdir -p /opt/cni/bin'; scp /opt/cni/bin/* $NODE:/opt/cni/bin/; done for NODE in $WorkNodes; do ssh $NODE 'mkdir -p /opt/cni/bin'; scp /opt/cni/bin/* $NODE:/opt/cni/bin/; done
5. 生成证书
下载生成证书工具
wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O /usr/local/bin/cfssl wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson
所有Master节点创建etcd证书目录
mkdir /etc/etcd/ssl -p
Master01节点生成etcd证书
[root@k8s-master01 pki]# pwd /root/k8s-ha-install/pki [root@k8s-master01 pki]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca 2019/11/02 20:39:26 [INFO] generating a new CA key and certificate from CSR 2019/11/02 20:39:26 [INFO] generate received request 2019/11/02 20:39:26 [INFO] received CSR 2019/11/02 20:39:26 [INFO] generating key: rsa-2048 2019/11/02 20:39:27 [INFO] encoded CSR 2019/11/02 20:39:27 [INFO] signed certificate with serial number 603981780847439086338945186785919518080443607000 [root@k8s-master01 pki]# cfssl gencert \ > -ca=/etc/etcd/ssl/etcd-ca.pem \ > -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \ > -config=ca-config.json \ > -hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.1.19,192.168.1.18,192.168.1.20 \ json | cfssljson -bare /etc/etcd/ssl/etcd > -profile=kubernetes \ > etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd 2019/11/02 20:48:51 [INFO] generate received request 2019/11/02 20:48:51 [INFO] received CSR 2019/11/02 20:48:51 [INFO] generating key: rsa-2048 2019/11/02 20:48:52 [INFO] encoded CSR 2019/11/02 20:48:52 [INFO] signed certificate with serial number 584188914871773268065873129590581350709445633441
将证书复制到其他节点
[root@k8s-master01 pki]# MasterNodes='k8s-master02 k8s-master03' [root@k8s-master01 pki]# WorkNodes='k8s-node01 k8s-node02' [root@k8s-master01 pki]# [root@k8s-master01 pki]# [root@k8s-master01 pki]# [root@k8s-master01 pki]# for NODE in $MasterNodes; do > ssh $NODE "mkdir -p /etc/etcd/ssl" > for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do > scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE} > done > done etcd-ca-key.pem 100% 1675 424.0KB/s 00:00 etcd-ca.pem 100% 1363 334.4KB/s 00:00 etcd-key.pem 100% 1679 457.4KB/s 00:00 etcd.pem 100% 1505 254.5KB/s 00:00 etcd-ca-key.pem 100% 1675 308.3KB/s 00:00 etcd-ca.pem 100% 1363 479.0KB/s 00:00 etcd-key.pem 100% 1679 208.1KB/s 00:00 etcd.pem 100% 1505 398.1KB/s 00:00
生成kubernetes证书
所有节点创建kubernetes相关目录
mkdir -p /etc/kubernetes/pki
[root@k8s-master01 pki]# cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca 2019/11/02 20:56:39 [INFO] generating a new CA key and certificate from CSR 2019/11/02 20:56:39 [INFO] generate received request 2019/11/02 20:56:39 [INFO] received CSR 2019/11/02 20:56:39 [INFO] generating key: rsa-2048 2019/11/02 20:56:39 [INFO] encoded CSR 2019/11/02 20:56:39 [INFO] signed certificate with serial number 204375274212114571715224985915861594178740016435 [root@k8s-master01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -hostname=10.96.0.1,192.168.1.88,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.1.19,192.168.1.18,192.168.1.20 -profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver 2019/11/02 20:58:37 [INFO] generate received request 2019/11/02 20:58:37 [INFO] received CSR 2019/11/02 20:58:37 [INFO] generating key: rsa-2048 2019/11/02 20:58:37 [INFO] encoded CSR 2019/11/02 20:58:37 [INFO] signed certificate with serial number 530242414195398724956103410222477752498137496229 [root@k8s-master01 pki]# cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca netes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client 2019/11/02 20:59:28 [INFO] generating a new CA key and certificate from CSR 2019/11/02 20:59:28 [INFO] generate received request 2019/11/02 20:59:28 [INFO] received CSR 2019/11/02 20:59:28 [INFO] generating key: rsa-2048 2019/11/02 20:59:29 [INFO] encoded CSR 2019/11/02 20:59:29 [INFO] signed certificate with serial number 400365003875188640769336046625866020115935574187 [root@k8s-master01 pki]# [root@k8s-master01 pki]# cfssl gencert -ca=/etc/kubernetes/pki/front-proxy-ca.pem -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client 2019/11/02 20:59:29 [INFO] generate received request 2019/11/02 20:59:29 [INFO] received CSR 2019/11/02 20:59:29 [INFO] generating key: rsa-2048 2019/11/02 20:59:31 [INFO] encoded CSR 2019/11/02 20:59:31 [INFO] signed certificate with serial number 714876805420065531759406103649716563367283489841 [root@k8s-master01 pki]# cfssl gencert \ > -ca=/etc/kubernetes/pki/ca.pem \ > -ca-key=/etc/kubernetes/pki/ca-key.pem \ > -config=ca-config.json \ > -profile=kubernetes \ > manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager 2019/11/02 20:59:56 [INFO] generate received request 2019/11/02 20:59:56 [INFO] received CSR 2019/11/02 20:59:56 [INFO] generating key: rsa-2048 2019/11/02 20:59:56 [INFO] encoded CSR 2019/11/02 20:59:56 [INFO] signed certificate with serial number 365163931170718211641188236585237940196007847195 2019/11/02 20:59:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements"). [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config set-cluster kubernetes \ > --certificate-authority=/etc/kubernetes/pki/ca.pem \ > --embed-certs=true \ > --server=https://192.168.1.88:8443 \ > --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig ig kubectl config set-context system:kube-controller-manager@kubernetes \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager@kubernetes \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig Cluster "kubernetes" set. [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config set-credentials system:kube-controller-manager \ > --client-certificate=/etc/kubernetes/pki/controller-manager.pem \ > --client-key=/etc/kubernetes/pki/controller-manager-key.pem \ > --embed-certs=true \ > --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig User "system:kube-controller-manager" set. [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config set-context system:kube-controller-manager@kubernetes \ > --cluster=kubernetes \ > --user=system:kube-controller-manager \ > --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig Context "system:kube-controller-manager@kubernetes" created. [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config use-context system:kube-controller-manager@kubernetes \ > --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig Switched to context "system:kube-controller-manager@kubernetes". [root@k8s-master01 pki]# cfssl gencert \ > -ca=/etc/kubernetes/pki/ca.pem \ > -ca-key=/etc/kubernetes/pki/ca-key.pem \ > -config=ca-config.json \ > -profile=kubernetes \ > scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler 2019/11/02 21:00:48 [INFO] generate received request 2019/11/02 21:00:48 [INFO] received CSR 2019/11/02 21:00:48 [INFO] generating key: rsa-2048 2019/11/02 21:00:49 [INFO] encoded CSR 2019/11/02 21:00:49 [INFO] signed certificate with serial number 536666325566380578973549296433116161533422391008 2019/11/02 21:00:49 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements"). [root@k8s-master01 pki]# kubectl config set-cluster kubernetes \ > --certificate-authority=/etc/kubernetes/pki/ca.pem \ > --embed-certs=true \ > --server=https://192.168.1.88:8443 \ > --kubeconfig=/etc/kubernetes/scheduler.kubeconfig heduler@kubernetes \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config use-context system:kube-scheduler@kubernetes \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig Cluster "kubernetes" set. [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config set-credentials system:kube-scheduler \ > --client-certificate=/etc/kubernetes/pki/scheduler.pem \ > --client-key=/etc/kubernetes/pki/scheduler-key.pem \ > --embed-certs=true \ > --kubeconfig=/etc/kubernetes/scheduler.kubeconfig User "system:kube-scheduler" set. [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config set-context system:kube-scheduler@kubernetes \ > --cluster=kubernetes \ > --user=system:kube-scheduler \ > --kubeconfig=/etc/kubernetes/scheduler.kubeconfig Context "system:kube-scheduler@kubernetes" created. [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config use-context system:kube-scheduler@kubernetes \ > --kubeconfig=/etc/kubernetes/scheduler.kubeconfig Switched to context "system:kube-scheduler@kubernetes". [root@k8s-master01 pki]# cfssl gencert \ > -ca=/etc/kubernetes/pki/ca.pem \ > -ca-key=/etc/kubernetes/pki/ca-key.pem \ > -config=ca-config.json \ > -profile=kubernetes \ > admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin 2019/11/02 21:01:56 [INFO] generate received request 2019/11/02 21:01:56 [INFO] received CSR 2019/11/02 21:01:56 [INFO] generating key: rsa-2048 2019/11/02 21:01:56 [INFO] encoded CSR 2019/11/02 21:01:56 [INFO] signed certificate with serial number 596648060489336426936623033491721470152303339581 2019/11/02 21:01:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for websites. For more information see the Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org); specifically, section 10.2.3 ("Information Requirements"). [root@k8s-master01 pki]# kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.1.88:8443 --kubeconfig=/etc/kubernetes/admin.kubeconfig s --user=kubernetes-admin --kubeconfig=/etc/kubernetes/admin.kubeconfig kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig Cluster "kubernetes" set. [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config set-credentials kubernetes-admin --client-certificate=/etc/kubernetes/pki/admin.pem --client-key=/etc/kubernetes/pki/admin-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/admin.kubeconfig User "kubernetes-admin" set. [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes --user=kubernetes-admin --kubeconfig=/etc/kubernetes/admin.kubeconfig Context "kubernetes-admin@kubernetes" created. [root@k8s-master01 pki]# [root@k8s-master01 pki]# kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig Switched to context "kubernetes-admin@kubernetes". [root@k8s-master01 pki]# for NODE in k8s-master01 k8s-master02 k8s-master03; do > \cp kubelet-csr.json kubelet-$NODE-csr.json; > sed -i "s/\$NODE/$NODE/g" kubelet-$NODE-csr.json; > cfssl gencert \ > -ca=/etc/kubernetes/pki/ca.pem \ > -ca-key=/etc/kubernetes/pki/ca-key.pem \ > -config=ca-config.json \ > -hostname=$NODE \ > -profile=kubernetes \ > kubelet-$NODE-csr.json | cfssljson -bare /etc/kubernetes/pki/kubelet-$NODE; > rm -f kubelet-$NODE-csr.json > done 2019/11/02 21:03:19 [INFO] generate received request 2019/11/02 21:03:19 [INFO] received CSR 2019/11/02 21:03:19 [INFO] generating key: rsa-2048 2019/11/02 21:03:20 [INFO] encoded CSR 2019/11/02 21:03:20 [INFO] signed certificate with serial number 537071233304209996689594645648717291884343909273 2019/11/02 21:03:20 [INFO] generate received request 2019/11/02 21:03:20 [INFO] received CSR 2019/11/02 21:03:20 [INFO] generating key: rsa-2048 2019/11/02 21:03:20 [INFO] encoded CSR 2019/11/02 21:03:20 [INFO] signed certificate with serial number 93675644052244007817881327407761955585316699106 2019/11/02 21:03:20 [INFO] generate received request 2019/11/02 21:03:20 [INFO] received CSR 2019/11/02 21:03:20 [INFO] generating key: rsa-2048 2019/11/02 21:03:21 [INFO] encoded CSR 2019/11/02 21:03:21 [INFO] signed certificate with serial number 369432006714320793581219965286097236027135622831 [root@k8s-master01 pki]# for NODE in k8s-master01 k8s-master02 k8s-master03; do > ssh $NODE "mkdir -p /etc/kubernetes/pki" > scp /etc/kubernetes/pki/ca.pem $NODE:/etc/kubernetes/pki/ca.pem > scp /etc/kubernetes/pki/kubelet-$NODE-key.pem $NODE:/etc/kubernetes/pki/kubelet-key.pem > scp /etc/kubernetes/pki/kubelet-$NODE.pem $NODE:/etc/kubernetes/pki/kubelet.pem > rm -f /etc/kubernetes/pki/kubelet-$NODE-key.pem /etc/kubernetes/pki/kubelet-$NODE.pem > done ca.pem 100% 1407 494.1KB/s 00:00 kubelet-k8s-master01-key.pem 100% 1679 549.9KB/s 00:00 kubelet-k8s-master01.pem 100% 1501 535.5KB/s 00:00 ca.pem 100% 1407 358.2KB/s 00:00 kubelet-k8s-master02-key.pem 100% 1675 539.8KB/s 00:00 kubelet-k8s-master02.pem 100% 1501 375.5KB/s 00:00 ca.pem 100% 1407 464.7KB/s 00:00 kubelet-k8s-master03-key.pem 100% 1679 566.2KB/s 00:00 kubelet-k8s-master03.pem 100% 1501 515.1KB/s 00:00 [root@k8s-master01 pki]# for NODE in k8s-master01 k8s-master02 k8s-master03; do > ssh $NODE "cd /etc/kubernetes/pki && \ > kubectl config set-cluster kubernetes \ > --certificate-authority=/etc/kubernetes/pki/ca.pem \ > --embed-certs=true \ > --server=https://192.168.1.88:8443 \ > --kubeconfig=/etc/kubernetes/kubelet.kubeconfig && \ > kubectl config set-credentials system:node:${NODE} \ > --client-certificate=/etc/kubernetes/pki/kubelet.pem \ > --client-key=/etc/kubernetes/pki/kubelet-key.pem \ > --embed-certs=true \ > --kubeconfig=/etc/kubernetes/kubelet.kubeconfig && \ > kubectl config set-context system:node:${NODE}@kubernetes \ > --cluster=kubernetes \ > --user=system:node:${NODE} \ > --kubeconfig=/etc/kubernetes/kubelet.kubeconfig && \ > kubectl config use-context system:node:${NODE}@kubernetes \ > --kubeconfig=/etc/kubernetes/kubelet.kubeconfig" > done Cluster "kubernetes" set. User "system:node:k8s-master01" set. Context "system:node:k8s-master01@kubernetes" created. Switched to context "system:node:k8s-master01@kubernetes". Cluster "kubernetes" set. User "system:node:k8s-master02" set. Context "system:node:k8s-master02@kubernetes" created. Switched to context "system:node:k8s-master02@kubernetes". Cluster "kubernetes" set. User "system:node:k8s-master03" set. Context "system:node:k8s-master03@kubernetes" created. Switched to context "system:node:k8s-master03@kubernetes".
创建ServiceAccount Key
[root@k8s-master01 pki]# openssl genrsa -out /etc/kubernetes/pki/sa.key 2048 Generating RSA private key, 2048 bit long modulus (2 primes) ...................................................................................+++++ ...............+++++ e is 65537 (0x010001) [root@k8s-master01 pki]# openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub writing RSA key [root@k8s-master01 pki]# for NODE in k8s-master02 k8s-master03; do > for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do fig; do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE} done done > scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE} > done > for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do > scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE} > done > done admin.csr 100% 1025 303.5KB/s 00:00 admin-key.pem 100% 1675 544.2KB/s 00:00 admin.pem 100% 1440 447.6KB/s 00:00 apiserver.csr 100% 1029 251.9KB/s 00:00 apiserver-key.pem 100% 1679 579.5KB/s 00:00 apiserver.pem 100% 1692 592.0KB/s 00:00 ca.csr 100% 1025 372.6KB/s 00:00 ca-key.pem 100% 1679 535.5KB/s 00:00 ca.pem 100% 1407 496.0KB/s 00:00 controller-manager.csr 100% 1082 273.4KB/s 00:00 controller-manager-key.pem 100% 1679 552.8KB/s 00:00 controller-manager.pem 100% 1497 246.7KB/s 00:00 front-proxy-ca.csr 100% 891 322.2KB/s 00:00 front-proxy-ca-key.pem 100% 1679 408.7KB/s 00:00 front-proxy-ca.pem 100% 1143 436.9KB/s 00:00 front-proxy-client.csr 100% 903 283.5KB/s 00:00 front-proxy-client-key.pem 100% 1679 265.2KB/s 00:00 front-proxy-client.pem 100% 1188 358.8KB/s 00:00 kubelet-k8s-master01.csr 100% 1050 235.6KB/s 00:00 kubelet-k8s-master02.csr 100% 1050 385.3KB/s 00:00 kubelet-k8s-master03.csr 100% 1050 319.8KB/s 00:00 kubelet-key.pem 100% 1679 562.2KB/s 00:00 kubelet.pem 100% 1501 349.4KB/s 00:00 sa.key 100% 1679 455.2KB/s 00:00 sa.pub 100% 451 152.3KB/s 00:00 scheduler.csr 100% 1058 402.0KB/s 00:00 scheduler-key.pem 100% 1675 291.8KB/s 00:00 scheduler.pem 100% 1472 426.2KB/s 00:00 admin.kubeconfig 100% 6432 1.5MB/s 00:00 controller-manager.kubeconfig 100% 6568 1.1MB/s 00:00 scheduler.kubeconfig 100% 6496 601.6KB/s 00:00 admin.csr 100% 1025 368.2KB/s 00:00 admin-key.pem 100% 1675 542.3KB/s 00:00 admin.pem 100% 1440 247.7KB/s 00:00 apiserver.csr 100% 1029 212.4KB/s 00:00 apiserver-key.pem 100% 1679 530.8KB/s 00:00 apiserver.pem 100% 1692 557.3KB/s 00:00 ca.csr 100% 1025 266.4KB/s 00:00 ca-key.pem 100% 1679 408.6KB/s 00:00 ca.pem 100% 1407 404.6KB/s 00:00 controller-manager.csr 100% 1082 248.9KB/s 00:00 controller-manager-key.pem 100% 1679 495.4KB/s 00:00 controller-manager.pem 100% 1497 262.1KB/s 00:00 front-proxy-ca.csr 100% 891 140.9KB/s 00:00 front-proxy-ca-key.pem 100% 1679 418.7KB/s 00:00 front-proxy-ca.pem 100% 1143 362.3KB/s 00:00 front-proxy-client.csr 100% 903 305.6KB/s 00:00 front-proxy-client-key.pem 100% 1679 342.7KB/s 00:00 front-proxy-client.pem 100% 1188 235.1KB/s 00:00 kubelet-k8s-master01.csr 100% 1050 215.1KB/s 00:00 kubelet-k8s-master02.csr 100% 1050 354.0KB/s 00:00 kubelet-k8s-master03.csr 100% 1050 243.0KB/s 00:00 kubelet-key.pem 100% 1679 591.3KB/s 00:00 kubelet.pem 100% 1501 557.0KB/s 00:00 sa.key 100% 1679 387.5KB/s 00:00 sa.pub 100% 451 154.0KB/s 00:00 scheduler.csr 100% 1058 191.5KB/s 00:00 scheduler-key.pem 100% 1675 426.3KB/s 00:00 scheduler.pem 100% 1472 561.3KB/s 00:00 admin.kubeconfig 100% 6432 1.3MB/s 00:00 controller-manager.kubeconfig 100% 6568 1.7MB/s 00:00 scheduler.kubeconfig 100% 6496 1.3MB/s 00:00
6. Kubernetes系统组件配置
etcd配置大致相同，注意修改每个Master节点的etcd配置的主机名和IP地址
name: 'k8s-master01' data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: 'https://192.168.1.19:2380' listen-client-urls: 'https://192.168.1.19:2379,http://127.0.0.1:2379' max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: 'https://192.168.1.19:2380' advertise-client-urls: 'https://192.168.1.19:2379' discovery: discovery-fallback: 'proxy' discovery-proxy: discovery-srv: initial-cluster: 'k8s-master01=https://192.168.1.19:2380,k8s-master02=https://192.168.1.18:2380,k8s-master03=https://192.168.1.20:2380' initial-cluster-token: 'etcd-k8s-cluster' initial-cluster-state: 'new' strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: 'off' proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true peer-transport-security: ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' peer-client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true debug: false log-package-levels: log-output: default force-new-cluster: false
所有Master节点创建etcd service并启动
[root@k8s-master01 pki]# cat /usr/lib/systemd/system/etcd.service [Unit] Description=Etcd Service Documentation=https://coreos.com/etcd/docs/latest/ After=network.target [Service] Type=notify ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml Restart=on-failure RestartSec=10 LimitNOFILE=65536 [Install] WantedBy=multi-user.target Alias=etcd3.service [root@k8s-master01 pki]# mkdir /etc/kubernetes/pki/etcd [root@k8s-master01 pki]# ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/ [root@k8s-master01 pki]# systemctl daemon-reload [root@k8s-master01 pki]# systemctl enable --now etcd Created symlink /etc/systemd/system/etcd3.service → /usr/lib/systemd/system/etcd.service. Created symlink /etc/systemd/system/multi-user.target.wants/etcd.service → /usr/lib/systemd/system/etcd.service.
高可用配置
所有Master节点安装keepalived和haproxy
yum install keepalived haproxy -y
HAProxy配置
[root@k8s-master01 pki]# cat /etc/haproxy/haproxy.cfg global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor listen stats bind *:8006 mode http stats enable stats hide-version stats uri /stats stats refresh 30s stats realm Haproxy\ Statistics stats auth admin:admin frontend k8s-master bind 0.0.0.0:8443 bind 127.0.0.1:8443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server k8s-master01 192.168.1.19:6443 check server k8s-master02 192.168.1.18:6443 check server k8s-master03 192.168.1.20:6443 check
KeepAlived配置
! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 2 weight -5 fall 3 rise 2 } vrrp_instance VI_1 { state MASTER interface ens160 mcast_src_ip 192.168.1.19 virtual_router_id 51 priority 100 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.1.88 } track_script {
chk_apiserver
} }
健康检查配置
[root@k8s-master01 keepalived]# cat /etc/keepalived/check_apiserver.sh #!/bin/bash err=0 for k in $(seq 1 5) do check_code=$(pgrep kube-apiserver) if [[ $check_code == "" ]]; then err=$(expr $err + 1) sleep 5 continue else err=0 break fi done if [[ $err != "0" ]]; then echo "systemctl stop keepalived" /usr/bin/systemctl stop keepalived exit 1 else exit 0 fi
启动HAProxy和KeepAlived
[root@k8s-master01 keepalived]# systemctl enable --now haproxy
[root@k8s-master01 keepalived]# systemctl enable --now keepalived
VIP测试
[root@k8s-master01 pki]# ping 192.168.1.88 PING 192.168.1.88 (192.168.1.88) 56(84) bytes of data. 64 bytes from 192.168.1.88: icmp_seq=1 ttl=64 time=1.39 ms 64 bytes from 192.168.1.88: icmp_seq=2 ttl=64 time=2.46 ms 64 bytes from 192.168.1.88: icmp_seq=3 ttl=64 time=1.68 ms 64 bytes from 192.168.1.88: icmp_seq=4 ttl=64 time=1.08 ms
Kubernetes组件配置
所有节点创建相关目录
[root@k8s-master01 pki]# mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes
所有Master节点创建kube-apiserver service
[root@k8s-master01 pki]# cat /usr/lib/systemd/system/kube-apiserver.service [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-apiserver \ --v=2 \ --logtostderr=true \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --insecure-port=0 \ --advertise-address=192.168.1.88 \ --service-cluster-ip-range=10.96.0.0/12 \ --service-node-port-range=30000-32767 \ --etcd-servers=https://192.168.1.19:2379,https://192.168.1.18:2379,https://192.168.1.20:2379 \ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/pki/sa.pub \ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --enable-bootstrap-token-auth=true \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \ --requestheader-allowed-names=aggregator \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User \ --token-auth-file=/etc/kubernetes/token.csv Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target
[root@k8s-master01 pki]# vim /etc/kubernetes/token.csv [root@k8s-master01 pki]# cat !$ cat /etc/kubernetes/token.csv d7d356746b508a1a478e49968fba7947,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
所有Master节点开启kube-apiserver
[root@k8s-master01 pki]# systemctl enable --now kube-apiserver
所有Master节点配置kube-controller-manager service
[root@k8s-master01 pki]# cat /usr/lib/systemd/system/kube-controller-manager.service [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-controller-manager \ --v=2 \ --logtostderr=true \ --address=127.0.0.1 \ --root-ca-file=/etc/kubernetes/pki/ca.pem \ --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/pki/sa.key \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \ --leader-elect=true \ --use-service-account-credentials=true \ --node-monitor-grace-period=40s \ --node-monitor-period=5s \ --pod-eviction-timeout=2m0s \ --controllers=*,bootstrapsigner,tokencleaner \ --allocate-node-cidrs=true \ --cluster-cidr=10.244.0.0/16 \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --node-cidr-mask-size=24 Restart=always RestartSec=10s [Install] WantedBy=multi-user.target
所有Master节点启动kube-controller-manager
[root@k8s-master01 pki]# systemctl daemon-reload [root@k8s-master01 pki]# systemctl enable --now kube-controller-manager Created symlink /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service → /usr/lib/systemd/system/kube-controller-manager.service.
所有Master节点配置kube-scheduler service
[root@k8s-master01 pki]# cat /usr/lib/systemd/system/kube-scheduler.service [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-scheduler \ --v=2 \ --logtostderr=true \ --address=127.0.0.1 \ --leader-elect=true \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig Restart=always RestartSec=10s [Install] WantedBy=multi-user.target [root@k8s-master01 pki]# systemctl daemon-reload [root@k8s-master01 pki]# systemctl enable --now kube-scheduler Created symlink /etc/systemd/system/multi-user.target.wants/kube-scheduler.service → /usr/lib/systemd/system/kube-scheduler.service.
7. TLS Bootstrapping配置
在Master01创建bootstrap
kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.1.88:8443 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config set-credentials tls-bootstrap-token-user --token=c8ad9c.2e4d610cf3e7426e --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config set-context tls-bootstrap-token-user@kubernetes --cluster=kubernetes --user=tls-bootstrap-token-user --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config use-context tls-bootstrap-token-user@kubernetes --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig
[root@k8s-master01 bootstrap]# pwd /root/k8s-ha-install/bootstrap [root@k8s-master01 bootstrap]# kubectl create -f bootstrap.secret.yaml secret/bootstrap-token-c8ad9c created clusterrolebinding.rbac.authorization.k8s.io/kubelet-bootstrap created clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-bootstrap created clusterrolebinding.rbac.authorization.k8s.io/node-autoapprove-certificate-rotation created clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
8. Node节点配置
复制证书至Node节点
[root@k8s-master01 bootstrap]# for NODE in k8s-node01 k8s-node02; do > ssh $NODE mkdir -p /etc/kubernetes/pki /etc/etcd/ssl /etc/etcd/ssl > for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do > scp /etc/etcd/ssl/$FILE $NODE:/etc/etcd/ssl/ > done > for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do > scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE} > done > done etcd-ca.pem 100% 1363 314.0KB/s 00:00 etcd.pem 100% 1505 429.1KB/s 00:00 etcd-key.pem 100% 1679 361.9KB/s 00:00 ca.pem 100% 1407 459.5KB/s 00:00 ca-key.pem 100% 1679 475.2KB/s 00:00 front-proxy-ca.pem 100% 1143 214.5KB/s 00:00 bootstrap-kubelet.kubeconfig 100% 2291 695.1KB/s 00:00 etcd-ca.pem 100% 1363 325.5KB/s 00:00 etcd.pem 100% 1505 301.2KB/s 00:00 etcd-key.pem 100% 1679 260.9KB/s 00:00 ca.pem 100% 1407 420.8KB/s 00:00 ca-key.pem 100% 1679 398.0KB/s 00:00 front-proxy-ca.pem 100% 1143 224.9KB/s 00:00 bootstrap-kubelet.kubeconfig 100% 2291 685.4KB/s 00:00
所有Node节点创建相关目录
mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/
所有节点配置kubelet service（Master节点不部署Pod也可无需配置）
[root@k8s-master01 bootstrap]# vim /usr/lib/systemd/system/kubelet.service [root@k8s-master01 bootstrap]# cat !$ cat /usr/lib/systemd/system/kubelet.service [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=docker.service Requires=docker.service [Service] ExecStart=/usr/local/bin/kubelet Restart=always StartLimitInterval=0 RestartSec=10 [Install] WantedBy=multi-user.target
[root@k8s-master01 bootstrap]# vim /etc/systemd/system/kubelet.service.d/10-kubelet.conf [root@k8s-master01 bootstrap]# cat !$ cat /etc/systemd/system/kubelet.service.d/10-kubelet.conf [Service] Environment=" KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig" Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml" Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1" ExecStart= ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS
[root@k8s-master01 bootstrap]# vim /etc/kubernetes/kubelet-conf.yml [root@k8s-master01 bootstrap]# cat !$ cat /etc/kubernetes/kubelet-conf.yml apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration address: 0.0.0.0 port: 10250 readOnlyPort: 10255 authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s cgroupDriver: cgroupfs cgroupsPerQOS: true clusterDNS: - 10.96.0.10 clusterDomain: cluster.local containerLogMaxFiles: 5 containerLogMaxSize: 10Mi contentType: application/vnd.kubernetes.protobuf cpuCFSQuota: true cpuManagerPolicy: none cpuManagerReconcilePeriod: 10s enableControllerAttachDetach: true enableDebuggingHandlers: true enforceNodeAllocatable: - pods eventBurst: 10 eventRecordQPS: 5 evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% evictionPressureTransitionPeriod: 5m0s failSwapOn: true fileCheckFrequency: 20s hairpinMode: promiscuous-bridge healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 20s imageGCHighThresholdPercent: 85 imageGCLowThresholdPercent: 80 imageMinimumGCAge: 2m0s iptablesDropBit: 15 iptablesMasqueradeBit: 14 kubeAPIBurst: 10 kubeAPIQPS: 5 makeIPTablesUtilChains: true maxOpenFiles: 1000000 maxPods: 110 nodeStatusUpdateFrequency: 10s oomScoreAdj: -999 podPidsLimit: -1 registryBurst: 10 registryPullQPS: 5 resolvConf: /etc/resolv.conf rotateCertificates: true runtimeRequestTimeout: 2m0s serializeImagePulls: true staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 4h0m0s syncFrequency: 1m0s volumeStatsAggPeriod: 1m0s
启动所有节点kubelet
systemctl daemon-reload
systemctl enable --now kubelet
查看集群状态
[root@k8s-master01 bootstrap]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 NotReady <none> 13m v1.16.2 k8s-master02 NotReady <none> 11m v1.16.2 k8s-master03 NotReady <none> 10m v1.16.2 k8s-node01 NotReady <none> 9m16s v1.16.2 k8s-node02 NotReady <none> 53s v1.16.2
Kube-Proxy配置
kubectl -n kube-system create serviceaccount kube-proxy kubectl create clusterrolebinding system:kube-proxy --clusterrole system:node-proxier --serviceaccount kube-system:kube-proxy SECRET=$(kubectl -n kube-system get sa/kube-proxy \ --output=jsonpath='{.secrets[0].name}') JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET \ --output=jsonpath='{.data.token}' | base64 -d) PKI_DIR=/etc/kubernetes/pki K8S_DIR=/etc/kubernetes kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://192.168.1.88:8443 --kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig kubectl config set-credentials kubernetes --token=${JWT_TOKEN} --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig kubectl config set-context kubernetes --cluster=kubernetes --user=kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig kubectl config use-context kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig
赋值Service文件
[root@k8s-master01 k8s-ha-install]# for NODE in k8s-master01 k8s-master02 k8s-master03; do > scp ${K8S_DIR}/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig > scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf > scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service > done cp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service done kube-proxy.kubeconfig 100% 3112 973.8KB/s 00:00 kube-proxy.conf 100% 813 196.5KB/s 00:00 kube-proxy.service 100% 288 115.5KB/s 00:00 kube-proxy.kubeconfig 100% 3112 580.5KB/s 00:00 kube-proxy.conf 100% 813 160.1KB/s 00:00 kube-proxy.service 100% 288 92.4KB/s 00:00 kube-proxy.kubeconfig 100% 3112 703.1KB/s 00:00 kube-proxy.conf 100% 813 244.1KB/s 00:00 kube-proxy.service 100% 288 54.6KB/s 00:00 [root@k8s-master01 k8s-ha-install]# [root@k8s-master01 k8s-ha-install]# for NODE in k8s-node01 k8s-node02; do > scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig > scp kube-proxy/kube-proxy.conf $NODE:/etc/kubernetes/kube-proxy.conf > scp kube-proxy/kube-proxy.service $NODE:/usr/lib/systemd/system/kube-proxy.service > done kube-proxy.kubeconfig 100% 3112 604.6KB/s 00:00 kube-proxy.conf 100% 813 197.6KB/s 00:00 kube-proxy.service 100% 288 95.5KB/s 00:00 kube-proxy.kubeconfig 100% 3112 749.3KB/s 00:00 kube-proxy.conf 100% 813 206.2KB/s 00:00 kube-proxy.service 100% 288 69.9KB/s 00:00
所有节点启动kube-proxy
[root@k8s-master01 k8s-ha-install]# systemctl daemon-reload [root@k8s-master01 k8s-ha-install]# systemctl enable --now kube-proxy Created symlink /etc/systemd/system/multi-user.target.wants/kube-proxy.service → /usr/lib/systemd/system/kube-proxy.service.
9. 安装calico
[root@k8s-master01 Calico]# cd /root/k8s-ha-install/Calico/ [root@k8s-master01 Calico]# POD_CIDR="10.244.0.0/16" \ > sed -i -e "s?192.168.0.0/16?$POD_CIDR?g" calico.yaml [root@k8s-master01 Calico]# kubectl create -f calico.yaml configmap/calico-config created customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-node created clusterrolebinding.rbac.authorization.k8s.io/calico-node created daemonset.apps/calico-node created serviceaccount/calico-node created deployment.apps/calico-kube-controllers created serviceaccount/calico-kube-controllers created
查看Calico状态
[root@k8s-master01 Calico]# kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-6d85fdfbd8-2jtj9 1/1 Running 0 3m38s calico-node-5t9kj 1/1 Running 0 3m38s calico-node-9ftns 1/1 Running 0 3m38s calico-node-b6rsl 1/1 Running 0 3m38s calico-node-hfqrd 1/1 Running 0 3m38s calico-node-lpcmp 1/1 Running 0 3m38s
查看集群状态
[root@k8s-master01 Calico]# kubectl cluster-info Kubernetes master is running at https://192.168.1.88:8443 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. [root@k8s-master01 Calico]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 Ready <none> 12h v1.16.2 k8s-master02 Ready <none> 13h v1.16.2 k8s-master03 Ready <none> 12h v1.16.2 k8s-node01 Ready <none> 12h v1.16.2 k8s-node02 Ready <none> 11h v1.16.2
至此集群安装完毕，其他组件可以参考本博客其他版本安装的文档。
参考《再也不踩坑的Kubernetes实战指南》
赞助作者一杯奶茶:
Running