K8s集群1.27最新版二进制高可用部署
二进制方式安装Kubernetes高可用集群,虽然安装过程较为复杂,但这也是每个技术人员必须要掌握的内容。同时,在安装过程中,也可以更加深刻地理解每个组件的工作原理。
一、系统环境配置
(1)主机名配置
#参考设置主机名 hostnamectl set-hostname master01 master01 master02 master03 node01 node02 #配置解析 cat >> /etc/hosts <<'EOF' 10.0.0.211 master01 10.0.0.212 master02 10.0.0.213 master03 10.0.0.214 node01 10.0.0.215 node02 EOF
(2)所有节点修改yum源
所有节点CentOS 7安装yum源如下: curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo curl -o /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
(3)所有节点安装常用软件
yum -y install bind-utils expect rsync wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git ntpdate
(4)将master01节点配置免密码登录其他节点
cat > password_login.sh <<'EOF' #!/bin/bash # 创建密钥对 ssh-keygen -t rsa -P "" -f /root/.ssh/id_rsa -q # 声明你服务器密码,建议所有节点的密码均一致,否则该脚本需要再次进行优化 export mypasswd=123.com # 定义主机列表 k8s_host_list=(master01 master02 master03 node01 node02) # 配置免密登录,利用expect工具免交互输入 for i in ${k8s_host_list[@]};do expect -c " spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i expect { \"*yes/no*\" {send \"yes\r\"; exp_continue} \"*password*\" {send \"$mypasswd\r\"; exp_continue} }" done EOF sh password_login.sh
(5)编写数据同步脚本
cat > /usr/local/sbin/data_rsync.sh <<'EOF' #!/bin/bash if [ $# -ne 1 ];then echo "Usage: $0 /path/to/file(绝对路径)" exit fi if [ ! -e $1 ];then echo "[ $1 ] dir or file not find!" exit fi fullpath=`dirname $1` basename=`basename $1` cd $fullpath k8s_host_list=(master01 master02 master03 node01 node02) for host in ${k8s_host_list[@]};do tput setaf 2 echo ===== rsyncing ${host}: $basename ===== tput setaf 7 rsync -az $basename `whoami`@${host}:$fullpath if [ $? -eq 0 ];then echo "命令执行成功!" fi done EOF chmod +x /usr/local/sbin/data_rsync.sh
二、系统环境优化
基础优化
(1)所有节点关闭firewalld,selinux,NetworkManager
systemctl disable --now firewalld systemctl disable --now NetworkManager setenforce 0 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
(2)所有节点关闭swap分区,fstab注释swap
swapoff -a && sysctl -w vm.swappiness=0 sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab free -h
(3)所有节点同步时间
手动同步时区和时间 ln -svf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime ntpdate ntp.aliyun.com - 定期任务同步("crontab -e") */5 * * * * /usr/sbin/ntpdate ntp.aliyun.com
(4)所有节点配置limit
cat >> /etc/security/limits.conf <<'EOF' * soft nofile 655360 * hard nofile 131072 * soft nproc 655350 * hard nproc 655350 * soft memlock unlimited * hard memlock unlimited EOF
(5)所有节点优化sshd服务
sed -i 's@#UseDNS yes@UseDNS no@g' /etc/ssh/sshd_config sed -i 's@^GSSAPIAuthentication yes@GSSAPIAuthentication no@g' /etc/ssh/sshd_config - UseDNS选项: 打开状态下,当客户端试图登录SSH服务器时,服务器端先根据客户端的IP地址进行DNS PTR反向查询出客户端的主机名,然后根据查询出的客户端主机名进行DNS正向A记录查询,验证与其原始IP地址是否一致,这是防止客户端欺骗的一种措施,但一般我们的是动态IP不会有PTR记录,打开这个选项不过是在白白浪费时间而已,不如将其关闭。 - GSSAPIAuthentication: 当这个参数开启( GSSAPIAuthentication yes )的时候,通过SSH登陆服务器时候会有些会很慢!这是由于服务器端启用了GSSAPI。登陆的时候客户端需要对服务器端的IP地址进行反解析,如果服务器的IP地址没有配置PTR记录,那么就容易在这里卡住了。
(6)Linux内核调优
cat > /etc/sysctl.d/k8s.conf <<'EOF' net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv6.conf.all.disable_ipv6 = 1 fs.may_detach_mounts = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl =15 net.ipv4.tcp_max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 65536 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_timestamps = 0 net.core.somaxconn = 16384 EOF sysctl --system
升级内核
为了集群的稳定性和兼容性,生产环境的内核最好升级到4.18版本以上
(1)下载并安装内核软件包 wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm yum -y localinstall kernel-ml* (2)更改内核启动顺序 grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)" grubby --default-kernel (3)更新软件版本,但不需要更新内核,因为内核已经更新到了指定的版本 yum -y update --exclude=kernel*
安装ipvsadm
(1)安装ipvsadm等相关工具 yum -y install ipvsadm ipset sysstat conntrack libseccomp (2)手动加载模块 modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack (3)创建要开机自动加载的模块配置文件 cat > /etc/modules-load.d/ipvs.conf << 'EOF' ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp ip_vs_sh nf_conntrack ip_tables ip_set xt_set ipt_set ipt_rpfilter ipt_REJECT ipip EOF (4)启动模块,如上图所示,这是Linux 3.10.X系列的内核模块,并不是我们需要的! lsmod | grep --color=auto -e ip_vs -e nf_conntrack 温馨提示: 在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack,4.18以下版本使用nf_conntrack_ipv4即可
重启集群
(1)查看现有内核版本 uname -r (2)检查默认加载的内核版本 grubby --default-kernel (3)重启所有节点 reboot (4)检查支持ipvs的内核模块是否加载成功,如上图所示,支持了更多的内核参数。 lsmod | grep --color=auto -e ip_vs -e nf_conntrack (5)再次查看内核版本 uname -r
升级前
升级后验证
三、基础组件安装
所有节点部署containerd环境
#加载 containerd模块 cat >/etc/modules-load.d/containerd.conf<<'EOF' overlay br_netfilter EOF systemctl restart systemd-modules-load.service cat >/etc/sysctl.d/99-kubernetes-cri.conf<<'EOF' net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF # 加载内核 sysctl --system #获取阿里云YUM源 wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo #查看YUM源中Containerd软件 yum list | grep containerd containerd.io.x86_64 1.4.12-3.1.el7 docker-ce-stable #下载安装: yum install -y containerd.io 生成containerd的配置文件 #创建目录 mkdir /etc/containerd -p && containerd config default > /etc/containerd/config.toml #生成配置文件 containerd config default > /etc/containerd/config.toml #编辑配置文件 vim /etc/containerd/config.toml ----- SystemdCgroup = false 改为 SystemdCgroup = true # sandbox_image = "k8s.gcr.io/pause:3.6" 改为: sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6" #启动 systemctl enable --now containerd systemctl status containerd #验证 ctr version runc -version
部署etcd和K8S程序(所有master节点)
(1)下载K8S,etcd的软件包 wget https://dl.k8s.io/v1.27.1/kubernetes-server-linux-amd64.tar.gz wget https://github.com/etcd-io/etcd/releases/download/v3.5.8/etcd-v3.5.8-linux-amd64.tar.gz (2)解压K8S的二进制程序包到PATH环境变量路径 tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} (3)解压etcd的二进制程序包到PATH环境变量路径 tar -xf etcd-v3.5.8-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.8-linux-amd64/etcd{,ctl} (4)将组建发送到其他节点 MasterNodes='master02 master03' WorkNodes='node01 node02' for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done for NODE in $WorkNodes; do scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done (5)查看kubernetes的版本 kube-apiserver --version kube-controller-manager --version kube-scheduler --version etcdctl version kubelet --version kube-proxy --version kubectl version (6)所有节点创建工作目录 mkdir -p /opt/cni/bin (7)切换分支,版本取决于所部署的K8S版本 git clone https://gitee.com/dukuan/k8s-ha-install.git cd k8s-ha-install/ git checkout manual-installation-v1.27.x
四、生成K8S集群证书文件
以下操作均在master01完成即可
master01下载证书管理工具
(1)master01节点下载证书管理工具 wget "https://pkg.cfssl.org/R1.2/cfssl_linux-amd64" -O /usr/local/bin/cfssl wget "https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson (2)所有Master节点创建etcd证书目录 mkdir /etc/etcd/ssl -p (3)所有节点创建kubernetes相关目录 mkdir -p /etc/kubernetes/pki
master01生成etcd证书
(1)生成etcd CA证书和CA证书的key cd /root/k8s-ha-install/pki cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca (2)颁发证书 cfssl gencert \ -ca=/etc/etcd/ssl/etcd-ca.pem \ -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \ -config=ca-config.json \ -hostname=127.0.0.1,master01,master02,master03,10.0.0.211,10.0.0.212,10.0.0.213 \ -profile=kubernetes \ etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd (3)将证书复制到其他节点 MasterNodes='master02 master03' for NODE in $MasterNodes; do ssh $NODE "mkdir -p /etc/etcd/ssl" for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE} done done
k8s组件apiserver相关证书
(1)生成kubernetes证书 cd /root/k8s-ha-install/pki cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca (2)生成apiserver的客户端证书 cfssl gencert -ca=/etc/kubernetes/pki/ca.pem -ca-key=/etc/kubernetes/pki/ca-key.pem -config=ca-config.json -hostname=10.96.0.1,10.0.0.101,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,10.0.0.211,10.0.0.212,10.0.0.213 -profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver (3)生成apiserver的聚合证书 cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca cfssl gencert -ca=/etc/kubernetes/pki/front-proxy-ca.pem -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem -config=ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client 温馨提示: (1)"10.96.0.0"是k8s service的网段,如果说需要更改k8s service网段,那就需要更改"10.96.0.1"; (2)如果不是高可用集群,10.0.0.101为Master01的IP,我这里这个是高可用的vip;
k8s组件controller manager相关证书
生成 controller-manage的证书 cfssl gencert \ -ca=/etc/kubernetes/pki/ca.pem \ -ca-key=/etc/kubernetes/pki/ca-key.pem \ -config=ca-config.json \ -profile=kubernetes \ manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager # 注意,如果不是高可用集群,10.0.0.101:6443改为master01的地址,6443改为apiserver的端口,默认是6443 # set-cluster:设置一个集群项 kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=https://10.0.0.101:6443 \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig # set-credentials 设置一个用户项 kubectl config set-credentials system:kube-controller-manager \ --client-certificate=/etc/kubernetes/pki/controller-manager.pem \ --client-key=/etc/kubernetes/pki/controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig # 设置一个环境项,一个上下文 kubectl config set-context system:kube-controller-manager@kubernetes \ --cluster=kubernetes \ --user=system:kube-controller-manager \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig # 使用某个环境当做默认环境 kubectl config use-context system:kube-controller-manager@kubernetes \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig
k8s组件scheduler相关证书
cfssl gencert \ -ca=/etc/kubernetes/pki/ca.pem \ -ca-key=/etc/kubernetes/pki/ca-key.pem \ -config=ca-config.json \ -profile=kubernetes \ scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler # 注意,如果不是高可用集群,10.0.0.101:6443改为master01的地址,6443改为apiserver的端口,默认是6443 kubectl config set-cluster kubernetes \ --certificate-authority=/etc/kubernetes/pki/ca.pem \ --embed-certs=true \ --server=https://10.0.0.101:6443 \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config set-credentials system:kube-scheduler \ --client-certificate=/etc/kubernetes/pki/scheduler.pem \ --client-key=/etc/kubernetes/pki/scheduler-key.pem \ --embed-certs=true \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config set-context system:kube-scheduler@kubernetes \ --cluster=kubernetes \ --user=system:kube-scheduler \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config use-context system:kube-scheduler@kubernetes \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig
生成admin的证书
cfssl gencert \ -ca=/etc/kubernetes/pki/ca.pem \ -ca-key=/etc/kubernetes/pki/ca-key.pem \ -config=ca-config.json \ -profile=kubernetes \ admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin # 注意,如果不是高可用集群,10.0.0.101:6443改为master01的地址,6443改为apiserver的端口,默认是6443 kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://10.0.0.101:6443 --kubeconfig=/etc/kubernetes/admin.kubeconfig kubectl config set-credentials kubernetes-admin --client-certificate=/etc/kubernetes/pki/admin.pem --client-key=/etc/kubernetes/pki/admin-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/admin.kubeconfig kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes --user=kubernetes-admin --kubeconfig=/etc/kubernetes/admin.kubeconfig kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig 温馨提示: 我们用同样的命令生成了admin.kubeconfig,scheduler.kubeconfig,controller-manager.kubeconfig,它们之间是如何区分的? 我们生成的证书会定义一个用户 admin,它是属于 system:masters 这个组,k8s 安装的时候会有一个 clusterrole,它是一个集群角色,相当于一个配置,它有着集群最高的管理权限,同时会创建一个 clusterrolebinding,它会把 admin 绑到 system:masters 这个组上,然后这个组上的所有用户都会有这个集群的权限
创建ServiceAccount Key
(1)ServiceAccount是k8s一种认证方式,创建ServiceAccount的时候会创建一个与之绑定的secret,这个secret会生成一个token openssl genrsa -out /etc/kubernetes/pki/sa.key 2048 openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub (2)发送证书至其他节点 for NODE in master02 master03; do for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done
五、etcd配置
1.master节点分别创建配置文件
master01节点的配置文件
cat > /etc/etcd/etcd.config.yml <<'EOF' name: 'master01' data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: 'https://10.0.0.211:2380' listen-client-urls: 'https://10.0.0.211:2379,http://127.0.0.1:2379' max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: 'https://10.0.0.211:2380' advertise-client-urls: 'https://10.0.0.211:2379' discovery: discovery-fallback: 'proxy' discovery-proxy: discovery-srv: initial-cluster: 'master01=https://10.0.0.211:2380,master02=https://10.0.0.212:2380,master03=https://10.0.0.213:2380' initial-cluster-token: 'etcd-k8s-cluster' initial-cluster-state: 'new' strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: 'off' proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true peer-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' peer-client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false EOF
master02节点的配置文件
cat > /etc/etcd/etcd.config.yml << 'EOF' name: 'master02' data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: 'https://10.0.0.212:2380' listen-client-urls: 'https://10.0.0.212:2379,http://127.0.0.1:2379' max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: 'https://10.0.0.212:2380' advertise-client-urls: 'https://10.0.0.212:2379' discovery: discovery-fallback: 'proxy' discovery-proxy: discovery-srv: initial-cluster: 'master01=https://10.0.0.211:2380,master02=https://10.0.0.212:2380,master03=https://10.0.0.213:2380' initial-cluster-token: 'etcd-k8s-cluster' initial-cluster-state: 'new' strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: 'off' proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true peer-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' peer-client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false EOF
master03节点的配置文件
cat > /etc/etcd/etcd.config.yml << 'EOF' name: 'master03' data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: 'https://10.0.0.213:2380' listen-client-urls: 'https://10.0.0.213:2379,http://127.0.0.1:2379' max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: 'https://10.0.0.213:2380' advertise-client-urls: 'https://10.0.0.213:2379' discovery: discovery-fallback: 'proxy' discovery-proxy: discovery-srv: initial-cluster: 'master01=https://10.0.0.211:2380,master02=https://10.0.0.212:2380,master03=https://10.0.0.213:2380' initial-cluster-token: 'etcd-k8s-cluster' initial-cluster-state: 'new' strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: 'off' proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true peer-transport-security: cert-file: '/etc/kubernetes/pki/etcd/etcd.pem' key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem' peer-client-cert-auth: true trusted-ca-file: '/etc/kubernetes/pki/etcd/etcd-ca.pem' auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false EOF
2.所有master节点启动etcd服务
(1)创建启动脚本 cat > /usr/lib/systemd/system/etcd.service <<'EOF' [Unit] Description=Etcd Service Documentation=https://coreos.com/etcd/docs/latest/ After=network.target [Service] Type=notify ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml Restart=on-failure RestartSec=10 LimitNOFILE=65536 [Install] WantedBy=multi-user.target Alias=etcd3.service EOF (2)启动服务 mkdir /etc/kubernetes/pki/etcd ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/ systemctl daemon-reload systemctl enable --now etcd systemctl status etcd (3)查看etcd状态 etcdctl --endpoints="10.0.0.211:2379,10.0.0.212:2379,10.0.0.213:2379" --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint status --write-out=table 温馨提示: etcd集群启动成功如图所示。
六、高可用配置
(haproxy+keepalived)
1.所有master节点安装keepalived和haproxy
yum -y install keepalived haproxy
2.所有master节点配置haproxy,配置文件各个节点相同
(1)备份配置文件 cp /etc/haproxy/haproxy.cfg{,`date +%F`} (2)所有节点的配置文件内容相同 cat > /etc/haproxy/haproxy.cfg <<'EOF' global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor frontend k8s-master bind 0.0.0.0:16443 bind 127.0.0.1:16443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server master01 10.0.0.211:6443 check server master02 10.0.0.212:6443 check server master03 10.0.0.213:6443 check EOF
3.所有master节点配置keepalived,配置文件各节点不同
备份配置文件
cp /etc/keepalived/keepalived.conf{,`date +%F`}
"master01"节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF' ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER interface eth0 mcast_src_ip 10.0.0.211 virtual_router_id 51 priority 101 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 10.0.0.101 } track_script { chk_apiserver } } EOF
"master02"节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF' ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER interface eth0 mcast_src_ip 10.0.0.212 virtual_router_id 51 priority 101 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 10.0.0.101 } track_script { chk_apiserver } } EOF
"master03"节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF' ! Configuration File for keepalived global_defs { router_id LVS_DEVEL script_user root enable_script_security } vrrp_script chk_apiserver { script "/etc/keepalived/check_apiserver.sh" interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER interface eth0 mcast_src_ip 10.0.0.213 virtual_router_id 51 priority 101 advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 10.0.0.101 } track_script { chk_apiserver } } EOF
温馨提示: 上述keepalived配置文件中的网卡名称是ens33,如果你的网卡名称是eth0,需要修改配置文件,否则keepalived启动后会自动退出或没有VIP
4.所有master节点配置KeepAlived健康检查文件
(1)创建检查脚本 cat > /etc/keepalived/check_apiserver.sh <<'EOF' #!/bin/bash err=0 for k in $(seq 1 3) do check_code=$(pgrep haproxy) if [[ $check_code == "" ]]; then err=$(expr $err + 1) sleep 1 continue else err=0 break fi done if [[ $err != "0" ]]; then echo "systemctl stop keepalived" /usr/bin/systemctl stop keepalived exit 1 else exit 0 fi EOF (2)添加执行权限 chmod +x /etc/keepalived/check_apiserver.sh 温馨提示: (1)我们通过KeepAlived虚拟出来一个VIP,VIP会配置到一个master节点上面,它会通过haproxy暴露的16443的端口反向代理到我们的三个master节点上面,所以我们可以通过VIP的地址加上16443访问到我们的API server; (2)健康检查会检查haproxy的状态,三次失败就会将KeepAlived停掉,停掉之后KeepAlived会跳到其他的节点;
5.启动服务
(1)启动harproxy systemctl daemon-reload systemctl enable --now haproxy (2)启动keepalived systemctl enable --now keepalived systemctl status keepalived (3)查看VIP,如图所示 ip a (4) 修改过网卡名执行此操作 sed -i 's#ens33#eth0#g' /etc/keepalived/keepalived.conf systemctl restart keepalived.service
七、二进制K8smaster组件配置
所有master节点启动Apiserver服务
#所有节点执行 (1)所有节点(k8s-master0[1-3])创建工作目录 mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes #master01执行 (2)master01节点创建配置文件 cat > /usr/lib/systemd/system/kube-apiserver.service << 'EOF' [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-apiserver \ --v=2 \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --advertise-address=10.0.0.211 \ --service-cluster-ip-range=10.96.0.0/12 \ --service-node-port-range=30000-32767 \ --etcd-servers=https://10.0.0.211:2379,https://10.0.0.212:2379,https://10.0.0.213:2379 \ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/pki/sa.pub \ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --enable-bootstrap-token-auth=true \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \ --requestheader-allowed-names=aggregator \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User # --token-auth-file=/etc/kubernetes/token.csv Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF #master02执行 (3)master02节点创建配置文件 cat > /usr/lib/systemd/system/kube-apiserver.service <<'EOF' [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-apiserver \ --v=2 \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --advertise-address=10.0.0.212 \ --service-cluster-ip-range=10.96.0.0/12 \ --service-node-port-range=30000-32767 \ --etcd-servers=https://10.0.0.211:2379,https://10.0.0.212:2379,https://10.0.0.213:2379 \ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/pki/sa.pub \ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --enable-bootstrap-token-auth=true \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \ --requestheader-allowed-names=aggregator \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User # --token-auth-file=/etc/kubernetes/token.csv Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF #master03执行 (4)master03节点创建配置文件 cat > /usr/lib/systemd/system/kube-apiserver.service << 'EOF' [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-apiserver \ --v=2 \ --allow-privileged=true \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --advertise-address=10.0.0.213 \ --service-cluster-ip-range=10.96.0.0/12 \ --service-node-port-range=30000-32767 \ --etcd-servers=https://10.0.0.211:2379,https://10.0.0.212:2379,https://10.0.0.213:2379 \ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \ --etcd-certfile=/etc/etcd/ssl/etcd.pem \ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \ --client-ca-file=/etc/kubernetes/pki/ca.pem \ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \ --service-account-key-file=/etc/kubernetes/pki/sa.pub \ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \ --service-account-issuer=https://kubernetes.default.svc.cluster.local \ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --enable-bootstrap-token-auth=true \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \ --requestheader-allowed-names=aggregator \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User # --token-auth-file=/etc/kubernetes/token.csv Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF #所有master节点执行 (5)启动服务 systemctl daemon-reload && systemctl enable --now kube-apiserver && systemctl status kube-apiserver
所有master节点启动ControllerManager服务
(1)所有节点创建配置文件 cat > /usr/lib/systemd/system/kube-controller-manager.service << 'EOF' [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-controller-manager \ --v=2 \ --bind-address=127.0.0.1 \ --root-ca-file=/etc/kubernetes/pki/ca.pem \ --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \ --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \ --service-account-private-key-file=/etc/kubernetes/pki/sa.key \ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \ --leader-elect=true \ --use-service-account-credentials=true \ --node-monitor-grace-period=40s \ --node-monitor-period=5s \ --controllers=*,bootstrapsigner,tokencleaner \ --allocate-node-cidrs=true \ --cluster-cidr=172.16.0.0/12 \ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \ --node-cidr-mask-size=24 Restart=always RestartSec=10s [Install] WantedBy=multi-user.target EOF (2)启动服务,查看状态 systemctl daemon-reload systemctl enable --now kube-controller-manager systemctl status kube-controller-manager
所有master节点启动Scheduler服务
(1)所有节点创建配置文件 cat > /usr/lib/systemd/system/kube-scheduler.service <<'EOF' [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-scheduler \ --v=2 \ --bind-address=127.0.0.1 \ --leader-elect=true \ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig Restart=always RestartSec=10s [Install] WantedBy=multi-user.target EOF (2)启动服务并查看状态,如上图所示 systemctl daemon-reload systemctl enable --now kube-scheduler systemctl status kube-scheduler
八、创建Bootstrapping自动颁发证书
1.master01节点创建bootstrap-kubelet.kubeconfig文件
cd /root/k8s-ha-install/bootstrap kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://10.0.0.101:6443 --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config set-credentials tls-bootstrap-token-user --token=c8ad9c.2e4d610cf3e7426e --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config set-context tls-bootstrap-token-user@kubernetes --cluster=kubernetes --user=tls-bootstrap-token-user --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config use-context tls-bootstrap-token-user@kubernetes --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig 温馨提示: "bootstrap-kubelet.kubeconfig"是一个keepalived用来向apiserver申请证书的文件,如果要修改bootstrap.secret.yaml的token-id和token-secret,需要保证c8ad9c字符串一致的,并且位数是一样的。还要保证上个命令的黄色字体:c8ad9c.2e4d610cf3e7426e与你修改的字符串要一致
2.所有master节点拷贝管理证书
所有master节点执行此操作
mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config
3.创建bootstrap
kubectl create -f bootstrap.secret.yaml
九、部署Node节点
分发证书
cd /etc/kubernetes/ for NODE in master02 master03 node01 node02; do ssh $NODE mkdir -p /etc/kubernetes/pki /etc/etcd/ssl /etc/etcd/ssl for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do scp /etc/etcd/ssl/$FILE $NODE:/etc/etcd/ssl/ done for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE} done done 温馨提示: node节点使用自动颁发证书的形式配置
Kubelet配置
集群所有节点操作
(1)所有节点创建工作目录 mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/ (2)所有节点配置kubelet service cat > /usr/lib/systemd/system/kubelet.service <<'EOF' [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=docker.service [Service] ExecStart=/usr/local/bin/kubelet Restart=always StartLimitInterval=0 RestartSec=10 [Install] WantedBy=multi-user.target EOF (3)所有节点配置kubelet service的配置文件 cat > /etc/systemd/system/kubelet.service.d/kubelet.conf <<'EOF' [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig" Environment="KUBELET_SYSTEM_ARGS=--runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause-amd64:3.2" Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' " ExecStart= ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS EOF (4)所有节点创建kubelet的配置文件 cat > /etc/kubernetes/kubelet-conf.yml <<'EOF' apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration address: 0.0.0.0 port: 10250 readOnlyPort: 10255 authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s cgroupDriver: systemd cgroupsPerQOS: true clusterDNS: - 10.96.0.10 clusterDomain: cluster.local containerLogMaxFiles: 5 containerLogMaxSize: 10Mi contentType: application/vnd.kubernetes.protobuf cpuCFSQuota: true cpuManagerPolicy: none cpuManagerReconcilePeriod: 10s enableControllerAttachDetach: true enableDebuggingHandlers: true enforceNodeAllocatable: - pods eventBurst: 10 eventRecordQPS: 5 evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% evictionPressureTransitionPeriod: 5m0s failSwapOn: true fileCheckFrequency: 20s hairpinMode: promiscuous-bridge healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 20s imageGCHighThresholdPercent: 85 imageGCLowThresholdPercent: 80 imageMinimumGCAge: 2m0s iptablesDropBit: 15 iptablesMasqueradeBit: 14 kubeAPIBurst: 10 kubeAPIQPS: 5 makeIPTablesUtilChains: true maxOpenFiles: 1000000 maxPods: 110 nodeStatusUpdateFrequency: 10s oomScoreAdj: -999 podPidsLimit: -1 registryBurst: 10 registryPullQPS: 5 resolvConf: /etc/resolv.conf rotateCertificates: true runtimeRequestTimeout: 2m0s serializeImagePulls: true staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 4h0m0s syncFrequency: 1m0s volumeStatsAggPeriod: 1m0s EOF (5)启动所有节点kubelet systemctl daemon-reload systemctl enable --now kubelet systemctl status kubelet (6)在master101节点上查看node信息,如上图所示。 kubectl get nodes
kube-proxy配置
(1)在“master01”节点生成"/etc/kubernetes/kube-proxy.kubeconfig"配置文件 cd /root/k8s-ha-install kubectl -n kube-system create serviceaccount kube-proxy kubectl create clusterrolebinding system:kube-proxy --clusterrole system:node-proxier --serviceaccount kube-system:kube-proxy SECRET=$(kubectl -n kube-system get sa/kube-proxy \ --output=jsonpath='{.secrets[0].name}') JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET \ --output=jsonpath='{.data.token}' | base64 -d) PKI_DIR=/etc/kubernetes/pki K8S_DIR=/etc/kubernetes kubectl config set-cluster kubernetes --certificate-authority=/etc/kubernetes/pki/ca.pem --embed-certs=true --server=https://10.0.0.101:6443 --kubeconfig=${K8S_DIR}/kube-proxy.kubeconfig kubectl config set-credentials kubernetes --token=${JWT_TOKEN} --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig kubectl config set-context kubernetes --cluster=kubernetes --user=kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig kubectl config use-context kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig (2)在“master01”将kube-proxy的systemd Service文件发送到其他节点 for NODE in master01 master02 master03 node01 node02; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig done (3)集群所有节点创建kube-proxy.conf配置文件 cat > /etc/kubernetes/kube-proxy.conf << EOF KUBE_PROXY_OPTS="--v=2 \ --config=/etc/kubernetes/kube-proxy-config.yml" EOF # 注意修改各个节点的"hostnameOverride"的值 cat > /etc/kubernetes/kube-proxy-config.yml << EOF kind: KubeProxyConfiguration apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 metricsBindAddress: 0.0.0.0:10249 clientConnection: kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig hostnameOverride: node02 #每个节点的名称都是不同的,注意修改 clusterCIDR: 172.30.0.0/16 EOF (4)所有节点使用systemd管理kube-proxy cat > /usr/lib/systemd/system/kube-proxy.service << EOF [Unit] Description=Kubernetes Proxy After=network.target [Service] EnvironmentFile=/etc/kubernetes/kube-proxy.conf ExecStart=/usr/local/bin/kube-proxy \$KUBE_PROXY_OPTS Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF (5)所有节点启动kube-proxy systemctl daemon-reload systemctl enable --now kube-proxy systemctl status kube-proxy 温馨提示: 如果更改了集群Pod的网段,需要更改kube-proxy.conf的clusterCIDR参数,比如我上面的案例自定义的网段为"172.30.0.0/16"。
十、部署网络插件
1.部署calico网络插件
cd /root/k8s-ha-install/calico/ # 更改此处为自己的pod网段 POD_SUBNET="172.30.0.0/16" sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@# value: "192.168.0.0/16"@ value: '"${POD_SUBNET}"'@g' calico.yaml kubectl apply -f calico.yaml
2.观察各节点是否部署成功
kubectl get pods -A
十一、附加组件部署
1.部署CoreDNS
(1)部署coreDNS cd /root/k8s-ha-install/ 修改"clusterIP"的值,如下图: kubectl create -f CoreDNS/coredns.yaml (2)查看状态 kubectl get po -n kube-system -l k8s-app=kube-dns (3)验证DNS组件 dig @10.96.0.10 metrics-server.kube-system.svc.cluster.local +short
2.部署Metrics Server
(1)部署Metrics Server cd /root/k8s-ha-install/metrics-server kubectl create -f . (2)查看node和pod的监控状态 kubectl top no kubectl top po -A
3.安装dashboard
(1)安装dashboard服务 cd /root/k8s-ha-install/dashboard/ kubectl create -f . (2)查看token并访问dashboard,如下图所示 kubectl get svc kubernetes-dashboard -n kubernetes-dashboard kubectl get pod -A -o wide | grep dashboard kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}') #浏览器访问https://10.0.0.101:30069 #如果访问页面出现不安全,鼠标点击空白处,输入thisisunsafe即可
十二、优化
1.自动补全功能
- docker自动补全功能 yum -y install bash-completion source /usr/share/bash-completion/bash_completion - kubectl 自动补全功能 echo "source <(kubectl completion bash)" >> ~/.bashrc && source ~/.bashrc
2.多master管理K8S集群验证
如果有任意一个master节点有问题,请验证kubeconfig文件是否有配置。 参考指令: mkdir -p /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config
3.验证集群高可用
如上图所示,K8S集群的VIP最开始在"master03"节点。 接下来,我们将该节点停机后,并不会影响K8S集群正常使用
4.测试集群是否正常
创建一个nginx的pod资源
cat >nginx.yaml<<'EOF' apiVersion: v1 kind: Pod metadata: name: web spec: containers: - name: nginx image: nginx:1.21 EOF kubectl apply -f nginx.yaml
至此,二进制部署kubernetes高可用集群,部署完成。