kubeadm quickly builds a k8s high-availability cluster

Enterprise 2023-12-17 00:47:40 views: null

1. Installation and optimization

1.1 Basic environment configuration

1. Environment introduction
(1). High availability cluster planning

CPU name IP address illustrate
k8s-master01 192.168.2.96 master node
k8s-master02 192.168.2.97 master node
k8s-master03 192.168.2.98 master node
k8s-node01 192.168.2.99 nodenode
k8s-node02 192.168.2.100 nodenode
k8s-master-vip 192.168.2.236 keepalived virtual ip

(2). Network segment planning

Network segment name Network segmentation
Host network segment 192.168.2.1/24
Pod network segment 172.16.0.0/12
Service network segment 10.0.0.0/16

2. Configuration information

Configuration information Remark
system version centos7.9
Docker version 20.10x
kubeadm version v1.23.17 
cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
 docker --version
Docker version 20.10.21, build baeda1f
 kubeadm version
kubeadm version: &version.Info{
    
    Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-02-22T13:33:14Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/amd64"}

Note: The host network segment, K8s Service network segment, and Pod network segment cannot be repeated!!!
3. Modify the host name
( 1) Modify the host name on each machine according to the planning information

hostnamectl set-hostname k8s-master01
hostnamectl set-hostname k8s-master02
hostnamectl set-hostname k8s-master03
hostnamectl set-hostname k8s-node01
hostnamectl set-hostname k8s-node02

4. Modify the hosts file
(1) Install the vim editor. If it is already installed, you can ignore it

yum insytall vim -y

(2) Modify the hosts file of each machine
vim /etc/hosts

192.168.2.96 k8s-master01
192.168.2.97 k8s-master02
192.168.2.98 k8s-master03
192.168.2.236 k8s-master-vip 
192.168.2.99 k8s-node01
192.168.2.100 k8s-node02

Note: If it is not a high-availability cluster, the VIP above is the IP of Master01! ! !

5. Install yum source

(1) Execute the following commands on each machine to configure the default yum source and install dependencies

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2

(2) Execute the following command on each machine to configure Docker’s yum source

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

(3) Execute the following command on each machine to configure the yum source of kubernetes

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

6. Installation of necessary tools

(1) Execute the following command on each machine to install the necessary tools

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git -y

7. Turn off the firewall, swap partition, dnsmasq, selinux

(1) Execute the following command on each machine to turn off the firewall

systemctl disable --now firewalld

(2) Execute the following command on each machine to shut down selinux

 setenforce 0
 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
 sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

(3) Execute the following command on each machine to shut down dnsmasq

systemctl disable --now dnsmasq

Failed to execute operation: No such file or directory
Note: If this is practiced through a VMware virtual machine, an error will be reported because there is no such service!!!

(4) Execute the following command on each machine to shut down NetworkManager

systemctl disable --now NetworkManager

Note: Do not close NetworkManager in public cloud!!!

(5) Execute the following command on each machine to close the swap partition

Temporarily closed

 swapoff -a && sysctl -w vm.swappiness=0

permanently closed

 sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

8. Clock synchronization

(1) Execute the following command on each machine to install ntpdate

 rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
 yum install ntpdate -y

(2) Execute the following command on each machine to synchronize the time

 ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
 echo 'Asia/Shanghai' >/etc/timezone
 ntpdate time2.aliyun.com

#Add scheduled task

 crontab -e
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com

9.Configure limit

(1) Execute the following command on each machine to configure limit

ulimit -SHn 65535

vim /etc/security/limits.conf
末尾添加以下内容
* soft nofile 65536
* hard nofile 131072
* soft nproc 65535
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited

10.Master01 node configuration key-free login

(1) Configure the following command on the Master01 node to enable it to log in to other nodes without a key

ssh-keygen -t rsa #按3次回车即可
for i in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

Note: After this operation is completed, you will be prompted to enter the password of other nodes 4 times!!!
(2) Remotely log in to the k8s-node02 node on the Master01 node to test and find that the test Success

ssh k8s-node02

11. Download source code files

(1) Download the source code file on the Master01 node

git clone https://gitee.com/jeckjohn/k8s-ha-install.git

(2) Execute the following command on the Master01 node to view the branches

cd k8s-ha-install
git branch -a

[root@192 k8s-ha-install]# git branch -a
* master
  remotes/origin/HEAD -> origin/master
  remotes/origin/manual-installation
  remotes/origin/manual-installation-v1.16.x
  remotes/origin/manual-installation-v1.17.x
  remotes/origin/manual-installation-v1.18.x
  remotes/origin/manual-installation-v1.19.x
  remotes/origin/manual-installation-v1.20.x
  remotes/origin/manual-installation-v1.20.x-csi-hostpath
  remotes/origin/manual-installation-v1.21.x
  remotes/origin/manual-installation-v1.22.x
  remotes/origin/manual-installation-v1.23.x
  remotes/origin/manual-installation-v1.24.x
  remotes/origin/manual-installation-v1.25.x
  remotes/origin/manual-installation-v1.26.x
  remotes/origin/manual-installation-v1.27.x
  remotes/origin/manual-installation-v1.28.x
  remotes/origin/master

1.2 Kernel upgrade

centos7.9 kernel upgrade

1.3Containerd as Runtime

If the installed version is lower than 1.24, you can choose either Docker or Containerd. If it is higher than 1.24, choose Containerd as the Runtime.

1. Execute the following command on each machine to install docker-ce-20.10. Note that Containerd will also be installed when installing docker here.

yum install docker-ce-20.10.* docker-ce-cli-20.10.* -y

2. Execute the following command on each machine to configure the modules required by Containerd

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

3. Execute the following command on each machine to load the module

modprobe -- overlay
modprobe -- br_netfilter

4. Execute the following command on each machine to configure the kernel required by Containerd

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

5. Execute the following command on each machine to load the kernel

sysctl --system

6. Execute the following command on each machine to configure the Containerd configuration file

mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml

7. Execute the following command on each machine to change the Cgroup of Containerd to Systemd, find containerd.runtimes.runc.options, and add SystemdCgroup = true (if it already exists, modify it directly, otherwise an error will be reported)

vim /etc/containerd/config.toml
...
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  BinaryName = ""
  CriuImagePath = ""
  CriuPath = ""
  CriuWorkPath = ""
  IoGid = 0
  IoUid = 0
  NoNewKeyring = false
  NoPivotRoot = false
  Root = ""
  ShimCgroup = ""
  SystemdCgroup = true

8. Execute the following command on each machine to change the Pause image of sandbox_image to the address that matches your own version http://registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6

vim /etc/containerd/config.toml

#原本内容
sandbox_image = "registry.k8s.io/pause:3.6"
#修改后的内容
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

9. Execute the following command on each machine to start Containerd and configure automatic startup at boot

systemctl daemon-reload
systemctl enable --now containerd
ls /run/containerd/containerd.sock /run/containerd/containerd.sock

10. Execute the following command on each machine to configure the runtime location of crictl client connection

cat > /etc/crictl.yaml <<EOF
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
EOF

11. Execute the following command on each machine to verify

ctr image ls
REF TYPE DIGEST SIZE PLATFORMS LABELS

1.4 Install Kubernetes components Kubeadm&Kubelet

1. Check the latest Kubernetes version on the Master01 node

yum list kubeadm.x86_64 --showduplicates | sort -r

2. Execute the following commands on each machine to install the latest version 1.23 of kubeadm, kubelet and kubectl

yum install kubeadm-1.23* kubelet-1.23* kubectl-1.23* -y

查看版本
kubeadm version

3. Execute the following command on each machine to change the Kubelet configuration and use Containerd as the Runtime. If you choose docker as the Runtime, no changes are required.

cat >/etc/sysconfig/kubelet<<EOF
KUBELET_KUBEADM_ARGS="--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock"
EOF

4. Execute the following command on each machine to set Kubelet to start automatically at boot (since it has not been initialized yet and there is no kubelet configuration file, kubelet cannot be started at this time and no management is required)

systemctl daemon-reload
systemctl enable --now kubelet

systemctl status kubelet

Note: Since it has not been initialized yet and there is no configuration file for kubelet, kubelet cannot be started at this time and no management is required.

1.5 High availability component installation

Public clouds need to use the load balancing that comes with the public cloud, such as Alibaba Cloud's SLB and Tencent Cloud's ELB, to replace haproxy and keepalived, because most public clouds do not support keepalived. In addition, if you use Alibaba Cloud, kubectl The control end cannot be placed on the master node. It is recommended to use Tencent Cloud because Alibaba Cloud's SLB has a loopback problem, that is, the SLB proxy server cannot access SLB reversely, but Tencent Cloud has fixed this problem.

Note: If it is not a high-availability cluster, haproxy and keepalived do not need to be installed!!!
1. Install HAProxy

(1) All Master nodes install HAProxy and KeepAlived through yum

yum install keepalived haproxy -y

(2) All Master nodes are configured with HAProxy, and the HAProxy configuration of all Master nodes is the same

mkdir /etc/haproxy
vim /etc/haproxy/haproxy.cfg

global
 maxconn 2000
 ulimit-n 16384
 log 127.0.0.1 local0 err
 stats timeout 30s

defaults
 log global
 mode http
 option httplog
 timeout connect 5000
 timeout client 50000
 timeout server 50000
 timeout http-request 15s
 timeout http-keep-alive 15s

frontend monitor-in
 bind *:33305
 mode http
 option httplog
 monitor-uri /monitor

frontend k8s-master
 bind 0.0.0.0:16443
 bind 127.0.0.1:16443
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 default_backend k8s-master

backend k8s-master
 mode tcp
 option tcplog
 option tcp-check
 balance roundrobin
 default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
 server k8s-master01 192.168.2.96:6443 check
 server k8s-master02 192.168.2.97:6443 check
 server k8s-master03 192.168.2.98:6443 check

(3) All Master nodes restart HAProxy and verify port 16443

systemctl restart haproxy
netstat -lntp | grep 16443
tcp        0      0 127.0.0.1:16443         0.0.0.0:*               LISTEN      1075/haproxy        
tcp        0      0 0.0.0.0:16443           0.0.0.0:*

2. Install KeepAlived

All Master nodes are configured with KeepAlived. The configurations are different. Pay attention to distinguishing the IP and network card (interface parameters) of each node.

(1) The configuration of the Master01 node is as follows

! Configuration File for keepalived
global_defs {
    
    
    router_id LVS_DEVEL
    script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    
    
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    
    
    state MASTER
    interface ens33
    mcast_src_ip 192.168.2.96
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
    
    
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
    
    
        192.168.2.236
    }
    track_script {
    
    
        chk_apiserver
    }
}

(2) The configuration of the Master02 node is as follows

! Configuration File for keepalived
global_defs {
    
    
    router_id LVS_DEVEL
    script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    
    
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    
    
    state MASTER
    interface ens33
    mcast_src_ip 192.168.2.97
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
    
    
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
    
    
        192.168.2.236
    }
    track_script {
    
    
        chk_apiserver
    }
}

(3) The configuration of the Master03 node is as follows

! Configuration File for keepalived
global_defs {
    
    
    router_id LVS_DEVEL
    script_user root
    enable_script_security
}
vrrp_script chk_apiserver {
    
    
    script "/etc/keepalived/check_apiserver.sh"
    interval 5
    weight -5
    fall 2
    rise 1
}
vrrp_instance VI_1 {
    
    
    state MASTER
    interface ens33
    mcast_src_ip 192.168.2.98
    virtual_router_id 51
    priority 101
    advert_int 2
    authentication {
    
    
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
    
    
        192.168.2.236
    }
    track_script {
    
    
        chk_apiserver
    }
}

(4) All master nodes configure KeepAlived health check files

vim /etc/keepalived/check_apiserver.sh 
#!/bin/bash

#初始化错误计数器
err=0
#循环三次检查HAProxy进程是否在运行
for k in $(seq 1 3)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        #如果未找到进程，增加错误计数器并等待一秒钟
        err=$(expr $err + 1)
        sleep 1
        continue
    else
        #如果找到进程，重置错误计数器并退出循环
        err=0
        break
    fi
done

#根据错误计数器的值，决定是否停止keepalived服务并退出脚本
if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

#赋权
chmod +x /etc/keepalived/check_apiserver.sh

3. All master nodes start haproxy and keepalived

systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived

4. Test VIP and verify whether keepalived is normal

ping 192.168.2.236 -c 4

2. Cluster construction

2.1Master node initialization

1. The Master01 node creates the kubeadm-config.yaml configuration file as follows

vim kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 7t2weq.bjbawausm0jaxury
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.2.96 #Master01节点的IP地址
  bindPort: 6443
nodeRegistration:
  criSocket: /run/containerd/containerd.sock 
  name: k8s-master01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master

---
apiServer:
  certSANs:
  - 192.168.2.236 #VIP地址/公有云的负载均衡地址
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.2.236:16443
controllerManager: {
    
    }
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.23.17 #此处版本号和kubeadm版本一致
networking:
  dnsDomain: cluster.local
  podSubnet: 172.16.0.0/12
  serviceSubnet: 10.0.0.0/16
scheduler: {
    
    }

2. Update the kubeadm file on the Master01 node

kubeadm config migrate --old-config kubeadm-config.yaml --new-config new.yaml

3. Copy the new.yaml file on the Master01 node to other master nodes

for i in k8s-master02 k8s-master03; do scp new.yaml $i:/root/; done

4. All Master nodes download the image in advance, which can save initialization time (other nodes do not need to change any configuration, including IP addresses)

kubeadm config images pull --config /root/new.yaml

5. All nodes are set to automatically start kubelet when they are powered on.

systemctl enable --now kubelet

6. Master01 node is initialized. After initialization, the corresponding certificate and configuration file will be generated in the /etc/kubernetes directory, and then other Master nodes can join Master01.

kubeadm init --config /root/new.yaml  --upload-certs

After successful normal execution, the following log can be output

[root@k8s-master01 ~]# kubeadm init --config /root/new.yaml  --upload-certs
[init] Using Kubernetes version: v1.23.17
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.0.0.1 192.168.2.96 192.168.2.236]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.2.96 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.2.96 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 22.532424 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "kubelet-config-1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
3b03fabd6969ee744908335536f94e0ac11d15be87edd918d8ad08324ddfdbb2
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 7t2weq.bjbawausm0jaxury
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.2.236:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:c5891bc7b53ee8e7548de96db1f4ed5ef353b77e572910f8aa3965040356701d \
	--control-plane --certificate-key 3b03fabd6969ee744908335536f94e0ac11d15be87edd918d8ad08324ddfdbb2

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.2.236:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:c5891bc7b53ee8e7548de96db1f4ed5ef353b77e572910f8aa3965040356701d

Replenish:

If initialization fails, reset and initialize again, the command is as follows (do not execute if it fails)

kubeadm reset -f ; ipvsadm --clear  ; rm -rf ~/.kube

7. Master01 node configures environment variables for accessing the Kubernetes cluster

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

#查看节点状态
kubectl get node
NAME           STATUS     ROLES                  AGE    VERSION
k8s-master01   NotReady   control-plane,master   4m5s   v1.23.17

2.2 Add Master and Node to k8s cluster

1. Add Master02 node and Master03 node to k8s cluster

kubeadm join 192.168.2.236:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:c5891bc7b53ee8e7548de96db1f4ed5ef353b77e572910f8aa3965040356701d \
	--control-plane --certificate-key 3b03fabd6969ee744908335536f94e0ac11d15be87edd918d8ad08324ddfdbb2

2. Add Node01 node and Node02 node to the k8s cluster

kubeadm join 192.168.2.236:16443 --token 7t2weq.bjbawausm0jaxury \
	--discovery-token-ca-cert-hash sha256:c5891bc7b53ee8e7548de96db1f4ed5ef353b77e572910f8aa3965040356701d

3. Check the node status on the Master01 node

kubectl get node 
NAME           STATUS     ROLES                  AGE     VERSION
k8s-master01   NotReady   control-plane,master   7m11s   v1.23.17
k8s-master02   NotReady   control-plane,master   2m28s   v1.23.17
k8s-master03   NotReady   control-plane,master   102s    v1.23.17
k8s-node01     NotReady   <none>                 106s    v1.23.17
k8s-node02     NotReady   <none>                 84s     v1.23.17

2.3 Calico component installation

1. Enter the corresponding branch directory on the Master01 node

cd /root/k8s-ha-install && git checkout manual-installation-v1.23.x && cd calico/

2. Extract the Pod network segment and assign it to the variable

POD_SUBNET=`cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= '{print $NF}'`

3. Modify calico.yaml file

sed -i "s#POD_CIDR#${POD_SUBNET}#g" calico.yaml

4.Install Calico

kubectl apply -f calico.yaml

5. Check node status

kubectl get node 
NAME           STATUS   ROLES                  AGE   VERSION
k8s-master01   Ready    control-plane,master   9h    v1.23.17
k8s-master02   Ready    control-plane,master   9h    v1.23.17
k8s-master03   Ready    control-plane,master   9h    v1.23.17
k8s-node01     Ready    <none>                 9h    v1.23.17
k8s-node02     Ready    <none>                 9h    v1.23.17

6. Check the pod status and observe that all pods are running

kubectl get po -n kube-system
NAME                                       READY   STATUS    RESTARTS      AGE
calico-kube-controllers-6f6595874c-tntnr   1/1     Running   0             8m52s
calico-node-5mj9g                          1/1     Running   1 (41s ago)   8m52s
calico-node-hhjrv                          1/1     Running   2 (61s ago)   8m52s
calico-node-szjm7                          1/1     Running   0             8m52s
calico-node-xcgwq                          1/1     Running   0             8m52s
calico-node-ztbkj                          1/1     Running   1 (11s ago)   8m52s
calico-typha-6b6cf8cbdf-8qj8z              1/1     Running   0             8m52s
coredns-65c54cc984-nrhlg                   1/1     Running   0             9h
coredns-65c54cc984-xkx7w                   1/1     Running   0             9h
etcd-k8s-master01                          1/1     Running   1 (29m ago)   9h
etcd-k8s-master02                          1/1     Running   1 (29m ago)   9h
etcd-k8s-master03                          1/1     Running   1 (29m ago)   9h
kube-apiserver-k8s-master01                1/1     Running   1 (29m ago)   9h
kube-apiserver-k8s-master02                1/1     Running   1 (29m ago)   9h
kube-apiserver-k8s-master03                1/1     Running   2 (29m ago)   9h
kube-controller-manager-k8s-master01       1/1     Running   2 (29m ago)   9h
kube-controller-manager-k8s-master02       1/1     Running   1 (29m ago)   9h
kube-controller-manager-k8s-master03       1/1     Running   1 (29m ago)   9h
kube-proxy-7rmrs                           1/1     Running   1 (29m ago)   9h
kube-proxy-bmqhr                           1/1     Running   1 (29m ago)   9h
kube-proxy-l9rqg                           1/1     Running   1 (29m ago)   9h
kube-proxy-nn465                           1/1     Running   1 (29m ago)   9h
kube-proxy-sghfb                           1/1     Running   1 (29m ago)   9h
kube-scheduler-k8s-master01                1/1     Running   2 (29m ago)   9h
kube-scheduler-k8s-master02                1/1     Running   1 (29m ago)   9h
kube-scheduler-k8s-master03                1/1     Running   1 (29m ago)   9h

2.4Metrics deployment

In the new version of Kubernetes, Metrics-server is used to collect system resources. Metrics can be used to collect the memory, disk, CPU and network usage of nodes and Pods.

1. Copy front-proxy-ca.crt of Master01 node to Node-01 node and Node-02 node

scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node01:/etc/kubernetes/pki/front-proxy-ca.crt
scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-node02:/etc/kubernetes/pki/front-proxy-ca.crt

2. Install metrics server on Master01 node

cd /root/k8s-ha-install/kubeadm-metrics-server
kubectl  create -f comp.yaml

3. Check the metrics-server deployment status on the Master01 node

kubectl get po -n kube-system -l k8s-app=metrics-server
NAME                              READY   STATUS    RESTARTS   AGE
metrics-server-5cf8885b66-jdjtb   1/1     Running   0          115s

4. Check the node usage on the Master01 node

kubectl top node 
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master01   130m         0%     1019Mi          12%       
k8s-master02   102m         0%     1064Mi          13%       
k8s-master03   93m          0%     971Mi           12%       
k8s-node01     45m          0%     541Mi           6%        
k8s-node02     57m          0%     544Mi           6%

2.5Dashboard deployment

Dashboard is a web-based Kubernetes user interface. You can use Dashboard to deploy container applications to a Kubernetes cluster, troubleshoot container applications, and manage cluster resources. You can use Dashboard to obtain overview information about applications running in the cluster, and you can also create or modify Kubernetes resources (such as Deployment, Job, DaemonSet, etc.). For example, you can auto-scaling a Deployment, initiate a rolling upgrade, restart a Pod, or use a wizard to create a new application. Dashboard also displays resource status information and all error messages in the Kubernetes cluster.

1. Install Dashboard on the Master01 node

cd /root/k8s-ha-install/dashboard/
kubectl  create -f .

2. View the Dashboard service on the Master01 node

kubectl get svc -n kubernetes-dashboard
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.0.159.210   <none>        8000/TCP        2m6s
kubernetes-dashboard        NodePort    10.0.241.159   <none>        443:31822/TCP   2m6s

3. Add startup parameters to the Google Chrome startup file to solve the problem of being unable to access the Dashboard

(1) Right-click Google Chrome (Chrome) and select [Properties]

(2) Add the following parameters at the [target] location. It is emphasized again that there are parameters in front of –test-type --ignore-certificate-errors

–test-type --ignore-certificate-errors
Insert image description here
4. View the token value on the Master01 node

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

5. Open Google Chrome (Chrome) and enter https://any node IP:service port. Here we take the Master01 node as an example.

https://192.168.2.97:32636/

6. Switch the named namespace to kube-system. The default defult namespace has no resources.
Insert image description here

2.6 Set Kube-proxy mode to ipvs

1. Change Kube-proxy to ipvs mode on the Master01 node. The default is iptables.

kubectl edit cm kube-proxy -n kube-system

Insert image description here
2. Update the Kube-Proxy Pod on the Master01 node

kubectl patch daemonset kube-proxy -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}" -n kube-system

3. Check the rolling update status of kube-proxy on the Master01 node

kubectl get po -n kube-system | grep kube-proxy
kube-proxy-2kz9g                           1/1     Running   0             58s
kube-proxy-b54gh                           1/1     Running   0             63s
kube-proxy-kclcc                           1/1     Running   0             61s
kube-proxy-pv8gc                           1/1     Running   0             59s
kube-proxy-xt52m                           1/1     Running   0             56s

4. Verify Kube-Proxy mode on Master01 node

curl 127.0.0.1:10249/proxyMode
ipvs

2.7Kubectl automatic completion

1. Enable kubectl auto-completion on the Master01 node

source <(kubectl completion bash) 
echo "source <(kubectl completion bash)" >> ~/.bashrc

2. Use a shorthand alias for kubectl on the Master01 node

alias k=kubectl
complete -o default -F __start_kubectl k

3. Cluster availability verification

1. Check whether the nodes are normal on the Master01 node and make sure they are all Ready.

kubectl get node
[root@k8s-master01 ~]# kubectl get node
NAME           STATUS   ROLES                  AGE    VERSION
k8s-master01   Ready    control-plane,master   2d2h   v1.23.17
k8s-master02   Ready    control-plane,master   2d     v1.23.17
k8s-master03   Ready    control-plane,master   2d     v1.23.17
k8s-node01     Ready    <none>                 2d     v1.23.17
k8s-node02     Ready    <none>                 2d     v1.23.17
[root@k8s-master01 ~]#

Insert image description here
2. Check whether all Pods are normal on the Master01 node, make sure that READY is in the form of N/N and STATUS is Running
Insert image description here
3. Check the cluster network segment on the Master01 node Conflict

(1) Check the SVC network segment on the Master01 node

kubectl get svc 
NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
kubernetes        ClusterIP   10.0.0.1       <none>        443/TCP           2d2h
nginx             NodePort    10.0.25.21     <none>        80:30273/TCP      47h
springboot-demo   NodePort    10.0.115.157   <none>        30000:30001/TCP   45h

(2) Check the POD network segment on the Master01 node. It is mainly divided into two segments. One segment uses the host network segment because it uses HostNetwork; the other segment uses the POD network segment
Insert image description here
4. Check whether resources are created normally on the Master01 node

(1) Create a deployment named cluster-test on the Master01 node

kubectl create deploy cluster-test --image=registry.cn-hangzhou.aliyuncs.com/zq-demo/debug-tools -- sleep 3600

(2) Check the deployment creation status on the Master01 node

kubectl get po
NAME                               READY   STATUS    RESTARTS      AGE
cluster-test-79b978867f-mcqgr      1/1     Running   2 (10m ago)   2d
nginx-85b98978db-vb9gc             1/1     Running   1 (10m ago)   47h
springboot-demo-6d87f66f6b-fxpgj   1/1     Running   1 (10m ago)   45h
springboot-demo-6d87f66f6b-p4tmd   1/1     Running   1 (10m ago)   45h
springboot-demo-6d87f66f6b-tmtkn   1/1     Running   1 (10m ago)   45h

5. Check whether the Pod can resolve the Service on the Master01 node

(1) Parse kubernetes on the Master01 node and observe that it is consistent with the SVC address above.

[root@k8s-master01 ~]# kubectl exec -it cluster-test-79b978867f-mcqgr -- bash
(06:53 cluster-test-79b978867f-mcqgr:/)  nslookup kubernetes
Server:		10.0.0.10
Address:	10.0.0.10#53

Name:	kubernetes.default.svc.cluster.local
Address: 10.0.0.1

(2) Parse kube-dns.kube-system on the Master01 node and observe that it is consistent with the SVC address above.

(06:53 cluster-test-79b978867f-mcqgr:/) nslookup kube-dns.kube-system
Server:		10.0.0.10
Address:	10.0.0.10#53

Name:	kube-dns.kube-system.svc.cluster.local
Address: 10.0.0.10

6. Whether each node can access kubernetes svc 443 of Kubernetes and service 53 of kube-dns

(1) Test kubernetes svc 443 accessing Kubernetes on each machine

[root@k8s-master02 ~]# curl https://10.0.0.1:443
curl: (60) Peer's Certificate issuer is not recognized.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

(2) Test service 53 accessing Kubernetes’ kube-dns on each machine

curl 10.0.0.10:53
curl: (52) Empty reply from server

7. Whether the Pod and the machine can communicate normally

(1) Check the pod node IP on the Master01 node

[root@k8s-master01 ~]# kubectl get po -owide
NAME                               READY   STATUS    RESTARTS      AGE   IP              NODE         NOMINATED NODE   READINESS GATES
cluster-test-79b978867f-mcqgr      1/1     Running   2 (16m ago)   2d    172.27.14.201   k8s-node02   <none>           <none>
nginx-85b98978db-vb9gc             1/1     Running   1 (16m ago)   47h   172.17.125.7    k8s-node01   <none>           <none>
springboot-demo-6d87f66f6b-fxpgj   1/1     Running   1 (16m ago)   45h   172.27.14.203   k8s-node02   <none>           <none>
springboot-demo-6d87f66f6b-p4tmd   1/1     Running   1 (16m ago)   45h   172.27.14.204   k8s-node02   <none>           <none>
springboot-demo-6d87f66f6b-tmtkn   1/1     Running   1 (16m ago)   45h   172.17.125.9    k8s-node01   <none>           <none>

(2) Ping test on Master01 node

[root@k8s-master01 ~]# ping 172.27.14.201
PING 172.27.14.201 (172.27.14.201) 56(84) bytes of data.
64 bytes from 172.27.14.201: icmp_seq=1 ttl=63 time=0.418 ms
64 bytes from 172.27.14.201: icmp_seq=2 ttl=63 time=0.222 ms
64 bytes from 172.27.14.201: icmp_seq=3 ttl=63 time=0.269 ms
64 bytes from 172.27.14.201: icmp_seq=4 ttl=63 time=0.364 ms
64 bytes from 172.27.14.201: icmp_seq=5 ttl=63 time=0.197 ms
^C
--- 172.27.14.201 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4106ms
rtt min/avg/max/mdev = 0.197/0.294/0.418/0.084 ms
[root@k8s-master01 ~]#

8. Check whether there is normal communication between Pod and Pod

(1) View the Pods in the default namespace on the Master01 node
[root@k8s-master01 ~]# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
cluster-test-79b978867f-mcqgr 1/1 Running 2 (19m ago) 2d 172.27.14.201 k8s-node02 < a i=4> nginx-85b98978db-vb9gc 1/1 Running 1 (19m ago) 47h 172.17.125.7 k8s-node01 springboot-demo-6d87f66f6b-fxpgj 1/1 Running 1 (19m ago) 45h 172.27.14.203 k8s-node02 springboot-demo-6d87f66f6b-p4tmd 1/1 Running 1 (19m ago) 45h 172.27.14.204 k8s-node02 springboot- demo-6d87f66f6b-tmtkn 1/1 Running 1 (19m ago) 45h 172.17.125.9 k8s-node01 You have new mail in /var/spool/mail/root [root@k8s-master01 ~]# (2) Pod under the kube-system namespace on the Master01 node [root@k8s- master01 ~]# kubectl get po -n kube-system -owide







NAME                                       READY   STATUS    RESTARTS      AGE    IP              NODE           NOMINATED NODE   READINESS GATES
calico-kube-controllers-6f6595874c-rvpbg   1/1     Running   4 (20m ago)   2d1h   172.18.195.8    k8s-master03   <none>           <none>
calico-node-6vwpm                          1/1     Running   2 (20m ago)   2d1h   192.168.2.99    k8s-node01     <none>           <none>
calico-node-b8kfr                          1/1     Running   3 (25m ago)   2d1h   192.168.2.96    k8s-master01   <none>           <none>
calico-node-blqnz                          1/1     Running   2 (20m ago)   2d1h   192.168.2.100   k8s-node02     <none>           <none>
calico-node-hpw8m                          1/1     Running   3 (20m ago)   2d1h   192.168.2.98    k8s-master03   <none>           <none>
calico-node-pbb9c                          1/1     Running   2             2d1h   192.168.2.97    k8s-master02   <none>           <none>
calico-typha-6b6cf8cbdf-4jt8j              1/1     Running   2 (20m ago)   2d1h   192.168.2.99    k8s-node01     <none>           <none>
coredns-65c54cc984-cdksw                   1/1     Running   2 (20m ago)   2d3h   172.18.195.9    k8s-master03   <none>           <none>
coredns-65c54cc984-qb5nz                   1/1     Running   2 (20m ago)   2d3h   172.18.195.7    k8s-master03   <none>           <none>
etcd-k8s-master01                          1/1     Running   3 (21m ago)   2d3h   192.168.2.96    k8s-master01   <none>           <none>
etcd-k8s-master02                          1/1     Running   2 (21m ago)   2d1h   192.168.2.97    k8s-master02   <none>           <none>
etcd-k8s-master03                          1/1     Running   2 (20m ago)   2d1h   192.168.2.98    k8s-master03   <none>           <none>
kube-apiserver-k8s-master01                1/1     Running   7 (20m ago)   2d3h   192.168.2.96    k8s-master01   <none>           <none>
kube-apiserver-k8s-master02                1/1     Running   3 (18m ago)   2d1h   192.168.2.97    k8s-master02   <none>           <none>
kube-apiserver-k8s-master03                1/1     Running   3 (20m ago)   2d1h   192.168.2.98    k8s-master03   <none>           <none>
kube-controller-manager-k8s-master01       1/1     Running   5 (25m ago)   2d3h   192.168.2.96    k8s-master01   <none>           <none>
kube-controller-manager-k8s-master02       1/1     Running   2 (21m ago)   2d1h   192.168.2.97    k8s-master02   <none>           <none>
kube-controller-manager-k8s-master03       1/1     Running   3 (19m ago)   2d1h   192.168.2.98    k8s-master03   <none>           <none>
kube-proxy-258rd                           1/1     Running   1 (20m ago)   2d     192.168.2.99    k8s-node01     <none>           <none>
kube-proxy-7vgnc                           1/1     Running   1 (20m ago)   2d     192.168.2.98    k8s-master03   <none>           <none>
kube-proxy-9gsq5                           1/1     Running   1 (21m ago)   2d     192.168.2.97    k8s-master02   <none>           <none>
kube-proxy-fltdx                           1/1     Running   1 (20m ago)   2d     192.168.2.100   k8s-node02     <none>           <none>
kube-proxy-hc7z9                           1/1     Running   1 (25m ago)   2d     192.168.2.96    k8s-master01   <none>           <none>
kube-scheduler-k8s-master01                1/1     Running   5 (25m ago)   2d3h   192.168.2.96    k8s-master01   <none>           <none>
kube-scheduler-k8s-master02                1/1     Running   2 (21m ago)   2d1h   192.168.2.97    k8s-master02   <none>           <none>
kube-scheduler-k8s-master03                1/1     Running   3 (19m ago)   2d1h   192.168.2.98    k8s-master03   <none>           <none>
metrics-server-5cf8885b66-w9dkt            1/1     Running   1 (20m ago)   2d     172.27.14.205   k8s-node02     <none>           <none>

(3) Enter cluster-test-79b978867f-429xg on the Master01 node to perform a ping test

[root@k8s-master01 ~]# kubectl exec -it cluster-test-79b978867f-mcqgr -- bash
(07:02 cluster-test-79b978867f-mcqgr:/) ping 192.168.2.99
PING 192.168.2.99 (192.168.2.99) 56(84) bytes of data.
64 bytes from 192.168.2.99: icmp_seq=1 ttl=63 time=0.260 ms
64 bytes from 192.168.2.99: icmp_seq=2 ttl=63 time=0.431 ms
64 bytes from 192.168.2.99: icmp_seq=3 ttl=63 time=0.436 ms
64 bytes from 192.168.2.99: icmp_seq=4 ttl=63 time=0.419 ms
64 bytes from 192.168.2.99: icmp_seq=5 ttl=63 time=0.253 ms
64 bytes from 192.168.2.99: icmp_seq=6 ttl=63 time=0.673 ms
64 bytes from 192.168.2.99: icmp_seq=7 ttl=63 time=0.211 ms
64 bytes from 192.168.2.99: icmp_seq=8 ttl=63 time=0.374 ms
64 bytes from 192.168.2.99: icmp_seq=9 ttl=63 time=0.301 ms
64 bytes from 192.168.2.99: icmp_seq=10 ttl=63 time=0.194 ms
^C
--- 192.168.2.99 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9217ms
rtt min/avg/max/mdev = 0.194/0.355/0.673/0.137 ms

Guess you like

Origin blog.csdn.net/yangfenggh/article/details/134693975
Recommended
Ranking
#2019110700005
What materials and procedures are required for patent transfer
What is the blockchain Ethereum triplet state root transaction root receipt root
Front-end study notes 04 --- About the insertion of html pictures and videos
Documents required for the filing of WeChat Mini Programs in special industries, the filing process of WeChat Mini Programs in special industries, how to file WeChat Mini Programs in special industries
2017 Qingdao-site tournament I The Squared Mosquito Coil
[BZOJ3165][HEOI2013]Segment (line segment tree without marking)
Kettle series: KettleEasyExpand, an open source Kettle universal plugin by Ma Jinju
The latest tutorial on making framework for iOS
DAX Section 6: Statistical Functions
Daily
More
2024-05-14(9)
2024-05-13(8)
2024-05-12(28)
2024-05-11(32)
2024-05-10(34)
2024-05-09(32)
2024-05-08(18)
2024-05-07(34)
2024-05-06(6)
2024-05-05(0)

Code World

© 2018 codetd.com