[k8s series] ubuntu22.04 in vmware uses kubeadm to install k8s cluster (1.24.4 or higher) (cka test environment construction)

author: ningan123
date: ‘2022-11-28 22:53’
updated: ‘2022-12-17 11:30’

To build a virtual machine, you can refer to another blog post of mine: [Network Growth Notes] Create ubuntu22.04.1 virtual machine in VMware 16.2.2 + modify static ip + change source + mobaxterm remote connection + modify command prompt color (cka test virtual machine construction)

Deployment environment preparation

3 ubuntu22.04 virtual machines

environment modification

Modify hostname (modify each)

hostnamectl set-hostname k8s-master // master节点
hostnamectl set-hostname k8s-node1  // node1节点
hostnamectl set-hostname k8s-node2  // node2节点

Restart the virtual machine, you can see that the hostname has taken effect

Modify the hosts file (master and all nodes)

Modify the /etc/hosts file and add the following content

cat >> /etc/hosts << EOF
192.168.1.100 k8s-master
192.168.1.101 k8s-node1
192.168.1.102 k8s-node2
EOF

Disable swap (master and all nodes)

Execute free -mcommand detection: If the line of Swap is not 0, it means that virtual memory swap is enabled and needs to be disabled.

sudo swapoff -a 
sudo sed -i '/swap/ s/^\(.*\)$/#\1/g' /etc/fstab 
# -i 插入修改模式，g 标识符表示全局查找替换,表示注释掉swap的那一行。 free -h

Modify kernel parameters (master and all nodes)

# Enable kernel modules
sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter



# Add some settings to sysctl
sudo tee /etc/sysctl.d/kubernetes.conf<<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF

# 重新加载 sysctl
sudo sysctl --system

Install containerd

# 安装关联软件
sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates

# 安装docker仓库
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

# 安装containerd
sudo apt update 
sudo apt install -y containerd.io

# 配置 containerd 用systemdcgroup启动.
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

# 重启
sudo systemctl restart containerd 
sudo systemctl enable containerd

You can see that containerd has been successfully started~

# 设置crictl
cat > /etc/crictl.yaml << EOF
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10 
debug: false
EOF

install k8s

Install components (master and all nodes)

# 添加apt repository for Kubernetes
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - 
sudo apt-add-repository "deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main"

apt-get update

# 查看版本
apt-cache madison kubeadm|head

apt install -y kubelet=1.24.4-00 kubeadm=1.24.4-00 kubectl=1.24.4-00

Download mirrors (master and all nodes)

# 使用国内阿里云镜像站点，查看所需镜像
kubeadm config images list \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version=v1.24.4

# 指定版本下载
kubeadm config images pull \
--kubernetes-version=v1.24.4 \
--image-repository registry.aliyuncs.com/google_containers

# 查看镜像
crictl images

Cluster configuration (master)

kubeadm config print init-defaults > kubeadm.yaml

Modify the cluster configuration as follows:
the left is the original exported file, and the right is the modified file

# 
kubeadm init \
--config /root/kubeadm.yaml \
--ignore-preflight-errors=SystemVerification \
--upload-certs

 journalctl -xefu  kubelet

Cluster configuration 2

kubeadm reset

Cluster configuration 3

Cluster configuration 4

The error message of journalctl -xefu kubelet is as follows:

11月 28 23:24:10 k8s-master kubelet[24909]: E1128 23:24:10.831329   24909 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.6\": failed to pull image \"registry.k8s.io/pause:3.6\": failed to pull and unpack image \"registry.k8s.io/pause:3.6\": failed to resolve reference \"registry.k8s.io/pause:3.6\": failed to do request: Head \"https://asia-east1-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6\": dial tcp 108.177.125.82:443: connect: connection refused"

cat /etc/containerd/config.toml  |grep image

# 替换
sed -i "s#registry.k8s.io/pause:3.6#registry.aliyuncs.com/google_containers/pause:3.6#g"  /etc/containerd/config.toml

# 重启containerd
systemctl daemon-reload && systemctl restart containerd

# 重新执行
kubeadm reset

kubeadm init \
--config /root/kubeadm.yaml \
--ignore-preflight-errors=SystemVerification \
--upload-certs

Cluster configuration 5

The error message of journalctl -xefu kubelet is as follows:

11月 29 00:11:19 k8s-master kubelet[42857]: E1129 00:11:19.795996   42857 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format \"slice:prefix:name\" for systemd cgroups, got \"/kubepods/burstable/pod444f719af02c349e8eca993c949ab6b9/eddc1ba340a885d45775c0bf4e1f634abf90ac1474ffea30ea6489ba5edd96ba\" instead: unknown"

Change back to systemd

# 重新执行
kubeadm reset

kubeadm init \
--config /root/kubeadm.yaml \
--ignore-preflight-errors=SystemVerification \
--upload-certs

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 最终配置
[root@k8s-master ~]# cat kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.1.100
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: k8s-master
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
  certSANs:
  - 192.168.1.100
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {
    
    }
dns: {
    
    }
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: 1.24.4
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {
    
    }
---
# 指定cgroup
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

node node

Both nodes execute the following method:

cat /etc/containerd/config.toml  |grep image

sed -i "s#registry.k8s.io/pause:3.6#registry.aliyuncs.com/google_containers/pause:3.6#g"  /etc/containerd/config.toml

systemctl daemon-reload && systemctl restart containerd

kubeadm join 192.168.1.100:6443 --token abcdef.0123456789abcdef \
        --discovery-token-ca-cert-hash sha256:3ce5d8f9fd07a09dd1625531de6876f3b41110c4e005a1c2b5470fbf074ab5cf

The cluster installation is successful~

network plugin

# 使用calico
curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml -o calico-3-24-1.yaml
kubectl apply -f calico-3-24-1.yaml

Wait a little longer after the deployment is complete, and it will become running~

The nodes are all ready~

test

kubectl create deployment nginx --image=nginx
kubectl get pod
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pod,svc

tool learning

grep 'registry.k8s.io/pause:3.6' /var/log/syslog

reference

# [Latest] ubuntu22.04 install kubernetes1.25 k8s1.25 first half configure
Ubuntu22.04 install K8S second half configure kubeadm config configuration ipvs configuration