Please use the command line to delete the previous work deployment container before updating the certificate, so as not to generate garbage data and cause unnecessary trouble to the test. Please ignore the time on the screenshot, just pay attention to the content. After all, when recording this blog, I have updated the certificate according to the tutorial, and they are all supplementary pictures or pictures with similar status found in the search.

1 When the certificate expires

The certificates used by the various components of the kubernetes cluster installed using kubeadm have a period of one year. After the expiration, the certificate becomes invalid, and the kubectl command cannot be used. When using the kubectl command, an error is reported and cannot connect to localhost:8080.

2 Kubernetes environment introduction

192.168.0.190 is the master node, 192.168.0.191 and 192.168.0.192 are node1 and node2 respectively. The operating system is linux 4.19.90-25.2.v2101.gfb01.ky10.aarch64

3 Steps to replace the certificate

3.1 View the expiration time of each certificate on the master

kubeadm certs check-expiration

3.2 View the kubelet certificate list on the master (192.168.0.190)

ls /var/lib/kubelet/pki/

3.3 View the expiration time of the kubelet certificate on the master (192.168.0.190)

openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text | grep Not

3.4 View the expiration time of the kubelet certificate on node1 and node2 (same as (2))

3.5 Backup the directory of relevant certificate files

(It is very important to develop a good habit of backup, and the cluster will be reinstalled if the certificate is invalidated)

cp -r /etc/kubernetes/ /tmp/backup/ # 静态pods配置以及证书
cp -r /var/lib/kubelet/pki/ /tmp/backup.crr #证书pem存放目录

3.6 Rotate the certificate on the master

[root@master kubernetes]#kubeadm certs renew all

(I borrowed the picture, because my certificate has been updated, and the record is made only after the update is completed)

3.7 View the certificate expiration date again

[root@master kubernetes]#kubeadm certs check-expiration

3.8 Check whether the /etc/kubernetes/pki/certificate has been updated

[root@master kubernetes]#ll -a /etc/kubernetes/pki/

The kubeconfig currently used by each component of kubernetes is in /etc/kubernetes/:

[root@master kubernetes]#ll -a /etc/kubernetes/

3.9 Remove the conf file

[root@master kubernetes]#rm -rf /etc/kubernetes/*.conf
[root@master kubernetes]#ls

3.10 Regenerate the kubeconfig file

#注意版本号
[root@master kubernetes]#kubeadm init --kubernetes-version=v1.23.4 phase kubeconfig all

3.11 Check whether a new configuration file has been generated

[root@master kubernetes]# ll -a /etc/kubernetes/

The required kubeconfig file is the above kubelet.conf.

3.12 Recopy the new kubeconfig file used by the administrator

#备份下/root/.kube/config 文件：
cp /root/.kube/config /tmp/kube.old/config
#替换：
[root@master kubernetes]# cp /etc/kubernetes/admin.conf ~/.kube/config

3.13 restart kube-scheduler

[root@master kubernetes]# docker ps -a | grep kube-scheduler | awk ‘{print $1}’
[root@master kubernetes]# docker rm -f 上述容器

#等待kube-scheduler彻底启动起来，即状态为1/1。
[root@master kubernetes]# kubectl get pods -n kube-system | grep scheduler

4 Rotating kubelet certificates

4.1 Rotate the kubelet certificate on the master

#查看kubelet当前所使用的证书
[root@master kubernetes]# ll -a /var/lib/kubelet/pki/


#通过kubeadm certs renew all更新的 k8s 证数，是不会更新 kubelet.conf 的证书的。
#所以此处的证书重启kubelet cho重新生成。
#因为前面已经重新生成了kubelet.conf，现在重启kubelet。
[root@master kubernetes]# systemctl restart kubelet


[root@master kubernetes]# ll -a /var/lib/kubelet/pki/

4.1.1 Check the certificate signing request (referred to as CSR) on the master

[root@master kubernetes]# kubectl get csr

#我的报的是“No Resource Found”,目前看来是不影响使用的，看其他人的是有信息的，不清楚具体原因。


[root@master kubernetes]# kubectl certificate approve csr-vg9bd
certificatesigningrequest.certificates.k8s.io/csr-vg9bd approved(未操作该步骤)

4.1.2 View the expiration time of the current kubelet certificate

[root@master kubernetes]# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text |grep Not


#PS:未过期的话应该是可以不处理的，我当时查看kubelet的证书时间还没过期，所以没有操作证书替换

4.2 Rotate the kubelet certificate on the node

4.2.1 Generating the kubelet.conf file required by node1

#在master上生成worker所需要的kubelet.conf临时放在/tmp下
[root@master tmp]# mkdir -p /tmp/worker

#生成node1所需要的kubelet.conf文件。注意更改成自己的信息
[root@master worker]# kubeadm init --kubernetes-version=v1.23.4 phase kubeconfig kubelet --node-name node1 --kubeconfig-dir /tmp/worker/
#[kubeconfig] Writing “kubelet.conf” kubeconfig file

#node1(192.168.0.191)上的/etc/kubernetes/目录里覆盖原来的kubelet.conf。先备份原先node1的kubelet.conf文件：
[root@node1 kubernetes]# mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubeletconf.bak

[root@master worker]# scp /tmp/worker/kubelet.conf [email protected]:/etc/kubernetes/

4.2.2 Switch to node1 and restart kubelet

[root@node1 kubernetes]# systemctl restart kubelet


#再次查看证书
[root@node1 kubernetes]# ll -a /var/lib/kubelet/pki/


#查看kubelet证书的过期时间。
[root@node1 kubernetes]# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text | grep Not


#切换到master（192.168.0.190）上，查看CSR。(未操作)
[root@master worker]# kubectl get csr

Other nodes operate in the same way as above.

5 Restart the cluster

reboot

Just remember to restart after modifying the configuration. Most of the time, restarting can solve many problems~~

6 sequelae

If the container cannot be automatically scheduled, try to manually allocate each node when there are few nodes, and then it can be scheduled. The specific reason is not clear yet.

If the previous task container appears to be running after the certificate is updated, but it is actually not working, you can use the command line to delete the container, or clean up the namespace in k8s to prevent the container from being unable to restart due to garbage data.

K8S certificate expiration solution to replace the certificate