kubernetes certificate online update

Expiration time of each certificate

/etc/kubernetes/pki/apiserver.crt                #1年有效期
/etc/kubernetes/pki/front-proxy-ca.crt           #10年有效期
/etc/kubernetes/pki/ca.crt                       #10年有效期
/etc/kubernetes/pki/apiserver-etcd-client.crt    #1年有效期
/etc/kubernetes/pki/front-proxy-client.crt       #1年有效期
/etc/kubernetes/pki/etcd/server.crt              #1年有效期
/etc/kubernetes/pki/etcd/ca.crt                  #10年有效期
/etc/kubernetes/pki/etcd/peer.crt                #1年有效期
/etc/kubernetes/pki/etcd/healthcheck-client.crt  #1年有效期
/etc/kubernetes/pki/apiserver-kubelet-client.crt #1年有效期

Can be used to check the expiration time of the certificate:

method 1

kubeadm alpha certs check-expiration

Method 2

openssl x509 -noout -dates -in /etc/kubernetes/pki/apiserver.crt
kubeadm alpha certs 命令详解:
Available Commands:
  certificate-key  生成证书和key
  check-expiration  检测证书过期时间
  renew            续订Kubernetes集群的证书
  kubeadm alpha certs命令仅支持v1.15及其以上的版本。
kubeadm alpha certs check-expiration

This command displays the expiration time/remaining time of the client certificate in the /etc/kubernetes/pki folder and the client certificate embedded in the KUBECONFIG file used by kubeadm.

If you are running a highly available cluster, this command needs to be executed on all control panel nodes.

Next, let's update our cluster certificate. The following operations are performed on the master node. First, back up the original certificate:

mkdir /etc/kubernetes.bak
cp -r /etc/kubernetes/pki/ /etc/kubernetes.bak
cp /etc/kubernetes/*.conf /etc/kubernetes.bak
然后备份 etcd 数据目录:
cp -r /var/lib/etcd /var/lib/etcd.bak

Update certificate with Kubernetes certificate API

In addition to the one-click manual certificate update mentioned above, you can also use the Kubernetes certificate API to perform manual certificate update. For the online environment, we may not take the risk of frequently updating the cluster or updating the certificate. These are risky after all. Therefore, we hope that the validity period of the generated certificate is long enough. Although this is not recommended from a security perspective, it is not recommended. In some scenarios, a sufficiently long certificate validity period is also very necessary. Many administrators manually change the source code of kubeadm to 10 years and then recompile to create a cluster. Although this method can achieve the purpose, it is not recommended to use this method, especially when you want to update the cluster. It must be updated with the new version. In fact, Kubernetes provides an API method to help us generate a long enough certificate validity period.
To use the built-in API method to sign, first we need to configure the --experimental-cluster-signing-duration parameter of the kube-controller-manager component and adjust it to 10 years. Here is the cluster installed by kubeadm, so we directly modify the static The Pod's yaml file can be:

vim /etc/kubernetes/manifests/kube-controller-manager.yaml
......
spec:
  containers:
  - command:
    - kube-controller-manager
     #设置证书有效期为 10 年
    - --experimental-cluster-signing-duration=87600h 
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
......

After the modification is completed, kube-controller-manager will automatically restart to take effect. Then we need to use the following command to create a certificate signing request for the Kubernetes certificate API. If you set up an external signer such as cert-manager, certificate signing requests (CSRs) will be automatically approved. Otherwise, you must manually approve the certificate using the kubectl certificate command. The following kubeadm command outputs the name of the certificate to be approved, and then waits for approval to occur:

kubeadm alpha certs renew all --use-api --config kubeadm.yaml &

The output is similar to the following:

[1] 2890
[certs] Certificate request "kubeadm-cert-kubernetes-admin-pn99f" created

#然后接下来我们需要去手动批准证书:

kubectl get csr
NAME                                  AGE   REQUESTOR          CONDITION
kubeadm-cert-kubernetes-admin-pn99f   64s   kubernetes-admin   Pending

Manually approve the certificate

kubectl certificate approve kubeadm-cert-kubernetes-admin-pn99f
certificatesigningrequest.certificates.k8s.io/kubeadm-cert-kubernetes-admin-pn99f approved 

用同样的方式为处于 Pending 状态的 csr 执行批准操作,直到所有的 csr 都批准完成为止。最后所有的 csr 列表状态如下所示:

kubectl get csr
NAME                                                AGE     REQUESTOR          CONDITION
kubeadm-cert-front-proxy-client-llhrj               30s     kubernetes-admin   Approved,Issued
kubeadm-cert-kube-apiserver-2s6kf                   2m43s   kubernetes-admin   Approved,Issued
kubeadm-cert-kube-apiserver-etcd-client-t9pkx       2m7s    kubernetes-admin   Approved,Issued
kubeadm-cert-kube-apiserver-kubelet-client-pjbjm    108s    kubernetes-admin   Approved,Issued
kubeadm-cert-kube-etcd-healthcheck-client-8dcn8     64s     kubernetes-admin   Approved,Issued
kubeadm-cert-kubernetes-admin-pn99f                 4m29s   kubernetes-admin   Approved,Issued
kubeadm-cert-system:kube-controller-manager-mr86h   79s     kubernetes-admin   Approved,Issued
kubeadm-cert-system:kube-scheduler-t8lnw            17s     kubernetes-admin   Approved,Issued
kubeadm-cert-ydzs-master-cqh4s                      52s     kubernetes-admin   Approved,Issued
kubeadm-cert-ydzs-master-lvbr5                      41s     kubernetes-admin   Approved,Issued

Check the validity period of the certificate after approval:

kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Nov 05, 2029 11:53 UTC   9y              no
apiserver                  Nov 05, 2029 11:54 UTC   9y              no
apiserver-etcd-client      Nov 05, 2029 11:53 UTC   9y              no
apiserver-kubelet-client   Nov 05, 2029 11:54 UTC   9y              no
controller-manager.conf    Nov 05, 2029 11:54 UTC   9y              no
etcd-healthcheck-client    Nov 05, 2029 11:53 UTC   9y              no
etcd-peer                  Nov 05, 2029 11:53 UTC   9y              no
etcd-server                Nov 05, 2029 11:54 UTC   9y              no
front-proxy-client         Nov 05, 2029 11:54 UTC   9y              no
scheduler.conf             Nov 05, 2029 11:53 UTC   9y              no

We can see that it has been extended for 10 years. This is because the validity period of the CA certificate is only 10 years.

But now we can't directly restart several components of the control panel. This is because the etcd corresponding to the cluster installed with kubeadm uses the /etc/kubernetes/pki/etcd/ca.crt certificate to perform the previous step by default. The certificate approved by the command kubectl certificate approve is issued with the default /etc/kubernetes/pki/ca.crt certificate, so we need to replace the CA certificate in etcd:

Copy the static Pod resource list first

cp -r /etc/kubernetes/manifests/ /etc/kubernetes/manifests.bak
vi /etc/kubernetes/manifests/etcd.yaml
......
spec:
  containers:
  - command:
    - etcd
    # 修改为 CA 文件
    - --peer-trusted-ca-file=/etc/kubernetes/pki/ca.crt
    - --trusted-ca-file=/etc/kubernetes/pki/ca.crt
......
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki  # 更改证书目录
      name: etcd-certs
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki  # 将 pki 目录挂载到 etcd 中去
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd 
      type: DirectoryOrCreate
    name: etcd-data
......

Since kube-apiserver needs to connect to the etcd cluster, the corresponding etcd ca file also needs to be revised:

vim /etc/kubernetes/manifests/kube-apiserver.yaml
......
spec:
  containers:
  - command:
    - kube-apiserver
    # 将etcd ca文件修改为默认的ca.crt文件
    - --etcd-cafile=/etc/kubernetes/pki/ca.crt
......

In addition, you need to replace the requestheader-client-ca-file file, the default is the /etc/kubernetes/pki/front-proxy-ca.crt file, and now you need to replace it with the default CA file, otherwise use the aggregation API, such as After installing the metrics-server and executing the kubectl top command, an error will be reported:

cp /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/front-proxy-ca.crt
cp /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/front-proxy-ca.key

Since it is a static Pod, the above components will automatically restart to take effect after the modification is completed. Since our current version of kubelet has automatic certificate rotation enabled by default, the kubelet certificate does not need to be managed anymore, so I will update the certificate to a validity period of 10. Be sure to back up the certificate directory before operation to prevent rollback of operation errors.

Guess you like

Origin blog.51cto.com/14034751/2552331