One of the problems and causes of the phenomenon
Kubernetes log error
When there Kubernetes cluster logs certificate has expired or is not yet valid an error message indicating that the certificate expired
Certificate expired reasons
- Server at the wrong time, leading to the certificate expired
- Indeed certificate expired
Certificate expires, many students will be very confused, I obviously certificate 签署10年is valid or 更久, just how 1年就过期了, following on to doubts.
Kubernetes cluster certificate
Certificate is divided into two clusters: one for the cluster Master、Etcdcertificate communication or the like. Second, for cluster Kubelet components certificate
Kubernetes cluster Kubelet assembly pit
When we build Kubernetes clusters, generally only declared for cluster Master、Etcdcredentials as the communication 10年 or 更久, but did not declare a cluster Kubelet 组件证书 , Kubelet 组件证书 the default is valid for 1年. After a cluster running year report will lead to certificate has expired or is not yet valid error, leading to 集群 Nodenot to 集群 Mastercommunicate properly.
Second, the solution
Add parameters
-
Modify
kubelet 组件配置, add the following particular parameters. 1 --feature-Gates = = RotateKubeletServerCertificate to true 2 --feature-Gates = = RotateKubeletClientCertificate to true . 3 # . 1 .8 version 1.8 supports comprising more auto-reload certificate replacement, the version of the service can only be restarted manually . 4 --rotate-Certificates
-
Modify
controller-manager 组件配置, add the following particular parameters -
1 # certificate is valid for 10 years 2 . 3 --experimental-Cluster-Signing-DURATION = 87600h0m0s . 4 . 5 --feature-Gates = = RotateKubeletServerCertificate to true
CSR is created automatically approve requests related ClusterRole
vim tls-instructs-csr.yaml && kubectl apply -f tls-instructs-csr.yaml
-
1 kind: ClusterRole 2 apiVersion: rbac.authorization.k8s.io/v1 3 metadata: 4 name: system:certificates.k8s.io:certificatesigningrequests:selfnodeserver 5 rules: 6 - apiGroups: ["certificates.k8s.io"] 7 resources: ["certificatesigningrequests/selfnodeserver"] 8 verbs: ["create"]
# Automatically approve kubelet-bootstrap user TLS bootstrapping first application for a certificate of CSR request
kubectl create clusterrolebinding nodeclient-auto-approve -csr --clusterrole = system: certificates.k8s.io: certificatesigningrequests: nodeclient --user = kubelet-bootstrap
# Automatic approval system: nodes group of users to update itself and apiserver kubelet communication of CSR certificate request
kubectl create clusterrolebinding node-client-auto -renew-crt --clusterrole = system: certificates.k8s.io: certificatesigningrequests: selfnodeclient --group = system : nodes
# Automatic approval system: nodes update the set of user ports kubelet 10250 api CSR certificate request
kubectl create clusterrolebinding node-server-auto -renew-crt --clusterrole = system: certificates.k8s.io: certificatesigningrequests: selfnodeserver --group = system: nodes
Restart kube-controller-manager and kubelet services
1 $ systemctl daemon reload- 2 $ systemctl restart the Controller-manager.service Kube- 3 # ssl into the configuration directory, delete kubelet certificate 4 $ RM -f kubelet-Client-current.pem kubelet-Client- * .pem kubelet.key kubelet.crt 5 6 # reboot to start, will be awarded ssl certificate is valid for 10 years after the start of the normal 7 $ systemctl restart kubelet 8 9 10 # into the ssl configuration directory, view the certificate is valid for 11 $ OpenSSL X509 - in kubelet-Client-Current. -noout -text PEM | grep " Not " 12 Not the Before: Nov 22 08:46:00 GMT 2019
13 Not the After: Nov 19 08:46:00 GMT 2029