kubernetes 之kubelet客户端证书过期问题处理 KubeClientCertificateExpiration apiserver (monitoring/k8s warning) Kubernetes API certificate is expiring in less than 7 days.

 

[FIRING:1] KubeClientCertificateExpiration apiserver (monitoring/k8s warning)
Kubernetes API certificate is expiring in less than 7 days. https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration
Alerts Firing:
Labels:
 - alertname = KubeClientCertificateExpiration
 - job = apiserver
 - prometheus = monitoring/k8s
 - severity = warning
Annotations:
 - message = Kubernetes API certificate is expiring in less than 7 days.
 - runbook_url = https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration
Source: http://prometheus-k8s-1:9090/graph?g0.expr=histogram_quantile%280.01%2C+sum+by%28job%2C+le%29+%28rate%28apiserver_client_certificate_expiration_seconds_bucket%7Bjob%3D%22apiserver%22%7D%5B5m%5D%29%29%29+%3C+604800&g0.tab=1


AlertmanagerUrl:
http://alertmanager-main-2:9093/#/alerts?receiver=wechat

经过排查，是kubelet使用的客户端证书过期， manager没有自动renew证书导致， 至于为什么没有自动轮换原因还没有查到。

以下是通过手动的方法更新证书，更新token， 删除 kubelet.kubeconfig ，重启kubelet服务，client.crt则会更新。 

openssl x509 -in kubelet-client.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            70:46:2e:69:03:d6:6f:01:4a:f0:98:a5:bf:94:b3:84:df:c5:64:dd
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=BeiJing, L=BeiJing, O=k8s, OU=Dongqiudi, CN=kubernetes
        Validity
            Not Before: Dec  5 02:15:00 2019 GMT
            Not After : Dec  4 02:15:00 2020 GMT
        Subject: O=system:nodes, CN=system:node:dqd-e-k8s-node07
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:2b:32:db:d9:20:16:b2:48:11:c8:00:42:db:de:
                    c1:7e:9c:18:4b:33:c8:22:79:08:af:fc:e3:71:cf:
                    6b:78:a6:e1:b4:fd:94:dd:07:81:9f:a0:63:d0:6d:
                    ed:13:32:de:25:0d:88:cd:af:3c:5a:03:c2:03:fd:
                    86:bb:2f:14:5d
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                0C:00:DF:DE:47:D8:6E:B1:2A:5A:95:69:B6:D5:76:B9:02:AD:CB:B0
            X509v3 Authority Key Identifier:
                keyid:53:DF:E0:0E:39:DD:DB:26:18:C7:75:AA:DF:63:A2:58:AE:C4:60:14


    Signature Algorithm: sha256WithRSAEncryption
         2d:8e:b7:9c:c6:69:d8:3c:e2:72:70:e5:6d:8b:80:58:64:54:
         a6:3c:3a:84:ca:f7:c6:18:a5:ae:65:9b:e6:24:3a:f5:34:13:
         77:93:39:1d:24:55:d4:96:4d:5f:2f:fe:79:26:85:4e:5e:0d:
         f0:5a:d8:3f:a9:17:42:bd:e9:15:e2:44:91:4c:a8:81:86:c2:
         ff:e4:3a:d8:42:75:d9:c6:24:11:45:8f:d7:98:c5:f8:01:d0:
         28:0e:b6:54:2c:88:a1:da:d6:5f:39:00:64:06:0f:c8:9d:11:
         16:bc:d3:3d:d4:5f:22:43:e7:3b:69:b1:2d:35:24:a2:4e:6b:
         b8:9b:d2:d9:01:84:3c:71:a9:23:44:ae:dc:c3:91:63:98:b4:
         a9:22:6b:aa:d6:ee:3d:6c:f2:55:67:f8:c1:e3:96:ca:aa:2c:
         44:50:4b:84:3e:85:a5:70:f4:e6:a6:bf:cf:88:b6:fe:9d:2d:
         a6:1f:1f:cb:54:3d:37:eb:b8:ce:4b:39:a7:44:90:e9:10:eb:
         56:8b:46:f2:d6:5d:f8:a3:f8:6f:af:48:a3:54:a1:4e:5d:d4:
         88:18:e6:2c:4e:e8:1f:e1:2e:cd:7e:8a:e5:1c:4d:26:e6:6c:
         ee:82:39:ad:e1:70:78:db:44:80:d7:9d:85:1b:a9:53:89:49:
         bd:78:b2:89