foreword
Container technology and orchestration management are the two core parts of the cloud-native ecosystem—the former is responsible for execution, and the latter is responsible for control and management, which together constitute the organism of cloud-native technology. We take Kubernetes as an example to analyze the possible risks faced by the container orchestration platform
Risks faced by container orchestration platforms
As the most popular cloud-native management and orchestration system, Kubernetes has powerful functions, but it also has high program complexity. We know that there is a certain degree of relationship between risk and program complexity, but it is not easy to analyze the risk of a complex system
The following is the architecture diagram of Kubernetes:
A common Kubernetes cluster consists of a Master node and three Worker nodes, and Pods communicate with each other through the CNI plug-in Flannel. Kubernetes' own system Pods (Pods in the kube-system namespace) mainly run on the Master node. In addition, each Worker node also has a Flannel Pod and a kube-proxy Pod. All business Pods are distributed in three on a Worker node. In addition, there is a Kubelet service on each node, responsible for managing containers
In fact, the orchestration system and the container are not completely independent. For example, we need to create a Pod in the form of a YAML declarative file in a Kubernetes cluster, and a Pod is only a logical concept that actually consists of one or more containers. Composition, the configuration of the container must be delivered in the form of Pod configuration
We mainly discuss the risks faced by the container orchestration system from the four aspects of Kubernetes interface, network, access control mechanism and software vulnerabilities
The risks of container infrastructure
In the Kubernetes environment, the risks faced by container infrastructure mainly include the following points:
- Insecure third-party components, malicious images that spread wildly, and sensitive information that is extremely easy to leak, Kubernetes provides two resources, ConfigMaps and Secrets, to store general configuration and sensitive information separately
- Insecure container applications, unlimited resource sharing, and insecure configuration and mounting
- Risks Existing in Cluster Networks
- The risk of access control mechanisms, that is, the risk of the container management program interface
- Container Software Vulnerabilities
Risks in Kubernetes Component Interfaces
There are many component vulnerabilities in Kubernetes, and most components provide services in the form of APIs based on HTTP or HTTPS. Among them, the services and their ports that we have more daily contact with are shown in the following table:
| components | default port | illustrate |
|---|---|---|
| API Server | 6443 | Secure port based on HTTPS |
| API Server | 8080 | Insecure HTTP port, not recommended to enable |
| Kubelet | 10248 | HTTP port for checking Kubelet health status |
| Kubelet | 10250 | HTTPS port for serving API Server |
| Dashboard | 8001 | Port providing HTTP service |
| etcd | 2379 | Port for communication between client and server |
| etcd | 2380 | Ports for communication between different server instances |
Next, we conduct a risk analysis of these services:
API Server
By default, API Server provides services on ports 8080 and 6443, of which port 8080 provides HTTP services without TLS encryption, and all requests reaching this port will bypass all authentication and authorization modules (but will still be approved input control module). This port is reserved mainly for the convenience of testing and the initial startup of the cluster
However, if the production environment opens port 8080, it is very dangerous even if the local loopback address (localhost) is bound. If this port is exposed to the Internet, any network-reachable attacker can directly interact with the API Server through this port, and then control the entire cluster
In contrast, port 6443 provides an HTTPS service encrypted with TLS, and incoming requests must pass authentication and authorization mechanisms to be successfully processed. When the authentication and authorization mechanism is configured correctly, the service and security provided by port 6443 will be higher
Dashboard
By default, we need to execute Kubectl proxy before we can access Dashboard through local port 8001. However, if the Dashboard port is directly mapped to the host node, or an additional address parameter is specified when executing Kubectl proxy, then all users who can access the host, including attackers, will be able to directly access the Dashboard
kubetcl proxy --address 0.0.0.0 --accept-hosts='^*$'
In addition, Dashboard requires login authentication by default. However, if the user adds the --enable-skip-login option to the startup parameters of Dashboard, the attacker can directly click the "Skip" button on the Dashboard interface and enter directly without logging in. Dashboard
Kubelet
We know that API Server is the nerve center of the entire Kubernetes, and provides a large number of application interfaces in the form of RESTful API. In fact, Kubelet also provides a RESTful API service for the API Server to call to obtain or change the resource status on a Kubernetes node
By default, Kubelet opens the API service on port 10250, and also listens on port 10248 for other components to check the running status of Kubelet:
curl http://IP:10248/healthz
The 10248 port service is relatively simple and there is no special risk, but the 10250 port may not. By default, the API Server needs to use a client certificate when accessing Kubelet's API, which is relatively safe, but there are still exceptions:
1) The attacker stole the client certificate of the API Server to access Kubelet in some way
2) Set the --anonymous-auth parameter of Kubelet to true, and the authorization mode to AlwaysAllow
If the above situation occurs, the network attacker can directly interact with Kubelet, so as to control the node where it is located
etcd
Various resources and their status in the Kubernetes cluster are stored in etcd. If there is a way to read the data in etcd, it is possible to obtain high authority and control the cluster. Currently, after etcd is started, it listens to two ports, 2379 and 2380. The former is used for client connections, and the latter is used for peer communication between multiple etcd instances. In multi-node clusters, in order to achieve high availability, etcd often monitors on node IP (non-local IP) to achieve intercommunication between multiple nodes, which may allow external attackers to access etcd
By default, the services provided by the two ports require corresponding certificates to access (anonymous access is prohibited), which guarantees the security of etcd. If an attacker steals the certificate, or the user sets etcd to allow anonymous access, the attacker may directly access etcd and steal data
Risks Existing in Cluster Networks
In order to achieve mutual communication between cluster Pods, after installing and deploying Kubernetes, we often need to install an additional network plug-in. Common network plug-ins include Flannel, Calico, and Cilium, etc.
In the absence of other network isolation policies and Pod security policies by default, because Pods can communicate with each other, and the root user in the Pod has CAP_NET_RAW permissions, network detection, sniffing, denial of service, and man-in-the-middle attacks may occur inside the cluster and other cyber attacks
Risks of Access Control Mechanisms
The access control mechanism in Kubernetes is mainly composed of three parts: authentication mechanism, authorization mechanism and admission mechanism. Each part usually has one or more specific implementation mechanisms to choose from. If the access control is too loose, high-privileged accounts may be abused, thereby posing a threat to Kubernetes itself and running containers. In addition, if unauthorized access to Kubernetes is allowed, an attacker may use this to gain direct access to the cluster administrator. permissions
Incurable Software Vulnerabilities
As a complex system, Kubernetes is naturally exposed to many security vulnerabilities, which naturally also belong to the risks faced by Kubernetes
Reference: "Cloud Native Security - Attack and Defense Practice and System Construction"