k8s cluster capacity management - kluster capacity

background

The three values ​​of the container platform: stability, efficiency, and cost are all inseparable from capacity management. Capacity management is a very important part of Kubernetes cluster management. It can ensure that the resources in the system are allocated and used reasonably, and avoid the problem of abnormal operation or low efficiency of the system caused by insufficient or wasted resources. Through capacity management, the utilization of system resources can be better controlled and optimized to ensure the stability and reliability of the Kubernetes cluster. Capacity management can also help administrators plan and predict the system better, avoiding the need for emergency capacity expansion due to insufficient resources, thereby improving system maintainability and reliability.

the problem we are facing

K8s cluster administrators are more or less troubled by the following problems:

• It is not clear what level the current cluster resource usage or remaining capacity is at;

• It is not clear how the current cluster resources are wasted;

• It is unclear how fragmented the current cluster resources are;

• It is not clear how to set the scheduling policy configuration value to improve resource utilization efficiency;

• ...

Resources are a typical quantifiable indicator. All the above problems can be quantified. What we lack is a useful tool.

Project Introduction

kluster-capacity ^[1] aims to solve the above problems by simulating the capabilities of real online schedulers. Currently, it supports three capabilities: capacity evaluation, scheduling simulation, and cluster compression.

capacity assessment

introduce

As new Pods are scheduled on nodes in the cluster, more and more resources are consumed. It is important to monitor the resources available in the cluster so that operators can increase the current resources in time before all resources are exhausted. Or, take different steps to increase available resources.

Cluster capacity includes the capacity of a single cluster node. Capacity covers CPU, memory, disk space, and other resources.

The overall remaining allocatable capacity is an estimate. The goal is to analyze the remaining allocatable resources and estimate the available capacity, that is, the number of Pod instances that can be scheduled in the cluster with a given resource requirement.

enhance

Here are some enhancements to the original cluster capacity:

• Supports using existing Pods directly from the cluster as Pod templates.

• Support batch simulation for different Pod templates.

run

# 直接使用指定的 pod 模板
$ ./kluster-capacity ce --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig> --pods-from-template <path to pod templates> 
# 使用集群中指定的 pod 作为模板
$ ./kluster-capacity ce --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig> --pods-from-cluster <namespace/name key of the pod>

For more operating parameters and functions, please execute the following commands:

$ ./kluster-capacity ce --help

demo

Assume the cluster is running with 4 nodes and 1 master node, each node has 2 CPUs and 4GB memory. And the resources required by each Pod are 150m CPU and 100Mi memory.

$ ./kluster-capacity ce --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig> --pods-from-template <path to pod templates> --verbose
Pod requirements:
 - cpu: 150m
 - memory: 100Mi

The cluster can schedule 52 instance(s) of the pod.
Termination reason: FailedScheduling: pod (small-pod-52) failed to fit in any node
fit failure on node (kube-node-1): Insufficient cpu
fit failure on node (kube-node-4): Insufficient cpu
fit failure on node (kube-node-2): Insufficient cpu
fit failure on node (kube-node-3): Insufficient cpu


Pod distribution among nodes:
 - kube-node-1: 13 instance(s)
 - kube-node-4: 13 instance(s)
 - kube-node-2: 13 instance(s)
 - kube-node-3: 13 instance(s)

As the number of pods running in the cluster increases, the number of pods that can be scheduled decreases when the analysis is run again.

$ ./kluster-capacity ce --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig> --pods-from-template <path to pod templates> --verbose
Pod requirements:
 - cpu: 150m
 - memory: 100Mi

The cluster can schedule 46 instance(s) of the pod.
Termination reason: FailedScheduling: pod (small-pod-46) failed to fit in any node
fit failure on node (kube-node-1): Insufficient cpu
fit failure on node (kube-node-4): Insufficient cpu
fit failure on node (kube-node-2): Insufficient cpu
fit failure on node (kube-node-3): Insufficient cpu


Pod distribution among nodes:
 - kube-node-1: 11 instance(s)
 - kube-node-4: 12 instance(s)
 - kube-node-2: 11 instance(s)
 - kube-node-3: 12 instance(s)

output format

ceThe command has a --output (-o)flag to format its output as json or yaml.

$ ./kluster-capacity ce --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig> --pods-from-template <path to pod templates> -o json|yaml

Scheduling simulation

introduce

The scheduler simulation takes all nodes, pods and other related resources in the current cluster as input, and simulates the process from no pods to creation and scheduling of all pods. This can be used to calculate cluster compression ratios to evaluate scheduling effectiveness or measure the quality of scheduling algorithms.

The results are more aggressive and idealistic than cluster compaction.

run

./kluster-capacity ss --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig>

For more operating parameters and functions, please execute the following commands:

$ ./kluster-capacity ss --help

It supports two termination conditions: AllSucceedand AllScheduled. The former means that the program ends after all pods are scheduled successfully, and the latter means that the program exits after all pods are scheduled at least once. The default value is AllSucceed. Exit conditions can be set using --exit-conditionthe flag .

demo

Assume the cluster is running with 4 nodes and 1 master node, each node has 2 CPUs and 4GB memory. There are 40 Pods with resource requirements of 100M CPU and 200Mi memory that need to be scheduled.

If the scheduler uses LeastAllocatedthe strategy , the scheduling result might look like this:

$ ./kluster-capacity ss --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig>
Termination reason: AllSucceed: 40 pod(s) have been scheduled successfully.

Pod distribution among nodes:
        - kube-node-1: 10 instance(s)
        - kube-node-2: 10 instance(s)
        - kube-node-3: 10 instance(s)
        - kube-node-4: 10 instance(s)

If you adjust the scheduler to use MostAllocatedthe strategy , the scheduling result may look like this:

$ ./kluster-capacity ss --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig>
Termination reason: AllSucceed: 40 pod(s) have been scheduled successfully.

Pod distribution among nodes:
        - kube-node-1: 20 instance(s)
        - kube-node-2: 20 instance(s)

The above scheduling results can be analyzed to evaluate the effectiveness of the scheduling strategy and the cluster capacity compression ratio. For example, the above results indicate that the cluster compression ratio is 2, which means that there is a 50% resource waste in the ideal case.

cluster compression

introduce

Cluster compaction takes the current state of the cluster, including all nodes, pods, and other related resources as input, and simulates the process of compacting the cluster by removing nodes. It can be used to calculate the compression ratio of the cluster, which is a measure of resource utilization efficiency.

Compared with simulated scheduling, the results of cluster compression are usually more visible and more operable.

run

./kluster-capacity cc --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig> --verbose

For more operating parameters and functions, please execute the following commands:

$ ./kluster-capacity cc --help

demo

Assume the cluster is running with 4 nodes and 1 master node, each node has 2 CPUs and 4GB memory. Run 40 Pods with resource requirements of 100M CPU and 200Mi memory.

./kluster-capacity cc --kubeconfig <path to kubeconfig> --schedulerconfig= <path to schedulerconfig> --verbose
2 node(s) in the cluster can be scaled down.

Termination reason: FailedSelectNode: could not find a node that satisfies the condition, 1 master node(s); 2 node(s) can't be scale down because of insufficient resource in other nodes;

nodes selected to be scaled down:
        - kube-node-1
        - kube-node-3

The above results show that, given the resource requirements of 40 pods, under the condition that all pods can be scheduled, the cluster can remove 2 nodes, and the compression ratio is 2, which means 50% resource waste.

evolution

Currently, the above three capabilities are supported, and other capabilities related to capacity and resource management will continue to be improved in the future, such as

• snapshot-based simulation

• Resource Fragmentation Analysis

Help us carry out relevant simulation operations based on the status of a certain moment in the history of the cluster, and analyze resource fragmentation, etc. Welcome to experience and put forward your valuable opinions, thank you!

References

[1]

kluster-capacity: https://github.com/k-cloud-labs/kluster-capacity