When using Kubernetes, out-of-memory (OOM) errors and CPU throttling are major pain points for resource handling in cloud applications. Why is that? CPU and memory requirements in cloud applications are becoming increasingly important because they are directly related to cloud costs.
Limits and requests allow you to configure how pods should allocate memory and CPU resources to prevent resource starvation and adjust cloud costs.
If the node does not have enough resources, the Pod may be driven out through preemption or node pressure. When a process runs out of memory (OOM), it is terminated because it does not have the resources it needs;
If the CPU consumption is higher than the actual limit, the process will start to throttle, but how to proactively monitor how close a Kubernetes Pod is to reaching OOM and CPU throttling?
2. Kubernetes OOM
Each container in a Pod requires memory to run.
Kubernetes limits are set per container in the Pod definition or Deployment definition.
All modern Unix systems have a way to kill processes in case they need to reclaim memory, which will be marked as Error 137 or OOMKilled.
This exit code 137 indicates that the process is using more memory than allowed and must terminate. This is a feature that exists in Linux where the kernel oom_score sets a value for processes running in the system. Additionally, it allows setting a value called oom_score_adj, which is used by Kubernetes to allow quality of service. It also has an OOM Killer feature that will audit processes and kill those that are using more memory than they should be allowed to use.
Note that in Kubernetes, a process can reach any of the following limits:
Kubernetes Limit set on the container;
Kubernetes ResourceQuota set on the namespace;
The actual memory size of the node.
3. Memory overuse
Limits can be higher than requests, so the sum of all limits can be higher than the node capacity, which is called overcommitment, which is common. In fact, if all containers use more memory than requested, it can run out of memory in the node, which usually results in some pods being killed to free up some memory.
① Monitor Kubernetes OOM
When using the node exporter in Prometheus, there is a metric called node_vmstat_oom_kill, it is important to track when an OOM kill occurs, but you may want to be aware of such events in advance before they occur.
Instead, you can check how close a process is to Kubernetes limits:
(sum by(namespace,pod,container)(rate(container_cpu_usage_seconds_total{
container!=""}[5m]))/ sum by(namespace,pod,container)(kube_pod_container_resource_limits{
resource="cpu"}))>0.8
② Kubernetes CPU throttling
CPU throttling is the behavior of slowing down a process when it is about to reach certain resource limits.
Similar to the memory situation, these limits may be:
Kubernetes Limit set on the container.
Kubernetes ResourceQuota set on the namespace.
The actual CPU size of the node.
Think of the following analogy, we have a highway with some traffic, where:
CPU is the way.
Vehicles represent processes, and each vehicle has a different size.
Multiple channels represent multiple cores.
A request would be for a dedicated road, such as a bike path. Throttling here manifests as traffic jams, and eventually, all processes will run, but everything will slow down.
③CPU process in Kubernetes
CPU is handled using shares in Kubernetes, where each CPU core is divided into 1024 shares and then distributed among all running processes using the cgroups (control group) feature of the Linux kernel.
All current processes can be processed, no action is required. If a process uses more than 100% of the CPU, then shares are in place. Like any Linux Kernel, Kubernetes uses the CFS (Completely Fair Scheduler) mechanism, so processes with more shares will get more CPU time.
Unlike memory, Kubernetes does not kill pods due to throttling:
CPU statistics can be viewed in /sys/fs/cgroup/cpu/cpu.stat.
4. CPU overuse
Setting limits or requests is important when you want to limit the resource consumption of a process. However, be careful not to set the total number of requests larger than the actual CPU size, as this means each container should have a certain number of CPUs.
Monitoring Kubernetes CPU throttling lets you check how close your processes are to Kubernetes limits:
(sum by(namespace,pod,container)(rate(container_cpu_usage_seconds_total
{
container!=""}[5m]))/ sum by(namespace,pod,container)(kube_pod_container_resource_limits{
resource="cpu"}))>0.8
If you want to track the amount of throttling that occurs in your cluster, cadvisor provides container_cpu_cfs_throttled_periods_total and container_cpu_cfs_periods_total, which make it easy to calculate the throttling percentage of all CPU cycles.
5. Best Practices
① Pay attention to limits and requests
Throttling is a way to set a maximum resource limit in a node, but these limits need to be treated with caution as you may end up with a situation where a process is throttled or terminated.
② Be prepared to be evicted
By setting requests very low, you might think this would grant your process the least amount of CPU or memory. But the kubelet will evict those pods with higher usage than requested first, thus marking them as the first to be killed.
If you need to protect a specific Pod from preemption (when kube-scheduler needs to allocate a new Pod), assign priority to the most important processes.
③ Throttling is the silent enemy
By setting unrealistic limits or overusing it, you may not realize that your process is being throttled, and performance suffers. Proactively monitor CPU usage and understand actual limits within containers and namespaces.
