kubernetes Resource Management

Foreword

In kubernetes environment, whether the cluster is bigger, the corresponding cluster resources (cpu, memory, storage) there is always a limit. And by default, we start the pod, pod container and running, the corresponding resources are available without restriction. In theory each pod, or run in the pod container, unlimited use of resources until the resources on the node where depleted, resulting node crashes, thereby affecting the stability of the cluster. To prevent this situation occurs, kubernetes provides us with the appropriate mechanism to limit the resources of a single pod can be used, the main object has the following two

• LimitRanger
• ResourceQuota

LimitRanger

1. LimitRanger is a strategy used to limit a single pod can consume resources at a particular namespace, mainly to prevent the situation where a single pod resources to seize all of the namespace finished, configure the following aspects

• Limited computational resources (CPU, Memory) single pod or container may be consumed within a certain range (maximum and minimum)
• PersistentVolumeClaim define a single storage may apply to some of the range
• Defining demand value (minimum) and the limit (maximum) to a particular resource ratio
• Automatically the next in the same namespace, no additional resources were limited defined pod

1. Use the following procedure

   1. Creating a cluster administrator to create LimitRange in a particular namespace
   2. Users create pod, container, PersistentVolumeClaims objects in this namespace
   3. LimitRanger access controller checks the information of the object to be created, if not the allocation of resources, will use the default value to assemble these objects
   4. If there is allocation of resources, configurations and checks to see if these specifications defined LimitRanger conflict, if conflict refuse to create the appropriate object
   5. If you define ResourceQuota under namespace, LimitRanger not define default resource limits, create the object nor the resources to be configured, create rejected

Quality of Service (QoS)

pod or container created to specify whether the resource request and limit values ​​based, kubernettes the pod is divided into three levels

• When all containers in the pod Guaranteed for all resources are defined and Limits Requests, and Requests Limits value and all have equal values ​​for all containers
• All containers BestEffort pod are not defined in the allocation of resources
• Burstable when a pod is not Guaranteed nor the BestEffort, the pod of QoS is Burstable. Pod for example, a portion of the container defines the Request Limit value or values, or are defined, but the values ​​are not equal

kubernets in the event of resource contention, it will give priority to ensuring the level of pod Guaranteed uptime, followed Burstable, finally BestEffort

ResourceQuota

ResourceQuota (resource quota) limited the total amount of resources available to all objects in a particular namespace, the other can create a limited number of various objects

Workflow is as follows

1. Because the entire namespace ResourceQuota limited, so that different groups can only work interfere with each other at different namespace
2. Cluster Administrator to create the appropriate ResourceQuota for each namespace
3. Users create objects pod, containers, etc. in their own namespace
4. If you want to create an object to apply the resources and ResourceQuota there is a conflict, the system will refuse to create

Actual demand

Resource information corresponding to the time limit by LimitRanger create or pod, kubernetes when the container itself because some defects result has been to seize the pod system resources, and automatically help us identify problems and have stopped these problems pod, if the pod is controlled controller, will help us to reschedule this pod, to make the system available to maintain.

For example, we have the following code block

@RequestMapping("/oom")
public String oom() {
    log.info("request to oom");
    PersonRepo pp = new PersonRepo();
    pp.autoCreatePerson();
    
    return "OK";
}

public static class PersonRepo {

    private List<Person> repo = new ArrayList<>();

    public void autoCreatePerson() {
        for (long l = 0; ; l++) {
            repo.add(new Person(l));
        }
    }
}

When invoked oom method, since the program code bug, causes the application will continue to create Person objects in memory, if unchecked, this program will continue to increase memory until the node points to kubernets all available memory is consumed do.

If you start adding resource information

...
image: 192.168.0.107/k8s/resource-quotas-oom:0.0.4
    ports:
    - containerPort: 8080
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "582Mi"
        cpu: "500m"
...

When this container is consumed memory reach 582Mi, k8s will directly kill off the vessel, prompt information OOMKilled, and the pod restart (the only one in this container in the pod).

While this ensures the availability of the system, but from the information we have no way to analyze what in the end is a problem with the code, under normal circumstances, we start the java application, will add the following command to start jvm parameters

 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/src/oomdump

Tell jvm, when the system oom, give us generate a dump file under a specific path, then we analyze the specific error code appears according to this dump file. If we want to be generated dump file before k8s kill in a container, we also need to set the size of available memory jvm

-server -Xms512m -Xmx512m

And if the jvm maximum memory (Xmx) value and k8s resource.limits.memory the same value or a small, run-time before the actual jvm appear oom, will first trigger OOMKill k8s, or can not generate the dump, so in order to generate a dump file, the value of resource.limits.memory than Xmx value larger (50M ~ 100M).

But after this setting, because this value has not yet reached resource.limits.memory, although the application appeared oom, but the vessel will not be stopped k8s cluster, pod and will not restart, resulting in slower response system (memory containers has been occupied, access to other requests will be slow, and the emergence of oom), this is not the result we want, so the jvm then an additional argument, let the application directly after oom stop

-XX:+ExitOnOutOfMemoryError 

In this way, the oom appears, will apply to Mr. dump, after termination, system log information

2020-02-24 10:25:15.832  INFO 6 --- [nio-8080-exec-2] c.falcon.resource.quotas.oom.RunOom      : request to oom
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /usr/src/oomdump ...
Heap dump file created [625772870 bytes in 25.588 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space

You can see from the log, Mr. system into a dump, then stopped himself off

pod change information

root@master:~# kubectl get pod -w
NAME                              READY   STATUS    RESTARTS   AGE
jenkins-68d8b54c45-7pdhs          1/1     Running   0          2d5h
oom-deployment-64664f9454-kp65n   1/1     Running   6          26h
oom-deployment-64664f9454-kp65n   0/1     Error     6          26h
oom-deployment-64664f9454-kp65n   1/1     Running   7          26h

You can see the pod appear error, then automatically restored to the Running state, k8s automatically stopped pod in a container and then start up, look container / usr / src / oomdump directory

root@master:~# kubectl exec -it oom-deployment-64664f9454-kp65n ls /usr/src
resource-quotas-oom-0.0.1-SNAPSHOT.jar
start.sh

How did not we generate dump files, because the container is restarted, the working directory has become the working directory of the new container, the container does not contain the file stopped.

View full pod information, the following fragment

root@master:~# kubectl get pod oom-deployment-64664f9454-kp65n -o yaml

...
 containerStatuses:
  - containerID: docker://70daa47c0f45c9d014e0f70b80a41ca6bc9fd219ca957cc035ee2a049d46166b
    image: 192.168.0.107/k8s/resource-quotas-oom:0.0.4
    imageID: docker-pullable://192.168.0.107/k8s/resource-quotas-oom@sha256:3ab65f8b5d17182abb0ca0ccc164e5f40bb1d0fcf14006c06af795d0419daa58
    lastState:
      terminated:
        containerID: docker://95776b99ec7a27c03327e45f9e1fc1d3f058c9b814400eabafa9606534c6bc2a
        exitCode: 3
        
...

A container has been stopped, the corresponding container ID: 95776b99ec7a27c03327e45f9e1fc1d3f058c9b814400eabafa9606534c6bc2a

To the node where the pod, to acquire a corresponding command by the dump

root@slave:/opt/k8s/work# docker ps -a |grep 95776b99ec7a
95776b99ec7a        ec0a1d08eb22                  "/bin/sh -c ./start.…"   3 hours ago         Exited (3) 13 minutes ago                       k8s_oom_oom-deployment-64664f9454-kp65n_default_9a47dc18-14bf-4d77-9ed4-5ac2c11d2bf7_6
root@slave:/opt/k8s/work# docker cp 95776b99ec7a:/usr/src/oomdump .
root@slave:/opt/k8s/work# ls -alh oomdump
-rw------- 1 root root 597M 2月  24 18:26 oomdump 

• Wherein 95776b99ec7a kubectl is acquired by the ID of the container top 12 (the default display Docker ID 12)

Found during execution, if the system repeatedly appear oom, k8s will help us to retain a container recently stopped, and go down before turning off the container will be automatically recovered straight away. This is because kubelet have found a garbage collection strategy, this parameter can be set via kubelet configuration parameters maximum-dead-containers-per- container, the default value is 1, it will only help us to retain a For details, please refer to Configuring kubelet Garbage Collection

doubt

While jvm generated by setting the parameters to achieve the program oom dump, after the restart, but after this set, in fact, resources provided no information k8s play its due role. I do not know whether it is the wrong method or k8s there are other mechanisms, then perform what action (preStop hook) before being OOMKill, such as the in-depth study to see if a more elegant implementation