Docker-Cgroup resource configuration

1. The role of Dcker resource control

Generally speaking, the container has no resource limit by default, and will use the host's resources to the greatest extent. On the Linux host, if the kernel detects that there is not enough memory to perform important system functions, it will throw an OOME (Out Of Memory Exception). Once the OOME occurs, Linux will start to check and kill the process to release the memory. Any process may be killed, including docker daemonand 其他重要的应用程序. If the wrong process is killed, this can reduce the use of the entire system.
In this case, Docker provides a resource control method that can limit the container's CPU, memory, and IO.

Two, Cgroup resource configuration method

Docker controls the resource quotas used by containers through Cgroups, including the three major aspects of CPU, memory, and disk, which basically covers common resource quotas and usage control.
Cgroup is the abbreviation of Control Groups. It is a mechanism provided by the Linux kernel to limit, record, and isolate the physical resources (such as CPU, memory, disk IO, etc.) used by the process group. It is used by
many projects such as LXC and docker. Realize process resource control. Cgroup itself is a basic structure that provides functions and interfaces for grouping processes. Specific resource management such as I/O or memory allocation control is realized through this function.
These specific resource management functions are called Cgroup subsystems, which are implemented by the following major subsystems:

• blkio: Set to limit the input and output control of each block device. For example: disk, CD, usb and so on.
• CPU: Use the scheduler to provide CPU access for cgroup tasks.
• cpuacct: Generate CPU resource reports for cgroup tasks.
• cpuset: If it is a multi-core CPU, this subsystem will allocate separate CPU and memory for cgroup tasks.
• devices: Allow or deny access to devices by cgroup tasks.
• freezer: Pause and resume cgroup tasks.
• memory: Set the memory limit of each cgroup and generate memory resource reports.
• net_cls: Mark each network packet for convenient use by cgroups. ns: Namespace subsystem.
• perf_event: The ability to monitor and track each group is added, and all threads belonging to a specific group and threads running on a specific CPU can be monitored.

Let's start using the stress test tool to test CPU and memory usage

2.1 Use stress tool to test CPU and memory

①Use Dockerfile to create a stress tool image based on Centos

mkdir stress
cd stress/
vim Dockerfile

FROM centos:7
MAINTAINER wt
RUN yum install -y wget
RUN wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
RUN yum install -y stress

docker build -t centos:stress .

②Use the following command to create a container. The -cpu-shares parameter value in the command does not guarantee that 1 cpu or how many GHz CPU resources can be obtained, it is only a flexible weighted value

docker run -itd --cpu-shares 100 centos:stress
docker run -tid --name cpu512 --cpu-shares 512 centos:stress stress -c 10   #容器产生10个子函数进程
docker run -tid --name cpu1024 --cpu-shares 1024 centos:stress stress -c 10 #再开启一个容器做比较
docker exec -it 容器ID bash    #进入容器使用top查看cpu使用情况,对比两个容器的%CPU，比例是1:2

By default, the CPU share of each Docker container is 1024. The share of a single container is meaningless. Only when multiple containers are running at the same time, the effect of the container's CPU weighting can be reflected.
For example, the CPU shares of two containers A and B are 1000 and 500 respectively. When the CPU allocates time slices, container A has twice the chance of obtaining CPU time slices than container B. However, the result of the allocation depends on the running status of the host and other containers at the time. In fact, there is no guarantee that container A will be able to obtain CPU time slices. For example, the process of container A is always idle, then container B can obtain more CPU time slices than container A.
In extreme cases, for example, there is only one container running on the host, even if its CPU share is only 50, it can monopolize the CPU resources of the entire host.
Cgroups will only take effect when the resources allocated by the container are in short supply, that is, when the resources used by the container need to be restricted. Therefore, it is impossible to determine how many CPU resources are allocated to a container based solely on the CPU share of a container. The result of resource allocation depends on the CPU allocation of other containers running at the same time and the running conditions of the processes in the container. You can set the priority of the container to use the CPU through the cpu share, such as starting two containers and running to view the CPU usage percentage.

2.2 CPU cycle limit

Docker provides two parameters –cpu-period and –cpu-quota to control the CPU clock cycles that the container can allocate.

• --Cpu-period is used to specify how long the container should redistribute the CPU usage.
• -Cpu-quota is used to specify the maximum amount of time that can be used to run the container in this cycle.
• When –cpu-quota is a multiple of –cpu-period, use multiple CPUs

Unlike --cpu-shares, this configuration specifies an absolute value, and the container's use of CPU resources will never exceed the configured value.

The unit of cpu-period and cpu-quota is microsecond (μs). The minimum value of cpu-period is 1000 microseconds, the maximum value is 1 second (10^6 μs), and the default value is 0.1 second (100000 μs).

The default value of cpu-quota is -1, which means no control is performed. The cpu-period and cpu-quota parameters are generally used in combination.

For example: the container process needs to use a single CPU for 0.2 seconds every 1 second. You can set the cpu-period to 1000000 (that is, 1 second) and the cpu-quota to 200000 (0.2 seconds). Of course, in the case of multi-core, if the container process is allowed to completely occupy two CPUs, you can set the cpu-period to 100000 (that is, 0.1 seconds) and the cpu-quota to 200000 (0.2 seconds).

docker run -itd --cpu-period 100000 --cpu-quota 200000 centos:stress
docker ps
docker exec -it 容器ID bash  

cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
200000

2.3 CPU Core control

For servers with multi-core CPUs, Docker can also control which CPU cores the container uses, that is, use the -cpuset-cpus parameter.
This is especially useful for servers with multiple CPUs, and can be configured with optimal performance for containers that require high-performance computing.

docker run -tid --name cpu1 --cpuset-cpus 0-1 centos:stress      #限制容器cpu1只能使用0和1两个内核

Executing the above command requires the host to be dual-core, which means that the created container can only use the 0 and 1 cores. The final cgroup CPU core configuration is as follows:

docker exec -it 容器ID bash

cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-1          0-1表示该容器能使用的内核为0、1两个

The following command can view the binding relationship between the container process and the CPU core

docker exec 容器id taskset -c -p 1

2.4 Mixed use of CPU quota control parameters

Use the cpuset-cpus parameter to specify that container A uses CPU core 0, and container B only uses CPU core 1.
On the host, only these two containers use the corresponding CPU cores, and they each occupy all the core resources, and cpu-shares have no obvious effect.
The cpuset-cpus and cpuset-mems parameters are only valid on servers with multi-core and multi-memory nodes, and must match the actual physical configuration, otherwise the purpose of resource control cannot be achieved.
When the system has multiple CPU cores, you need to set the container CPU core through the cpuset-cpus parameter to facilitate the test.

##宿主系统修改为4核心CPU
docker run -tid --name cpu3 --cpuset-cpus 1 --cpu-shares 512 centos:stress stress -c 1
docker run -tid --name cpu4 --cpuset-cpus 3 --cpu-shares 1024 centos:stress stress -c 1

top   #按1查看每个核心的占用

docker exec -it 容器ID bash

Summary: The above centos:stress image has the stress tool installed to test the CPU and memory load. By executing the stress -c 1 command on the two containers respectively,
a random load will be given to the system and 1 process will be generated. This process repeatedly calculates the square root of the random number generated by rand() until the resource is exhausted.
Observing the CPU usage rate on the host machine, the usage rate of the third core is close to 100%, and the CPU usage rate of a batch of processes obviously has a 2:1 usage ratio comparison.

2.5 Memory limit

Similar to the operating system, the memory that the container can use includes two parts: physical memory and Swap.
Docker uses the following two sets of parameters to control the usage of container memory.

-m or –memory: Set the usage limit of (physical) memory, such as 100M, 1024M.
–memory-swap: Set the usage limit of memory + swap.
Executing the following command allows the container to use up to 200M of memory and 300M of swap

docker run -it -m 200M --memory-swap=300M progrium/stress --vm 1 --vm-bytes 280M

--vm 1：启动一个内存工作线程
--vm-bytes 280M：每个线程分配280M内存

By default, the container can use all the free memory on the host.
Similar to the cgroups configuration of the CPU, Docker will automatically create the corresponding cgroup configuration file for the container in the directory /sys/fs/cgroup/memory/docker/<container complete id>

If the memory allocated by the thread is greater than or equal to the set 300M, the stress thread reports an error and the container exits.

docker run -it -m 200M --memory-swap=300M progrium/stress --vm 1 --vm-bytes 300M

2.6 Limitations of Block IO

By default, all containers can read and write to the disk equally. You can change the priority of container block 10 by setting the -blkio-weight parameter.
–Blkio-weight is similar to –cpu-shares. It sets the relative weight value and the default is 500.
In the following example, the bandwidth of container A to read and write to the disk is twice that of container B.

docker run -it --name container_A --blkio-weight 600 centos:stress
cat /sys/fs/cgroup/blkio/blkio.weight
600

docker run -it --name container_B --blkio-weight 300 centos:stress
cat /sys/fs/cgroup/blkio/blkio.weight
300

2.7 bps and iops limits

bps is byte per second, the amount of data read and written per second.
iops is io per second, the number of IOs per second.

The bps and iops of the container can be controlled by the following parameters:

--device-read-bps，限制读某个设备的 bps。
--device-write-bps，限制写某个设备的 bps。
--device-read-iops，限制读某个设备的 iops。
--device-write-iops，限制写某个设备的 iops。

The following example is to limit the rate at which the container writes /dev/sda to 5 MB/s.

As a comparative test, if the speed is not limited, the results are as follows:

Use the dd command to test the speed of writing to the disk in the container. Because the file system of the container is on host /dev/sda
, writing files in the container is equivalent to writing to host /dev/sda. In addition, oflag=direct specifies direct IO to write files,
so that --device-write-bps can take effect.