Use Cgroup for resource management in Docker

1. Cgroup resource configuration method

1. Docker controls the resource quotas used by containers through Cgroup, including the three major aspects of CPU, memory, and disk, which basically covers common resource quotas and usage control.

2. Cgroup is the abbreviation of Control Groups. It is a mechanism provided by the Linux kernel to limit, record, and isolate the physical resources (such as CPU, memory, disk I0, etc.) used by process groups. It is used by many projects such as LXC and docker. Used to implement process resource control. Cgroup itself is the basic structure that provides functions and interfaces for grouping processes. Specific resource management such as I/O or memory allocation control is realized through this function.
These specific resource management functions are called Cgroup subsystems. The following major subsystems are implemented:

• blkio: Set to limit the input and output control of each block device. For example: disk, CD, usb and so on.
• CPU: Use the scheduler to provide CPU access for cgroup tasks.
• cpuacct: Generate CPU resource reports for cgroup tasks.
• cpuset: If it is a multi-core CPU, this subsystem will allocate separate CPU and memory for cgroup tasks.
• devices: Allow or deny access to devices by cgroup tasks.
• freezer: Pause and resume cgroup tasks.
• memory: Set the memory limit of each cgroup and generate memory resource reports.
• net_cls: Mark each network packet for convenient use by cgroups.
• ns: Namespace subsystem.
• perf_event: Added the ability to monitor and track each group, which can monitor all threads belonging to a specific group and threads running on a specific CPU.

Let's start using the stress test tool to test the CPU and memory usage.

2. Use the stress tool to test CPU and memory

First use Dockerfile to create a stress tool image based on Centos

[root@localhost ~]# mkdir /opt/stress
[root@localhost ~]# vim /opt/stress/Dockerfile
FROM centos:7
MAINTAINER panrj
RUN yum install -y wget
RUN wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
RUN yum install -y stress
[root@localhost ~]# cd /opt/stress/
[root@localhost stress]# docker build -t centos:stress .
[root@localhost stress]# docker run -itd --cpu-shares 100 centos:stress

Description: By default, the CPU share of each Docker container is 1024. The share of a single container is meaningless. Only when multiple containers are running at the same time, the effect of the container's CPU weighting can be reflected.
For example, the CPU shares of two containers A and B are 1000 and 500 respectively. When the CPU allocates time slices, container A has twice the chance of obtaining CPU time slices than container B.
However, the result of the allocation depends on the running status of the host and other containers at the time. In fact, there is no guarantee that the container A will get the CPU time slice. For example, the process of container A is always idle,
then container B can obtain more CPU time slices than container A. In extreme cases, for example, there is only one container running on the host, even if its CPU share is only 50, it can monopolize the CPU resources of the entire host.

Cgroups will only take effect when the resources allocated by the container are in short supply, that is, when the resources used by the container need to be restricted. Therefore, it is impossible to determine how many CPU resources are allocated to a container based solely on the CPU share of a container. The
result of resource allocation depends on the CPU allocation of other containers running at the same time and the running status of the processes in the container.
You can set the priority of the container to use the CPU through the cpu share, such as starting two containers and running to view the CPU usage percentage.

[root@localhost stress]# docker run -tid --name cpu512 --cpu-shares 512 centos:stress stress -c 10 //容器产生10个子函数进程
[root@localhost stress]# docker exec -it f4953c0d7e76 bash //进入容器使用top查看cpu使用情况

Turn it on at this time-a container for comparison

[root@localhost stress]# docker run -tid --name cpu1024 --cpu-shares 1024 centos:stress stress -c 10

[root@localhost stress]# docker exec -it 5590c57d27b0 bash //进容器使用top对比两个容器的%CPU,比例是1:2

Compare and view results

Three, CPU cycle limit

Docker provides two parameters --cpu-period and --cpu-quota to control the CPU clock cycles that the container can allocate.

--cpu-periodIt is used to specify how long the container's CPU usage should be re-allocated.

--cpu-quotaIt is used to specify how much time can be used to run this container in this cycle.

Unlike -cpu-shares, this configuration specifies an absolute value, and the container's use of CPU resources will never exceed the configured value.
The unit of cpu-period and cpu-quota is microsecond (us). The minimum value of cpu-period is 1000 microseconds, the maximum value is 1 second (10^6μs), and the default value is 0.1 second (100000 us).

The default value of cpu-quota is -1, which means no control. The cpu-period and cpu-quota parameters are generally used in combination.

E.g: The container process needs to use a single CPU for 0.2 seconds every 1 second. You can set the cpu-period to 1000000 (that is, 1 second) and the cpu-quota to 200000 (0.2 seconds).

Of course, in the case of multi-core, if the container process is allowed to completely occupy two CPUs, you can set the cpu-period to 100000 (that is, 0.1 second) and the cpu-quota to 200000 (0.2 seconds).

[root@localhost stress]# docker run -tid --cpu-period 100000 --cpu-quota 200000 centos:stress
[root@localhost stress]# docker exec -it 3a61d26ac823 bash
[root@98d2aaa50019 /]# cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
100000
[root@98d2aaa50019 /]# cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
200000

Four, CPU Core control

For servers with multi-core CPUs, Docker can also control which CPU cores the container runs, that is, use the -cpuset-cpus parameter.
This is especially useful for servers with multiple CPUs, and can be configured with optimal performance for containers that require high-performance computing.

[root@localhost stress]# docker run -tid --name cpu1 --cpuset-cpus 0-1 centos:stress
执行以上命令需要宿主机为双核，表示创建的容器只能用0、1两个内核。最终生成的cgroup的 CPU 内核配置如下:
[root@localhost stress]# docker exec -it 631eea630b21 bash

[root@631eea630b21 /# cat /sys/fs/cgroup/cpuset/cpuset.cpus
0-1

通过下面指令可以看到容器中进程与CPU内核的绑定关系，达到绑定CPU内核的目的。
[root@localhost stress]# docker exec 631eea630b21 taskset -c -p 1 //容器内部第- 个进程号pid为1被绑定到指定CPU上运行
pid 1's current affinity list:0,1

5. Mixed use of CPU quota control parameters

Use the cpuset-cpus parameter to specify that container A uses CPU core 0, and container B only uses CPU core 1.
On the host, only these two containers use the corresponding CPU cores, and they each occupy all the core resources, and cpu-shares have no obvious effect .
The cpuset-cpus and cpuset-mems parameters are only valid on servers with multi-core and multi-memory nodes, and must match the actual physical configuration, otherwise the purpose of resource control cannot be achieved.

When the system has multiple CPU cores, it is necessary to set the container CPU core through the cpuset-cpus parameter to facilitate the test.

//宿主系统修改为4核心CPU
[root@localhost stress]# docker run -tid --name cpu3 --cpuset-cpus 1 --cpu-shares 512 centos:stress stress -c 1
[root@localhost stress]# docker exec -it 84598dfadd34 bash
[root@localhost stress]# exit


[root@localhost stress]# docker run -tid --name cpu4 --cpuset-cpus 3 --cpu-shares 1024 centos:stress stress -c 1
[root@localhost stress]# top //记住按1查看每个核心的占用

Tasks: 172 total, 2 running, 170 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0us, 0.0sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0si, 0.0 st
%Cpu1 : 0.0us, 0.0sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0si, 0.0 st
%Cpu2 : 0.0us, 0.0sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0si, 0.0 st
%Cpu3 :100.0us, 0.0sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0si, 0.0 st
KiB Mem : 7994072 total, 6394056 free, 450124 used, 1149892 buff/cache
KiB Swap: 4194300 total, 4194300 free,  0 used.7174064 avail Mem

[root@localhost stress]# docker exec -it 0eed2c8a20df bash

总结:上面的centos:stress镜像安装了stress工具，用来测试CPU和内存的负载。通过在两个容器上分别执行stress-c 1命令,
将会给系统- 个随机负载，产生1个进程。这个进程都反复不停的计算由rand()产生随机数的平方根，直到资源耗尽。
观察到宿主机上的CPU使用率，第三个内核的使用率接近100%， 并且一批进程的CPU使用率明显存在2:1的使用比例的对比。

Sixth, memory limit

Similar to the operating system, the memory that can be used by the container includes two parts: physical memory and swap.

Docker controls the usage of container memory through the following two sets of parameters.

-m或--memory: 设置内存的使用限额，例如100M、1024M。
--memory-swap: 设置内存+swap的使用限额。

Execute the following command to allow the container to use up to 200M of memory and 300M of swap.

[root@localhost stress]# docker run -it -m 200M --memory-swap=300M progrium/stress --vm 1 --vm-bytes 280M
--vm 1: 启动1个内存工作线程。
--vm-bytes 280M: 每个线程分配280M内存。

By default, the container can use all the free memory on the host.
Similar to the cgroups configuration of the CPU, Docker will automatically /sys/fs/cgroup/memory/docker/<容器的完整长ID>create the corresponding cgroup configuration file in the directory for the container

If the memory allocated by the worker thread exceeds 300M and the allocated memory exceeds the limit, the stress thread reports an error and the container exits.

[root@localhost stress]# docker run -it -m 200M --memory-swap=300M progrium/stress --vm 1 --vm-bytes 310M

Seven, Block IO limitations

By default, all containers can read and write to the disk equally. You can change the priority of the container block IO by setting the -blkio-weight parameter.
–Blkio-weight is similar to –cpu-shares. It sets the relative weight value and the default is 500.

In the following example, the bandwidth of container A to read and write to the disk is twice that of container B

[root@localhost ~]#docker run -it --name container_A --blkio-weight 600 centos:stress
[root@0d16f3608f62 /]# cat /sys/fs/cgroup/blkio/blkio.weight
600

[root@localhost ~]#docker run -it --name container_B --blkio-weight 300 centos:stress
[root@ae6c2a3370b6 /]# cat /sys/fs/cgroup/blkio/blkio.weight
300

8. Limitations of bps and iops

bps It is byte per second, the amount of data read and written per second.
iops Is io per second, the number of IOs per second.

The bps and iops of the container can be controlled by the following parameters:

–device-read-bps，限制读某个设备的 bps。
–device-write-bps，限制写某个设备的 bps。
–device-read-iops，限制读某个设备的 iops。
–device-write-iops，限制写某个设备的 iops。

The following example limits the rate at which the container writes /dev/sda to 5 MB/s.

[root@localhost docker]# docker run -it --device-write-bps /dev/sda:5MB centos:stress

[root@c4a16f026e5a /]# dd if=/dev/zero of=test bs=1M count=1024 oflag=direct   //可以按ctrl+c中断查看
906+0 records in
906+0 records out
950009856 bytes (950 MB) copied, 181.202 s, 5.2 MB/s

Use the dd command to test the speed of writing to the disk in the container. Because the file system of the container is on host /dev/sda
, writing files in the container is equivalent to writing to host /dev/sda. In addition, oflag=direct specifies direct IO to write files,
so that --device-write-bps can take effect.

The result shows that the speed limit is about 5MB/s. As a comparative test, if the speed is not limited, the results are as follows.

[root@localhost docker]# docker run -it centos:stress

[root@012ab0128349 /]# dd if=/dev/zero of=test bs=1M count=1024 oflag=direct
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.928611 s, 1.2 GB/s