docker-------Cgroup resource configuration method

Preface

By default, there is no resource limit for a container, because it is a process itself. When a container occupies too many resources, it will affect other containers. Therefore, the reasonable allocation of container resources is a problem that administrators must pay attention to.
docker passes Cgroup To control the resource quota used by the container, including the three major aspects of CPU, memory, and disk, which basically covers common resource quotas and usage control

1. Cgroup resource configuration

Cgroup is the abbreviation of Control Groups. It is a mechanism provided by the Linux kernel to limit, record, and isolate the physical resources (such as CPU, memory, disk l0, etc.) used by process groups. It is used by many projects such as LXC and docker. Realize process resource control. Cgroup itself is a basic structure that provides functions and interfaces for grouping processes. Specific resource management such as I/O or memory allocation control is realized through this function. These specific resource management functions are called Cgroup subsystems. , There are the following major subsystems:

• blkio: set to limit the input and output control of each device, such as: disk, CD, usb, etc.
• CPU: Use the scheduler to provide CPU access for cgroup tasks
• cpuacct: Generate CPU resource reports for cgroup tasks
• cpuset: If it is a multi-core CPU, this subsystem will allocate separate CPU and memory for cgroup tasks
• devices: Allow or deny access to devices by cgroup tasks
• freezer: Pause and resume cgroup tasks
• memory: Set the memory limit of each cgroup and generate a memory resource report
• net_cls: mark each network packet for convenient use by cgroup
• ns: Namespace subsystem
• perf_event: Increase the monitoring and tracking capabilities of each group, which can monitor all threads belonging to a specific group and allow threads on a specific CPU

Two, test CPU and memory

Use Dockerfile to create a centos-based stress tool image

mkdir /opt/stress
cd /opt/stress

vim Dockerfile
FROM centos:7
MAINTAINER HZH<397615552@qq.com>
RUN yum install -y wget
RUN wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
RUN yum install -y stress

docker build -t centos:stress .

Use the following command to create a container. The -cpu-shares parameter value in the command does not guarantee that you can get 1 vcpu or how many GHz of CPU resources, it is only a flexible weighted value

docker run -itd --cpu-shares 100 centos:stress

说明:默认情况下，每个Docker容器的CPU份额都是1024。 单独一个容器的份额是没有意义的。只有在同时运行多个容器时，容器的CPU加权的效果才能体现出来。
For example: start 2 containers and allow to view the percentage of CPU usage

docker run -itd --name cpu512 --cpu-shares 512 centos:stress stress -c 10  #容器产生10个子函数进程
docker exec -it xxxxxxxx bash  		#进入容器使用top查看cpu使用情况

再开一个容器做比较
docker run -itd ---name cpu1024 --cpu-shares 1024 centos:stress stress -c 10
docker exec -it xxxxxxxxx bash      #进入容器使用top对比两个容器的%CPU,比例是1：2

Three, CPU cycle limit

1. Docker provides two parameters --cpu-period and --cpu-quota to control the CPU clock cycles that the container can allocate

• --Cpu-period: It is used to specify how long the CPU usage of the container should be redistributed
• --Cpu-quota: It is used to specify the maximum amount of time that can be used to run the container in this cycle
• Unlike –cpu-share, this configuration is an absolute value specified, and the use of CPU resources by the container will never exceed the configured value
  . The unit of –cpu-period and –cpu-quota is microseconds.
• The minimum value of -cpu-period is 1000 microseconds, the maximum value is 1 second, and the default is 0.1 second
• The default value of –cpu-quota is -1, which means no control. The –cpu-period and –cpu-quota parameters are generally used in combination

For example: the container process needs to use a single CPU for 0.2 seconds every 1 second. You can set the cpu-period to 100000 (that is, 0.1 seconds) and the cpu-quota to 200000 (0.2 seconds). Of course, in the case of multiple cores, if allowed If the container process completely occupies two CPUs, you can set cpu-period to 100000 (that is, 0.1 second) and cpu-quota to 200000 (0.2 second)

docker run -itd --cpu-period 100000 --cpu-quota 200000 centos:stress
docker exec -it xxxxxxxxxx
cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us

Four, CPU Core control

For servers with multi-core CPUs, Docker can also control which CPU cores the container is allowed to use, that is, use the -cpuset-cpus parameter.
This is especially useful for servers with multiple CPUs and can configure the best performance for containers that require high-performance computing.

docker run -itd --name cpu1 --cpuset-cpus 2-3 centos:stress			#2-3表示创建容器只能使用第2、3这两个内核，
docker exec -it xxxxxxxxxxx bash
cat /sys/fs/cgroup/cpuset/cpuset.cpus

docker exec xxxxxx taskset -c -p 1
pid 1's current affinity list: 2,3

5. Mixed use of CPU quota control parameters

Use the cpuset-cpus parameter to specify that container A uses CPU core 0, and container B uses only CPU core 3.
On the host, only these two containers use the corresponding CPU cores. They each occupy all the core resources, and cpu-shares have no obvious effect
. -cpus, cpuset-mems parameters are only valid on servers on multi-core and multi-memory nodes, and must match the actual physical configuration, otherwise the purpose of resource control cannot be achieved.
In the case of multiple CPU cores in the system, it needs to be passed The cpuset-cpus parameter is to set the container CPU core to facilitate testing

docker run -itd --name cpu3 --cpuset-cpus 1 --cpu-shares 512 centos:stress stress -c 1 
docker exec -it xxxxxxxxxxx bash
exit 
top #按1查看每个核心的占用

docker run -itd --name cpu4 --cpuset-cpus 3 --cpu-shares 1024 centos:stress stress -c 1 
docker exec -it xxxxxxx bash    

总结：上面的centos:stress镜像安装了stress工具，用来测试CPU和内存的负载，通过在两个容器上分别执行stress -c 1命令，将会给系统一个随机负载，产生1个进程，这个进程都会反复不停的计算由rand()产生随机数的平方根，直到资源耗尽，观察到宿主机上CPU使用率，第三个内核的使用率接近100%，并且一批进程的CPU使用率明显存在2：1 的使用比例的对比

Sixth, memory limit

Similar to the operating system, the memory that the container can use includes two parts: physical memory and Swap
docker controls the usage of container memory through the following two sets of parameters

• -m or --memory: set memory usage limit
• --Memory-swap: set memory + swap usage limit
• --Vm 1: start 1 memory worker thread
• –Vm-bytes 280M: each thread allocates 280M memory, the value here should be less than –memory-swap, otherwise an error will be reported

docker run -it -m 200M --memory-swap=300M progrium/stress --vm 1 --vm-bytes 280M

docker run -it -m 200M --memory-swap=300M progrium/stress --vm 1 --vm-bytes 310M			#工作线程分配的内存超过300M或者等于300M，分配的内存超过限额，stress线程会报错，容器退出

–Vm-bytes is equal to –memory-swap value, the result is an error

Seven, Block IO limitations

By default, all containers can read and write to the disk equally. You can change the priority of the container block IO through the –blkio-weight parameter.
–blkio-weight is similar to –cpu-shares. The relative weight is set. The default is 500.
For example : The bandwidth of container A to read and write the disk is twice that of container B

docker run -it --name container_A --blkio-weight 600 centos:stress
cat /sys/fs/cgroup/blkio/blkio.weight

docker run -it --name container_B --blkio-weight 300 centos:stress
cat /sys/fs/cgroup/blkio/blkio.weight

8. Limitations of bps and iops

bps is byte per second: the amount of data read and written per second
iops is io per second: the number of IOs per second.
The bps and iops of the container can be controlled by the following parameters

–Device-read-bps: limit reading of bps of a certain device
–device-write-bps: limit writing of bps of a certain device
–device-read-iops: limit reading of iops of a certain device
–device-write-iops: limit Write the iops of a certain device

The following example is to limit the rate at which the container writes /dev/sda to 5MB/s

docker run -it --device-write-bps /dev/sda:5MB centos:stress
dd if=/dev/zero of=test bs=1M count=1024 oflag=direct			#可以按ctrl+c中断查看

docker run -it centos:stress
dd if=/dev/zero of=test bs=1M count=1024 oflag=direct