Cgroups is a mechanism provided by the Linux system kernel. This mechanism can integrate or separate a series of system tasks and machine subtasks into different groups registered according to resource divisions according to requirements, thereby providing a framework for system resource management. Simply put, Cgroups can limit and record the physical group members (such as CPU, Memory, IO, etc.) used by task groups, providing a basic guarantee for container virtualization, and is the cornerstone of building a series of virtualization management tools such as Docker.
From the launch of the open-source Docker in 2013 and the emergence of the open-source Kubernetes in 2014, to the current popularization and popularity of cloud-native technology and ecosystem, container technology has gradually become one of the mainstream basic cloud-native technologies. The use of container technology can well achieve resource level limitation and isolation, which all rely on the Cgroups and Namespace technologies provided by the Linux system kernel.
Cgroups are mainly used to manage the allocation and limitation of resources; Namespace is mainly used to encapsulate abstraction, limitation, and isolation of resources, so that processes in the namespace have their own global resources. The Cgroups and Namespace technologies provided by the Linux kernel provide a basic guarantee for container virtualization and are the cornerstone of building a series of virtualization management tools such as Docker.
Cgroups is the abbreviation of control groups. It is a mechanism provided by the Linux kernel to limit, record, and isolate the physical resources (such as CPU, Memory, IO, etc.) used by process groups.
By using Cgroups, system administrators can exercise fine-grained control over allocation, sorting, rejection, management, and monitoring of system resources. Hardware resources can be intelligently allocated between applications and users, increasing overall efficiency. It was originally proposed by Google engineers, and later integrated into the Linux kernel, which is also one of the foundations of the current lightweight virtualization technology XC (Linux Container).
Cgroups is similar to Namespace in that it also groups processes, but its purpose is different from Namespace. Namespace is to isolate resources between process groups, while Cgroups is to perform unified resource monitoring and limitation on a group of processes.
Cgroups is divided into two versions, v1 and v2. v1 was implemented earlier and has more functions, but because the functions in it are implemented in a scattered manner, the planning is not very good, which leads to some inconveniences in use and maintenance. , the emergence of v2 is to solve this problem in v1. In the latest 4.5 kernel, Cgroups v2 claims that it can be used in the production environment, but the functions it supports are still very limited. There are Cgroups, Namespace, v1 and v2 can be mixed, but this will be more complicated, so generally no one will use it.
2. The main structure of Cgroups
Cgroups is a mechanism for managing processes in groups under Linux. From the perspective of the user layer, Cgroups technology is to organize all the processes in the system into independent trees. Each tree contains all the processes of the system. Each node of the tree is a process group, and each tree is associated with one or more subsystems. The function of the tree is to group processes, and the function of the subsystem is to operate these groups. The main structure of Cgroups is as follows:
Cgroups mainly include the following two parts:
subsystem: A subsystem is a kernel module. After it is associated with a cgroup tree, it will perform specific operations on each node (process group) of the tree. Subsystem is often called resource controller, because it is mainly used to schedule or limit the resources of each process group, but this statement is not completely accurate, because sometimes we group processes into groups just to do some monitoring and observe their status, such as perf_event subsystem. So far, Linux supports 12 subsystems, such as limiting CPU usage time, limiting memory usage, counting CPU usage, freezing and resuming a group of processes, and so on.
hierarchy: A hierarchy can be understood as a cgroup tree, each node of the tree is a process group, and each tree is associated with zero or more subsystems. In a tree, all processes in the Linux system will be included, but each process can only belong to one node (process group). There can be many cgroup trees in the system, and each tree is associated with different subsystems. A process can belong to multiple trees, that is, a process can belong to multiple process groups, but these process groups are associated with different subsystems.
At present, Linux supports 12 kinds of subsystems. If you do not consider the situation that is not associated with any subsystem (systemd belongs to this case), you can build up to 12 cgroup trees in Linux, and each tree is associated with a subsystem. Of course, you can only build one tree, and then let this tree associate all subsystems. When a cgroup tree is not associated with any subsystem, it means that the tree only groups processes. As for what to do on the basis of grouping, it will be decided by the application itself, as is the case with systemd.
3. Why do you need Cgroups?
In Linux, there has always been the concept and demand for grouping processes, such as session group, progress group, etc. Later, as people have more and more needs for this, such as the need to track the memory and IO usage of a group of processes situation, etc., so cgroup appeared, which is used to group processes uniformly, and monitor processes and resource control management on the basis of grouping.
For example, if the antivirus software ESET or ClamAV is installed in the Linux system, the antivirus takes up too many system resources and affects the system's business operation. What should I do? What should I do if a single virtual machine process or docker process uses too many resources? What should I do if a single Java resource takes up too much system memory? Cgroup is a tool that can control and solve the above problems. Cgroup is implemented in the linux kernel and used to control linux system resources.
4. How are Cgroups implemented?
On CentOS 7 systems (including Red Hat Enterprise Linux 7), resource management settings can be moved from the process level to the application level by bundling the cgroup hierarchy system with the systemd unit tree. By default, systemd will automatically create a hierarchy of slice, scope and service units (the specific meaning will be explained later) to provide a unified structure for the cgroup tree.
This structure can be further modified by creating a custom slice through the systemctl command. If the resources of the system are regarded as a piece of pie, then all resources will be divided into 3 cgroups by default: System, User and Machine. Each cgroup is a slice, and each slice can have its own sub-slice, as shown in the following figure:
Take CPU resources as an example to explain some keywords that appear in the above figure. As shown in the figure above, the system creates 3 top-level slices (System, User and Machine) by default, and each slice will get the same CPU usage time (only when the CPU is busy), if user.slice wants to get 100% CPU Use time, and the CPU is relatively idle at this time, then user.slice will be able to get what it wants. The meanings of these three top-level slices are as follows:
system.slice: the default location of all system services;
user.slice: The default location of all user sessions. Each user session will create a sub-slice under this slice. If the same user logs in to the system multiple times, the same sub-slice will still be used;
machine.slice: The default location of all virtual machines and Linux containers. One of the methods to control the use of CPU resources is shares. Shares are used to set the relative value of CPU (which can be understood as weight), and it is for all CPUs (cores). The default value is 1024, so in the figure above, the CPU shares of httpd, sshd, crond and gdm are all 1024, and the CPU shares of System, User and Machine are also 1024.
Assuming that 4 services are running on the system, two users are logged in, and a virtual machine is also running, and assuming that each process requires as much CPU resources as possible (every process is very busy), then:
system.slice will get 33.333% of the CPU usage time, and each service will get 1/4 of the CPU usage time from the resources allocated by system.slice, that is, 8.25% of the CPU usage time;
user.slice will get 33.333% CPU usage time, and each logged-in user will get 16.5% CPU usage time, assuming there are two users: tom and jack, if tom logs out or kills all users under the user session process, jack can use 33.333% of the CPU usage time;
machine.slice will get 33.333% of the CPU usage time, if the virtual machine is shut down or in idle state, then system.slice and user.slice will get 50% of the CPU resources from the 33.333% of the CPU resources, and then both Assigned to their sub-slices.
Five, the role of Cgroups
The initial goal of Cgroups is to provide a unified framework for resource management, which not only integrates existing subsystems such as cpuset, but also provides interfaces for developing new subsystems in the future. The current cgroups are suitable for a variety of application scenarios, from the resource control of a single process to the realization of OS level virtualization (OS Level Virtualization). The framework diagram is as follows:
Cgroups provide the following functionality:
Limit the number of resources that a process group can use (Resource limiting). For example, the memory subsystem can set a memory usage limit for the process group. Once the memory used by the process group reaches the limit and then applies for memory, OOM (out of memory) will occur. ;
Process group priority control (Prioritization), for example: you can use the cpu subsystem to assign a specific cpu share to a process group;
Record the number of resources used by the process group (Accounting), for example: you can use the cpuacct subsystem to record the cpu time used by a process group;
Process group isolation (Isolation), for example: using the ns subsystem, different process groups can use different namespaces to achieve the purpose of isolation. Different process groups have their own process, network, and file system mounting spaces;
Process group control (Control), for example: use the freezer subsystem to suspend and resume the process group.
6. Cgroups related concepts and interrelationships
① Related concepts
Task (task): In cgroups, a task is a process of the system.
Control group (control group): A control group is a group of processes divided according to some standard. Resource control in Cgroups is implemented in units of control groups. A process can join a control group and also migrate from one process group to another control group. The processes of a process group can use the resources allocated by cgroups in units of control groups, and are subject to the restrictions set by cgroups in units of control groups.
Hierarchy: Control groups can be organized in a hierarchical form, that is, a control group tree. The child node control group on the control group tree is the child of the parent node control group, and inherits the specific attributes of the parent control group.
Subsystem: A subsystem is a resource controller. For example, the cpu subsystem is a controller that controls the CPU time allocation. Subsystems must be attached to a level to function. After a subsystem is attached to a certain level, all control groups on this level are controlled by this subsystem.
② Mutual relationship
Every time a new level is created in the system, all tasks in the system are the default cgroup of that level (called the root cgroup, this cgroup is automatically created when the level is created, and the cgroups created in the level later are all this cgroup's descendants) initial members.
A subsystem can only be attached to at most one level.
A hierarchy can have multiple subsystems attached to it.
A task can be a member of multiple cgroups, but these cgroups must be at different levels.
When a process (task) in the system creates a child process (task), the child task automatically becomes a member of the cgroup where its parent process resides, and then the child task can be moved to a different cgroup as needed, but it always inherits at the beginning The cgroup of its parent task.
Seven, Cgroups subsystem
① Subsystem subsystem
It can be seen that there are many subdirectories such as cpu and memory under /sys/fs/cgroup, which are also called subsystem subsystems:
It is a set of resource control modules, generally including the following items:
net_cls: Classify the network packets generated by the processes in the cgroup, so that Linux's tc (traffic controller) can distinguish the packets from a cgroup according to the classification and do current limiting or monitoring. This subsystem uses the class identifier (classid) to mark the network packets, which allow the Linux traffic control program (tc) to identify packets generated from specific cgroups;
net_prio: Set the priority of network traffic generated by processes in cgroup;
memory: Control the memory usage of processes in cgroup;
cpuset: Set the cpu and memory available to the processes in the cgroup on a multi-core machine. This subsystem allocates independent CPUs (in multi-core systems) and memory nodes for tasks in the cgroup;
freezer: Suspend and resume (resume) the processes in the cgroup, this subsystem suspends or resumes the tasks in the cgroup;
blkio: Set access control for input and output of block devices (such as hard disks). This subsystem sets input/output restrictions for block devices, such as physical devices (disks, SSDs, USB, etc.);
cpu: Set the CPU usage of the processes in the cgroup. This subsystem uses the scheduler to provide cgroup task access to the CPU;
cpuacct: Statistics of the CPU usage of the processes in the cgroup, this subsystem automatically generates the CPU report used by the tasks in the cgroup;
devices: Control access to devices by processes in cgroups 16 This subsystem allows or denies tasks in cgroups to access devices.
② How to check which subsystems the current system supports?
You can know which subsystems the current system supports by viewing /proc/cgroups (since Linux 2.6.24), as follows:
hierarchy: The ID of the cgroup tree associated with the subsystem. If multiple subsystems are associated with the same cgroup tree, then their field will be the same. For example, the cpu and cpuacct here are the same, indicating that they are bound to the same tree. If This field will be 0 in the following situations:
The current subsystem is not bound to any cgroup tree;
The current subsystem has been bound to the tree of cgroup v2;
The current subsystem is not enabled by the kernel.
num_cgroups: the number of process groups in the cgroup tree associated with the subsystem, that is, the number of nodes on the tree;
enabled: 1 means enabled, 0 means not enabled (you can control the enabling of the subsystem by setting the kernel startup parameter cgroup_disable).
③ CPU subsystem under Cgroups
The cpu subsystem is used to control the cpu time slices available to all processes in a cgroup. The cpu subsystem mainly involves 5 interfaces: cpu.cfs_period_us, cpu.cfs_quota_us, cpu.shares, cpu.rt_period_us, cpu.rt_runtime_us.cpu.
cfs_period_us: cfs_period_us represents the bandwidth of a cpu, in microseconds, the total CPU bandwidth of the system: the number of cpu cores * cfs_period_us cpu.
cfs_quota_us: cfs_quota_us indicates the cpu bandwidth that Cgroup can use, in microseconds, cfs_quota_us is -1, which means that the CPU used is not limited by cgroup, the minimum value of cfs_quota_us is 1ms (1000), and the maximum value is 1s. Combined with cfs_period_us, the You can limit the cpu used by the process. For example, configure cfs_period_us=10000 and cfs_quota_us=2000. Then the process can use 2 cpu cores;
cpu.shares: Through cfs_period_us and cfs_quota_us, the cpu usage of cgroup can be limited in absolute proportion, that is, cfs_quota_us/cfs_period_us is equal to the cpu cores that the process can use, and cannot exceed this value, while cpu.shares limits the cpu of cgroup in relative proportion. For example: a task with cpu.shares set to 1 in both cgroups will have the same CPU time, but a task with cpu.shares set to 2 in the cgroup can use CPU time as much as cpu in the cgroup Tasks with .shares set to 1 can use twice as much CPU time;
cpu.rt_runtime_us: In microseconds (µs, represented by "us" here), specify the longest continuous access time for tasks in cgroup to CPU resources in a certain period of time. This limit was established to prevent tasks in a cgroup from monopolizing CPU time. If the tasks in the cgroup should have access to CPU resources for 4 out of every 5 seconds, set cpu.rt_runtime_us to 4000000 and cpu.rt_period_us to 5000000.
cpu.rt_period_us: In microseconds (µs, represented by "us" here), specify the frequency of cgroup reallocation of CPU resource access in a certain period of time. If tasks in a cgroup should have access to CPU resources for 4 seconds out of every 5 seconds, set cpu.rt_runtime_us to 4000000 and cpu.rt_period_us to 5000000. Note that sched_rt_runtime_us is the guaranteed time and maximum occupied time of real-time tasks. If the real-time tasks are not used, they can be allocated to non-real-time tasks, and the final time occupied by real-time tasks cannot exceed this value. Refer to Linux-85 about sched_rt_runtime_us and sched_rt_period_us. The restriction on the cpu.rt_period_us parameter is that it must be smaller than the parameter value of the same name in the parent directory.
The current real-time process scheduling algorithm may cause some real-time processes to be starved to death. As follows, A and B are parallel, and the running time of A just covers the running time of B:
* group A: period=100000us, runtime=50000us
- this runs for0.05s once every 0.1s
* group B: period= 50000us, runtime=25000us
- this runs for0.025s twice every 0.1s (or once every 0.05 sec).
In Real-Time group scheduling, it is proposed that SCHED_EDF (Earliest Deadline First scheduling) is being developed to prioritize the scheduling of the real-time process that ends first.
④ Install Cgroups in CentOS
#若系统未安装则进行安装，若已安装则进行更新
yum install libcgroup
#查看运行状态，并启动服务
[root@localhost ~] service cgconfig status
Stopped[root@localhost ~] service cgconfig start
Starting cgconfig service:[OK]
service cgconfig status 9Running1011
#查看是否安装cgroup
[root@localhost ~] grep cgroup /proc/filesystems
There are many properties that can be set here, but not all properties are used to set cgroup, only need to pay attention to Block, CPU and Memory. If you want to set the cgroup through the configuration file, the service can directly create the corresponding configuration file under the /etc/systemd/system/xxx.service.d directory, and the slice can be directly in the /run/systemd/system/xxx.slice.d directory Create the corresponding configuration file below. In fact, setting the cgroup through the systemctl command line tool will also be written to the configuration file in this directory:
If you want to strictly control CPU resources, set the upper limit of CPU resources, that is, no matter whether the CPU is busy or not, the use of CPU resources cannot exceed this upper limit. It can be set by the following two parameters:
cpu.cfs_period_us = The period for counting the CPU usage time, the unit is microseconds (us);
cpu.cfs_quota_us = CPU time allowed in a cycle (referring to the time of single core, multi-core needs to be accumulated when setting);
systemctl can set the upper limit of CPU resource usage through the CPUQuota parameter. For example, if you want to set the CPU resource usage limit of user tom to 20%, you can execute the following command:
mkdir /cgroup/cpuset
mount ‐t cgroup ‐o cpuset red /cgroup/cpuset
……
mkdir /cgroup/blkio
[root@localhost ~] vi /etc/cgrules.conf
[root@localhost ~] echo 524288000>/cgroup/memory/foo/memory.limit_in_b ytes
Use cgroup to temporarily adjust the process, just pass the command directly. If you want to control the process persistently, that is, it will still be valid after restarting, you need to write it into the configuration files /etc/cgconfig.conf and /etc/cgrules.conf.
Nine, view Cgroup
① View cgroup through systemd
systemd-cgls command:
View through the systemd-cgls command, it will return the overall cgroup level of the system, the top level of the cgroup tree is composed of slices, as shown below:
It can be seen that the top layer of the system cgroup hierarchy consists of user.slice and system.slice. Because there are no virtual machines and containers running in the system, there is no machine.slice, so when the CPU is busy, user.slice and system.slice will each get 50% of the CPU usage time.
There are two sub-slices under user.slice: user-1000.slice and user-0.slice, and each sub-slice is named with a User ID (UID), so it is easy to identify which slice belongs to which user. For example, from the above output information, it can be seen that user-1000.slice belongs to user tom, and user-0.slice belongs to user root.
systemd-cgtop command:
The systemd-cgls command only provides a snapshot of static information at the cgroup level. To view the dynamic information at the cgroup level, you can view it through the systemd-cgtop command:
The statistical data and control options provided by scope systemd-cgtop are similar to the top command, but this command only displays those services and slices that have resource statistics enabled. If you want to enable the resource statistics function of sshd.service, you can do the following:
At this time, run the systemd-cgtop command again, and you can see the resource usage statistics of sshd.
② View cgroup through proc
How to check which cgroups the current process belongs to You can know which cgroups the specified process belongs to by viewing /proc/[pid]/cgroup (since Linux 2.6.24), as follows:
The ID of the cgroup tree corresponds to the ID in the /proc/cgroups file;
All subsystems bound to the cgroup tree are separated by commas. Here name=systemd means that it is not bound to any subsystem, but it is named systemd;
The path of the process in the cgroup tree, that is, the relative path of the cgroup to which the process belongs relative to the mount point.
③ View cgroup through /sys
View the usage limit of CPU resources under cgroup:
This means that user tom can use 20 milliseconds of CPU time in one usage cycle (100 milliseconds). Regardless of whether the CPU is idle or not, the CPU resources used by this user will not exceed this limit.
The value of CPUQuota can exceed 100%. For example, if the CPU of the system is multi-core, and the value of CPUQuota is 200%, then the slice can use the CPU time of 2 cores.
