Principles of Linux operating system-cgroups process resource management and limitation

table of Contents

cgroups

Cgroups (control groups) is a mechanism provided by Linux Kernel that can limit the resources used by a single process or multiple processes. It can achieve fine-grained control of resources such as CPU, Memory, I/O, etc. The lightweight container technology Docker The resource restriction capabilities provided by cgroups are used to complete operating system-level resource control, and then to realize the virtualization of the operating system.

Subsystem of cgroups

cgroups defines a subsystem for each resource that can be controlled. The typical subsystems are as follows:

• cpu subsystem : mainly restricts the CPU usage of the process.
• cpuacct subsystem : It can count the CPU usage report of the processes in cgroups.
• cpuset subsystem : you can allocate separate CPU or Memory nodes for processes in cgroups.
• Memory subsystem : You can limit the memory usage of the process.
• devices subsystem : can control the process to be able to access certain devices.
• blkio subsystem : can limit the block device I/O of the process.
• net_cls subsystem : you can mark the network data packets of processes in cgroups, and then you can use the tc (traffic control) module to control the data packets.
• Freezer subsystem : You can suspend or resume processes in cgroups.
• ns subsystem : You can make processes under different cgroups use different namespaces.

Note that each subsystem needs to cooperate with other modules of Kernel to complete resource control, such as:

• CPU resources is restricted by the process scheduling module to complete the configuration according to the cpu subsystems;
• Memory resource limit is the memory modules according to the configuration of the memory subsystem to complete.
• The control of network data packets requires the Traffic Control module to complete according to the configuration of the net_cls subsystem.

The hierarchy of cgroups

The kernel uses cgroup Struct (structure) to represent the resource limit of a control group to several cgroups subsystems.

Cgroup Struct can be organized into a tree, called a cgroups Hierarchy (hierarchy), each cgroups Hierarchy can attach several cgroups subsystems, and cgroups Hierarchy can limit the resources of the cgroups subsystems attached to it. Each cgroups subsystem can only be attached to one cgroups Hierarchy.

In the cgroups Hierarchy A and B in the above figure, each Hierarchy is a tree structure, and each node of the tree is a cgroup Struct, such as: cpu_cgrp, memory_cgrp.

1. cgroups Hierarchy A: Attach the cpu subsystem and cpuacct subsystem. The cgroup Struct in the current cgroups Hierarchy can limit the CPU resources and count the CPU usage of the process.
2. cgroups Hierarchy B: Attach the memory subsystem. The cgroup Struct in the current cgroups Hierarchy can limit the resources of the Memory.

And, in each cgroups Hierarchy, each node (cgroup Struct) can set different restriction weights for resources. As shown in the figure above, the processes in the cgrp1 group can use 60% of the CPU time slice, and the processes in the cgrp2 group can use 20% of the CPU time slice.

cgroups and processes

As mentioned above, the Linux Kernel uses the cgroups subsystem to limit system resources, and it is also mentioned that the cgroups subsystem needs to be attached to the cgroups Hierarchy to control the resources of the process. So how does Kernel associate the Processor with the cgroups hierarchy?

After the node (cgroup Struct) in the cgroups Hierarchy is created, the processor can be added to the control task list of a node. All processors in the control list of a node will be restricted by the resources of the current node. At the same time, a Processor can also be added to the nodes of different cgroups Hierarchy, because different cgroups Hierarchy can be responsible for different system resources.

So Processor and cgroup Struct are in a many-to-many relationship, which means that different processors are controlled by different cgroup Structs and thus have different resource restrictions.

The figure above describes the relationship between processes and cgroups. P represents a process. A pointer in the descriptor of each process points to a css_set (cgroups subsystem set) data structure, and a process that points to a css_set will be added to the css_set process list. A process can only belong to one css_set, and a css_set can contain multiple processes. Processes belonging to the same css_set are subject to the resource restrictions associated with the css_set.

Among them, M×N Linkage means that css_set can have many-to-many association with cgroup Struct. But the implementation of cgroups does not allow css_set to associate multiple cgroup Structs under the same cgroups hierarchy at the same time. This is because cgroups do not allow multiple restriction configurations for the same resource. When a css_set is associated with multiple cgroups hierarchy nodes, it indicates that it is necessary to control multiple resources of the processes under the current css_set. When a cgroups node is associated with multiple css_sets, it indicates that the process lists under multiple css_sets are subject to the same restrictions of the same resource.

cgroups and file system

Linux uses a variety of data structures to implement the configuration of cgroups in the kernel, associating Processor and cgroup Struct, then how does Linux allow user-mode processes to use the functions of cgroups?

Linux File System is implemented as VFS (Virtual File System). VFS can hide the details of different types of file system providers at the bottom, and provide a unified file system API interface for user-mode processes. Cgroups also exposes functions to user mode through VFS. The connection between cgroups and VFS is called cgroups File System.

File systems implemented based on VFS must implement the objects defined by the VFS general file model, as is cgroups File System. By implementing the VFS general file system model, cgroups hides the details of maintaining the cgroups Hierarchy in these implementation functions of the cgroups File System .

Use of cgroups

Mount cgroups File System

Before using cgroups, you must first mount the cgroups File System, command:

mount -t cgroup -o subsystems name /cgroup/name

• subsystems indicates the cgroups subsystems that need to be mounted.
• /cgroup/name represents the mount point.

As mentioned above, this command creates a cgroups Hierarchy in the kernel.

For example, if you mount the four cgroupssubsystems of cpuset, cpu, cpuacct, and memory to the /cgroup/cpu_and_mem directory, execute:

mount -t cgroup -o remount,cpu,cpuset,memory cpu_and_mem /cgroup/cpu_and_mem

In CentOS, after yum install libcgroupinstalling the cgroups module, the mount point of the cgroups subsystem is automatically generated in the /etc/cgconfig.conf configuration file:

mount {
    cpuset  = /cgroup/cpuset;
    cpu = /cgroup/cpu;
    cpuacct = /cgroup/cpuacct;
    memory  = /cgroup/memory;
    devices = /cgroup/devices;
    freezer = /cgroup/freezer;
    net_cls = /cgroup/net_cls;
    blkio   = /cgroup/blkio;
}

Each configuration above is equivalent to the expanded mount command.

Create child nodes

After mounting a cgroups subsystem to the mount point, you can create a node (cgroup Struct) in the cgroups Hierarchy by creating a folder under the mount point or using the cgcreate command. For example, through the command:

cgcreate -t sankuai:sankuai -g cpu:test

You can create a cgroup Struct named test under cpu subsystem. The result is as follows:

$ ls
cgroup.event_control  cgroup.procs  cpu.cfs_period_us  cpu.cfs_quota_us  cpu.rt_period_us   cpu.rt_runtime_us  cpu.shares  cpu.stat  lxc  notify_on_release  release_agent  tasks  test

Then you can configure the resources that need to be restricted by writing the required values ​​to different files under test. A variety of different configurations can be performed under each subsystem, and the parameters that need to be configured are different. For detailed parameter settings, please refer to the man cgroups manual. For example: using the cgset command can also set the parameters of the cgroups subsystem, the command is:

cgset -r parameter=value path_to_cgroup

When you need to delete a cgroups node, you can use the cgdelete command. For example, to delete the above test node, you can use the command:

cgdelete -r cpu:test

Add process to child node

There are also many ways to add a process to a cgroups child node. You can write the pid directly into the task file under the child node. You can also add a process through cgclassify, the instruction is:

cgclassify -g subsystems:path_to_cgroup pidlist

You can also directly use cgexec to start a process under a certain cgroups, the command is:

gexec -g subsystems:path_to_cgroup command arguments