Introduction: The emergence of eBPF technology makes kernel resource monitoring more convenient and efficient, and containerized monitoring is more suitable for cloud-native scenarios. Based on the observability achieved by eBPF, it is possible to safely and efficiently expand kernel functions without modifying the kernel source code or loading kernel modules, which solves the above problems well. This article will introduce how to implement container-level TCP connection monitoring based on eBPF from the perspective of the network.
1. Technical background
OpenCloudOS is suitable for large-scale container cluster server scenarios. In order to improve server resource utilization and reduce service and customer server costs, a resource isolation solution for hybrid deployment of off-line and online services is proposed. The resource isolation QoS (Quality of Service) solution mainly includes four modules: CPU, I/O, memory and network.
Performing QoS on these four server resources can deploy users' online and offline services on the same server. While ensuring the service quality of online services, it can effectively improve resource utilization and help reduce costs and increase efficiency. Continuous development concept. With the application of large-scale hybrid deployment of offline and online services in OpenCloudOS, how to better monitor and feedback service status in real time has become an urgent problem for business operation and maintenance personnel.
Based on the observability implemented by the Linux kernel, it has the advantages of good performance and high flexibility. However, kernel-based implementations are usually difficult and cumbersome. In the traditional implementation method, developers need to modify the kernel source code to recompile or load the kernel module to achieve performance observation, which may deal with complex and difficult to debug situations, making performance observation extremely difficult.
Tools such as ftrace and perf based on technologies such as kprobe, tracepoint, and perf events move data from the kernel state to the user state for processing by piling in the kernel function. The disadvantages brought about by the above methods of achieving observability are also obvious. Unprotected kernel modules have great security risks. At the same time, the cost of implementation is relatively high, and it cannot cope with changing scenarios.
The emergence of eBPF technology makes kernel resource monitoring more convenient and efficient, and containerized monitoring is more suitable for cloud-native scenarios. Based on the observability achieved by eBPF, it is possible to safely and efficiently expand kernel functions without modifying the kernel source code or loading kernel modules, which solves the above problems well. This article will introduce how to implement container-level TCP connection monitoring based on eBPF from the perspective of the network.
2. Introduction to eBPF
eBPF is a technology for running sandbox programs in the Linux kernel. It can safely and efficiently extend kernel functions without modifying kernel source code or loading kernel modules. It can be regarded as a virtual system that executes user code on some hook points in the kernel machine.
The code written by the user is compiled into bytecode by clang and loaded into the linux kernel. The security of the bytecode is guaranteed by the verifier engine, and then the bytecode is converted into local machine code by the embedded JIT compiler. eBPF is triggered by events. When an event arrives, the user-mode BPF program is executed in the kernel space to change the processing flow of the kernel.
After eBPF was introduced into the kernel, it gradually evolved into a general-purpose execution engine, based on which performance analysis tools, network filters, etc. can be developed. The eBPF program architecture emphasizes security and stability, looks like a kernel module, but does not require recompilation of the kernel.
During the running of the eBPF program, the system will not crash due to program problems. Due to its security, efficiency, programmability, and advantages in speed and performance, eBPF has played an important role in kernel observation, trace monitoring, application performance tuning, flow control, performance statistics, security and other fields.
3. Problems in the practical application of eBPF tools
With the large-scale deployment and use of offline and online hybrid functions, more businesses in practical applications are isolated and scheduled based on containers. Machine-based system-level network monitoring and tracking tools cannot implement container-level control, and are not suitable for cloud-native container scenarios.
How to implement more detailed cgroup and process-level resource monitoring and help administrators better understand the resource usage of the system has become an issue of widespread concern. Therefore, the containerization of monitoring tools is particularly important. OpenCloudOS transmits the subsystem state ID of the cgroup level to the user space, and establishes the connection between the path of the cgroup subsystem and the ID. At the same time, the adaptation hides the structural differences between cgroup v1 and v2, so that the kernel only collects the network connection status information of specific cgroups in the required subsystem. Based on this, the containerized monitoring of BCC and libbpf network tools are realized respectively.
4. Examples of network monitoring tools
The following example experimental environment is:
Release: OC 8.6
Kernel: TK4-5.4
Tool: BCC Network Monitoring Tool
1. tcpconnect
Monitor tcp network connection based on cgroup, display source IP, destination IP, destination port and other status information.
Count the number of tcp connections over a period of time based on cgroup.
2. tcpconnlat
Based on the cgroup monitoring tcp connection establishment time, display the connection status information.
3. tcprtt
Based on the cgroup statistics of the distribution of tcp rtt within a period of time, display the status information of the connection.
4. tcptrace
Monitor the tcp network connection based on the filter conditions, track the life cycle of the skb message in the kernel, and output the time delay, address, CPU, network port and other information of each message at each point in the protocol stack.
5. tcplife
Track the life cycle of tcp connection based on cgroup, and display statistics such as connection survival time.
6. tcpdrop
Monitor tcp network connections based on cgroup, track the data packets dropped by the kernel, and display information such as data packet addresses, ports, and call stacks.
Reference link :
BPF Documentation： https://www.infradead.org/~mchehab/kernel_docs/bpf/index.html BPF Portability and CO-RE：https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html Andrii Nakryiko's Blog： https://nakryiko.com/
If you encounter technical problems when using eBPF tools or OpenCloudOS, you can scan the QR code below to join the community communication group to obtain technical support and exchange experience.