Introduction to linux tracing and profiling

Various trace debugging tools are often used in work, such as strace, ltrace, kprobes, tracepoints, uprobes, ftrace, perf, and eBPF. What is the relationship between them? This article provides an overview in general, and welcomes corrections if there are errors.

1.1.1 System Tracking

System traces can be divided into data sources, data collection mechanisms, and trace front-ends (interactions that collect and analyze data).

Data sources can be divided into probes and tracepoints, the corresponding sources are:

probes :kprobes/uprobes

tracepoints :USDT/kernel tracepoints / lttng-ust

The probe can be modified at runtime to enable tracing. The tracepoint is compiled into the program and can be enabled or activated when it is used. Using the tracepoint will not cause any loss when it is not activated, and it also has a small overhead in the activated state.

1.1.1.1 kprobes

kprobes is the debug mechanism of the linux kernel and can also be used to monitor events in production systems. It can also be used to find performance bottlenecks, specify events, and track problems.

Gregg's can be used

https://github.com/kernel-z/perf-tools/blob/master/kernel/kprobe

To track the printing of files in the system, you can use the following:

./kprobe 'p:myopen do_sys_open filename=+0(%si):string'

You can print the system call open in the system.

The applicable scenarios of kprobes are: 1. Tracking system calls, which have the corresponding kernel function do_sys_open; 2. When knowing which kernel functions are called, locate network protocol stack or file IO performance problems; 3. Kernel developers are used to locate the kernel question.

1.1.1.2 uprobes

Uprobes are similar to kprobes, mainly to detect user-mode functions, such as malloc.

Specific reference:

http://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.html

1.1.1.3 USDT/dtrace probes

USDT is user-mode statically defined traces, which is dtrace probe.

If the program compiles dtrace probes, it can be consumed using tools such as eBPF/systemtap. Of course, many programs such as python do not compile dtrace probes by default. If they are compiled, they can be used to trace python function calls.

1.1.1.4 tracepoints

Tracepoints are also in the kernel. Compared to kprobes, the changes are relatively small.

1.1.1.5 lttng-para

lttng-ust is a tracking system that can compile probes into programs, and all tracking events occur in user mode. It's fast because no context switch is required.

1.1.2 Data Collection Mechanism

In order to understand the collection and analysis of trace data, it is important to understand how to get data out of the kernel and into your hands. Observe several parts that come with the kernel.

1.1.2.1 ftrace

ftrace is more difficult to use directly. A lot of data is located in /sys/kernel/debug/tracing. To interact with ftrace, you can read and write files within it.

1.1.2.2 perf_events

Use the system call perf_event_open to get data from the kernel. The kernel writes events to user-mode memory, which can be read directly.

1.1.2.3 eBPF

Write the eBPF program (usually in C, or use a tool to generate the program), and then have the kernel attach the probe to the kprobe/uprobe/tracepoint/dtrace probe. Then the program will write the data to the eBPF cache to get the accurate data.

eBFP is available on the latest kernel version.

1.1.1 tracepoint

Finally, look at the tracepoint.

There are functions in the form of trace_XX in the kernel. These are the tracepoints of the kernel, which are defined in include/linux/tracepoint.h.

For trace_ to work, you need to call register_trace_##name, associate it with a probe function, and execute the probe function when trace_ is called

Each tracepoint in the kernel provides a hook to call the probe function. A tracepoint can be turned on or off. When open, the probe function is associated with the tracepoint ; when it is closed, the probe function is not associated with the tracepoint . When the tracepoint is closed, the impact on the kernel is very small, only a very small time overhead (a branch condition judgment) and a very small space overhead (a function call statement and several data structures ) are added. When a tracepoint is opened, the user-provided probe function is called every time the tracepoint executes.

1.1.2 About tracker selection

If your current or future computer is running a kernel greater than 4.9, then use eBPF, but in older versions eBPF may not help you, so ftrace is worth investing in analysis.

Perf trace is relatively simple, and the loss is relatively low, so you can get started directly.

Using kprobes is also a good idea.

You can use perf_events (aka perf) to do CPU profiling, which can then be described by a flame graph. Of course, perf can also do a lot of things, here it is to do CPU profiling first.

Performance guru Gregg has the following advice:

If you are a performance engineer, you need to choose a tracer such as SystemTap, LTTng or sysdig. Compared with LTTng, SystemTap is more powerful. sysdig needs to add kprobe or tracepoints.

Use perf and ftrace as much as possible at work, which has been integrated into eBPF, and then supplemented with SystemTap or LTTng.