Continuing from the previous article: "BPF Performance Tools - Insights into Linux System and Application Performance" Study Notes - Chapter 1 Introduction (1)
1.2 What are Tracing, Snooping, Sampling, Profiling and Observability?
These are terms used to classify analytical techniques and tools.
- Tracing
Tracing is event-based logging - the type of instrumentation used by these BPF tools. You may already use some specialized tracing tools, such as Linux strace, to record and print system call events. There are many tools that do not track events, but measure them using fixed statistical counters and print summaries afterwards. Linux top is one example. A feature of the tracker is its ability to record raw events and event metadata. Such data can be massive and may need to be post-processed into summaries. BPF makes it possible for program tracers to run small programs on events to perform custom on-the-fly statistical summaries or other operations, thus avoiding expensive post-processing.
- Snooping
Although strace has "trace" in its name, not all tracers are like this (referring to those with "trace" in their name). For example, tcpdump is another specialized tracer of network packets. (Maybe it should be named tcptrace?). The Solaris operating system has its own version of tcpdump called snoop, so named because it is used to snoop network packets. I was the first to develop and release many tracing tools, and that was on Solaris, where I (perhaps sadly) used the term "snoop" for my early tools. That's why we now have execsnoop, opensnoop, biosnoop, etc. Snooping, event dumping and tracing usually refer to the same thing. These tools will be introduced (one by one) in the following chapters.
In addition to the tool name, the term "tracing" is also used to describe BPF, especially by kernel developers, when used for observability.
- Sampling
Sampling tools take a subset of measurements to draw a rough picture of a target; this is also called creating a profile or profile. There is a BPF tool called profile that can perform timer-based sampling of running code. For example, it can sample every 10 milliseconds, or in other words, it can sample 100 times per second (on each CPU). One advantage of samplers is that they can have a lower performance overhead than trackers because they only measure one of a much larger set of events. The disadvantage is that sampling only provides a rough picture and events may be missed.
- Observability
Observability is about understanding a system through observation and classifying the tools that enable this functionality. These tools include tracking tools, sampling tools, and fixed counter-based tools. It does not include benchmarking tools, which perform workload experiments to modify the state of the system. The BPF tools in this book are observability tools that use BPF for program tracing.