1. First, let's figure out what in the end is ftrace
Its author: Steven Rostedt
For ftrace authors say,
ftrace is a tool to track the internal operation of the kernel, there are several categories tracked mentioned here
a static tracker Static tracepoints at The Kernel kernel of the WITHIN (Event Tracing)
①scheduling (scheduling)
②interrupts (interrupt)
③file Systems (File System )
④virtual the Guest connections with host (connecting virtual users and hosts)
Second, the dynamic kernel function trace (dynamic kernel function tracing)
①trace All functions the WITHIN at the kernel (track kernel all functions)
②pick and the choose the What functions to the trace (pick to track function)
③call Graphs (function call graph)
④stack Usage (use the function stack ???)
Third, the delay tracker (latency tracers)
①how Long interrupts are Disabled
②how Long preemption (seize) iS Disabled
③how Long interrupts and / or preemption is disabled
Fourth, the wake-up delay (the wake up latency)
①how IT Takes A Long Process to the After RUN IT IS Woken
Wikipedia definition given is this
ftrace (abbreviated from Function Tracer) is a tracing framework for the Linux kernel. Although its original name, Function Tracer, came from ftrace’s ability to record information related to various function calls performed while the kernel is running, ftrace’s tracing capabilities cover a much broader range of kernel’s internal operations.
With its various tracer plugins, ftrace can be targeted at different static tracepoints, such as scheduling events, interrupts, memory-mapped I/O, CPU power state transitions, and operations related to file systems and virtualization. Also, dynamic tracking of kernel function calls is available, optionally restrictable to a subset of functions by using globs, and with the possibility to generate call graphs and provide stack usage reports. At the same time, ftrace can be used to measure various latencies within the Linux kernel, such as for how long interrupts or preemption are disabled.
An ftrace-enabled Linux kernel is built by enabling the CONFIG_FUNCTION_TRACER kernel configuration option. The entire runtime interaction with ftrace is performed through readable and writable virtual files contained in a specifically mounted debugfs file system; as a result, ftrace requires no specialized userspace utilities to operate. However, there are additional userspace utilities that provide more advanced features for data recording, analysis and visualization; examples of such utilities are trace-cmd and KernelShark.
Internally, ftrace relies on the gcc’s profiling mechanism to prepend machine instructions to the compiled versions of all source-level kernel functions, which redirect the execution of functions to the ftrace’s trampolines and tracer plugins that perform the actual tracing. These “entry point” instructions created by gcc are altered by ftrace when the kernel is booted, and varied later at runtime by ftrace between NOPs and actual jumps to the tracing trampolines, depending on the tracing types and options configured at runtime.
ftrace is developed primarily by Steven Rostedt, and it was merged into the Linux kernel mainline in kernel version 2.6.27, which was released on October 9, 2008
2. and Let's look at how it is applied
Usually, we use linux system kernel has done its support for ftrace, so we are here to talk about how to use the linux system ftrace
ftrace provide access interface to the user mode by debugfs, debugfs will create a directory / sys / kernel / debug, there is a tracing directory under the directory in which to store the ftrace service interface document provided to the user access
- README file inside this directory file a brief introduction, self-interested view it
- current_tracer used to set or display the track currently in use; echo command writes the file name of the tracker may be switched to a different tracker. After the system starts, the default value is NOP, i.e. without any tracking operation. After completion of the implementation of a tracking task, the tracker may be reset by writing to the file nop.
- available_tracers records the list of currently compiled into the kernel tracker, you can view its contents by cat; a list of names tracker used when writing current_tracer tracker file name must be listed in the file.
- trace file provides an interface to view the acquired trace information. See the file commands to view with cat tracked core activity, its contents may be saved as a log file for subsequent viewing.
- tracing_on used to control the pause tracking. Sometimes I think when certain events observed temporarily closed track, the file can be written to 0 to stop the trace, trace buffer in such relatively new part is related to an event of interest; write 1 may continue to track.
- set_graph_function set to clearly show the relationship of function calls, the display of information structure similar to the C language code, so that in the analysis of the core operation of the process will be more intuitive. Use when using function_graph tracker; default sequence generates call relation to all function, the function of particular concern can be specified by writing to the file as needed.
- buffer_size_kb for setting a single track buffer size used by the CPU. Tracker will trace information is written to the cache, trace cache per CPU is as big. Trace cache implemented in the form of a ring buffer, if the track too much information, the old information is overwritten by new trace information. Note that to change the value of the file need to first set current_tracer nop can.
- available_filter_functions record the current kernel functions that can be tracked. For functions that are not listed in the file can not track their activities.
- set_ftrace_filter and set_ftrace_notrace when compiling the kernel configuration using dynamic ftrace after (check CONFIG_DYNAMIC_FTRACE option). The former is used to display the specified function to be traced, the latter opposite effects, for the specified function is not tracking. If a function name also appears in both files, the status of implementation of this function will not be tracked. These documents also support the simple expression of the form contains wildcards, such an expression may be used to target multiple objective functions;
In the opening chapter, we quote the definition of the author and a description of Wikipedia to understand ftrace is doing, it keeps track of what things, and now we have to practice it and see the current system which provides a tracker bar
root@ubuntu:/sys/kernel/debug/tracing# cat available_tracers
hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt wakeup function nop
- nop tracking does not track any core activities that will write current_tracer nop file tracker before it can be used to delete, and empty the previously collected trace information that is refreshed trace file.
- function tracker may track the execution of kernel function; function specifies to be tracked can be displayed in the file set_ftrace_filter.
- function_graph tracker may show similar source C function call graph, so that it is intuitive to see some of the; calling functions specified to be generated by the flowchart can display the file set_grapch_function.
ftrace also supports a number of other trackers such as sched_switch tracker can track the process of scheduling events in the kernel, irqsoff tracker and preemptoff tracker were tracking code is Off interrupt code and prohibit the process of preemption, and record the maximum time off, preemptirqsoff the tracker can be seen as a combination thereof.
ftrace framework supports extensions to add a new tracker. Readers can refer to the kernel source package Documentation / documentation and source files in the kernel / trace in the trace directory to see other trackers use and how to add a new tracker.
Use fucntion tracker
function tracker can track calls the kernel function can be used for debugging or analyzing bug, can also be used to understand and observe the process of implementation of the Linux kernel. Example uses a tracker function is given below.
root@ubuntu:/sys/kernel/debug/tracing# pwd
/sys/kernel/debug/tracing
root@ubuntu:/sys/kernel/debug/tracing# echo function > current_tracer
root@ubuntu:/sys/kernel/debug/tracing# echo 1 > tracing_on
# 在这里等待一段时间，
root@ubuntu:/sys/kernel/debug/tracing# echo 0 > tracing_on
root@ubuntu:/sys/kernel/debug/tracing# cat trace | head -20
# tracer: function
#
# entries-in-buffer/entries-written: 205147/8183627 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
<...>-38381 [000] d... 10529.718505: __update_load_avg_cfs_rq <-update_blocked_averages
<...>-38381 [000] d... 10529.718505: __accumulate_pelt_segments <-__update_load_avg_cfs_rq
<...>-38381 [000] d... 10529.718506: __update_load_avg_cfs_rq <-update_blocked_averages
<...>-38381 [000] d... 10529.718506: __accumulate_pelt_segments <-__update_load_avg_cfs_rq
<...>-38381 [000] d... 10529.718507: __update_load_avg_cfs_rq <-update_blocked_averages
<...>-38381 [000] d... 10529.718507: __accumulate_pelt_segments <-__update_load_avg_cfs_rq
<...>-38381 [000] d... 10529.718508: __update_load_avg_cfs_rq <-update_blocked_averages
<...>-38381 [000] d... 10529.718509: __accumulate_pelt_segments <-__update_load_avg_cfs_rq
<...>-38381 [000] d... 10529.718509: __update_load_avg_cfs_rq <-update_blocked_averages
<idle>-0 [000] d... 10818.707183: quiet_vmstat <-tick_nohz_idle_stop_tick
<idle>-0 [000] d... 10818.707184: need_update <-quiet_vmstat
<idle>-0 [000] d... 10818.707184: first_online_pgdat <-need_update
<idle>-0 [000] d... 10818.707186: next_zone <-need_update
<idle>-0 [000] d... 10818.707187: next_zone <-need_update
<idle>-0 [000] d... 10818.707187: next_zone <-need_update
<idle>-0 [000] d... 10818.707187: next_zone <-need_update
<idle>-0 [000] d... 10818.707188: next_zone <-need_update
<idle>-0 [000] d... 10818.707188: next_online_pgdat <-next_zone
root@ubuntu:/sys/kernel/debug/tracing#
File format trace information given very clear. First of all, the field "tracer:" gives the name of the tracker currently used, here function tracker. Then the tracking information recording format, the name of the corresponding task TASK field, PID field is given the progress of the task ID, field CPU # CPU number indicates the operating tracked function, where you can see the number of idle processes running at 0 CPU which process ID is 0; tIMESTAMP is the time stamp field, the format "secs.usecs", the corresponding time stamp indicating when executing the function; the fUNCTION a given function is being traced, the calling function by the symbol " <- "indicate, this relationship can be observed calling function.
https://lwn.net/Articles/365835/
https://lwn.net/Articles/366796/
