Out of interest in the Linux operating system and a strong desire for underlying knowledge, this article has been compiled. This article can also be used as an indicator to test basic knowledge, and the article covers all aspects of a system. Without sound computer system knowledge, network knowledge and operating system knowledge, it is impossible to fully grasp the tools in the documentation. In addition, system performance analysis and optimization is a long-term series.
This document is mainly combined with the blog post of Linux expert Brendan Gregg, senior performance architect of Netflix, who updated the Linux performance tuning tool, and collected articles related to Linux system performance optimization. It is mainly combined with the principles involved in the blog post and performance testing tools.
Background knowledge: Having background knowledge is what you need to know when analyzing performance issues. Such as hardware cache; another example is the operating system kernel. The behavior details of the application are often intertwined with these things. These low-level things will affect the performance of the application in unexpected ways. For example, some programs cannot make full use of the cache, resulting in performance degradation. For example, calling too many system calls unnecessarily, causing frequent kernel/user switching, etc.
performance analysis tool
First look at a picture:
The picture above is a performance analysis shared by Brendan Gregg. All the tools in it can be used to man obtain its help documents. The following is a brief introduction to the conventional usage:
vmstat--virtual memory statistics
vmstat (VirtualMeomoryStatistics, virtual memory statistics) is a common tool for monitoring memory in Linux, which can monitor the overall situation of the virtual memory, process, CPU, etc. of the operating system. Common usage of vmstat: that is, sampling once vmstat interval times every second, a total of sampling times, if omitted , the data will be collected until the user manually stops. Simply give an example:intervaltimestimes
ctrl+c Data collection can be stopped using vmstat .
The first line shows the average value of the system since it was started, the second line starts to show what is happening now, the next line will show what happened every 5 seconds interval, the meaning of each column is in the header, as follows:
-
procs: r This column shows how many processes are waiting for cpu, and column b shows how many processes are sleeping uninterruptibly (waiting for IO).
-
The memory:swapd column shows how many blocks have been swapped out of disk (page swapping), the remaining columns show how many blocks are free (unused), how many blocks are being used as buffers, and how many are being used Operating system cache.
-
swap: Shows swap activity: how many blocks are being swapped in (from disk) and swapped out (to disk) per second.
-
io: Shows how many blocks were read (bi) and written (bo) from the block device, usually reflecting hard disk I/O.
-
system: Displays the number of interrupts (in) and context switches (cs) per second.
-
cpu: Displays the percentage of all cpu time spent on various operations, including executing user code (non-kernel), executing system code (kernel), idle and waiting for IO.
Insufficient memory performance: free memory decreases sharply, recycling buffer and cache does not help, extensive use of swap partition (swpd), frequent page swapping (swap), increase in the number of read and write disks (io), and page fault interrupts (in). The number of switching (cs) increases, the number of processes (b) waiting for IO increases, and a large amount of CPU time is used to wait for IO (wa)
iostat -- for reporting CPU statistics
iostat is used to report central processing unit (CPU) statistics and input/output statistics for the entire system, adapters, tty devices, disks, and CD-ROMs. By default, it displays the same cpu usage information as vmstat. Use the following command to display extended Device Statistics:
The first row shows the average since system boot, and then the incremental average, one row per device.
The abbreviation habit of common linux disk IO indicators: rq is request, r is read, w is write, qu is queue, sz is size, a is verage, tm is time, svc is service.
-
rrqm/s and wrqm/s: Combined read and write requests per second, "combined" means that the operating system takes multiple logical requests from the queue and merges them into one request to the actual disk.
-
r/s and w/s: The number of read and write requests sent to the device per second.
-
rsec/s and wsec/s: The number of sectors read and written per second.
-
avgrq –sz: number of requested sectors.
-
avgqu –sz: The number of requests waiting in the device queue.
-
await: The time spent on each IO request.
-
svctm: actual request (service) time.
-
%util: Percentage of time spent with at least one active request.
dstat--system monitoring tool
dstat shows the cpu usage, disk io, network packet sending and page changing. The output is colorful and readable. Compared with the input of vmstat and iostat, it is more detailed and intuitive. When in use, just enter the command directly, of course you can also use specific parameters.
as follows:dstat –cdlmnpsy
iotop--LINUX process real-time monitoring tool
The iotop command is a special command to display hard disk IO. The interface style is similar to the top command, and it can display which process generates the IO load. It is a top tool used to monitor disk I/O usage. It has a UI similar to top, including PID, user, I/O, process and other related information.
Can be used non-interactively:
iotop –bod interval
To view each process I/O, you can use
pidstat，pidstat –d instat
pidstat--monitor system resources
pidstat is mainly used to monitor the occupation of system resources by all or specified processes, such as CPU, memory, device IO, task switching, threads, etc.
Instructions:
pidstat –d interval
pidstat can also be used to count CPU usage information:
pidstat –u interval
Statistical memory information:
pidstat –r interval
top
-
The summary area of the top command displays five areas of system performance information:
-
Load: time, number of logged-in users, average system load;
-
process: run, sleep, stop, zombie;
-
cpu: user state, core state, NICE, idle, waiting for IO, interrupt, etc.;
-
Memory: total, used, free (system perspective), buffer, cache;
-
Swap partition: total, used, free
The task area displays by default: process ID, effective user, process priority, NICE value, virtual memory, physical memory and shared memory used by the process, process status, CPU usage, memory usage, cumulative CPU time, and process command line information.
htop
htop is an interactive process viewer for Linux systems, a text-mode application (in console or X terminal), requires ncurses.
Htop allows users to operate interactively, supports color themes, can scroll through the process list horizontally or vertically, and supports mouse operation.
Compared with top, htop has the following advantages:
-
You can scroll through the process list horizontally or vertically to see all processes and complete command lines.
-
On startup, it is faster than top.
-
You don't need to enter the process number when killing the process.
-
htop supports mouse operation.
mpstat
mpstat is the abbreviation of Multiprocessor Statistics, which is a real-time system monitoring tool. It reports some statistics about the CPU, which are stored in /proc/stat files. In a multi-CPUs system, it can not only view the average status information of all CPUs, but also view the information of specific CPUs. Common usage:
mpstat –P ALL interval times
netstat
netstat is used to display statistical data related to IP, TCP, UDP and ICMP protocols, and is generally used to check the network connection status of each port of the machine.
Common usage:
netstat –npl # 可以查看你要打开的端口是否已经打开。
netstat –rn # 打印路由表信息。
netstat –in # 提供系统上的接口信息，打印每个接口的MTU,输入分组数，输入错误，输出分组数，输出错误，冲突以及当前的输出队列的长度。
ps - display the status of the current process
There are too many ps parameters, please refer to the specific usage method man ps
Commonly used methods:
ps aux #hsserver
ps –ef |grep #hundsun
To kill a program:
ps aux | grep mysqld | grep –v grep | awk ‘{print $2 }’ xargs kill -9
Kill zombie process:
ps –eal | awk ‘{if ($2 == “Z”){print $4}}’ | xargs kill -9
strace
Track the system calls and received signals generated during program execution to help analyze abnormal conditions encountered during program or command execution.
Example: To check which configuration file mysqld loads on linux, you can run the following command:
strace –e stat64 mysqld –print –defaults > /dev/null
uptime
It can print the total running time of the system and the average load of the system. The meanings of the last three numbers output by the uptime command are the average load of the system within 1 minute, 5 minutes, and 15 minutes, respectively.
lsof
lsof (list open files) is a tool to list the open files of the current system. Through the lsof tool, you can view this list to detect and troubleshoot the system. Common usage:
View file system blocking
lsof /boot
Check which process the port number is occupied by
lsof -i : 3306
See what files users have open
lsof –u username
See what files a process has open
lsof –p 4838
View remote open network links
lsof –i @192.168.34.128
perf
perf is a system performance optimization tool that comes with the Linux kernel. The advantage lies in the close integration with the Linux Kernel. It can be firstly applied to new features added to the Kernel to check hot functions and cash miss ratios, thereby helping developers optimize program performance.
The basic principle of performance tuning tools such as perf, Oprofile, etc. is to sample the monitored object. The simplest case is to sample according to the tick interrupt, that is, trigger the sampling point within the tick interrupt, and judge the current context of the program in the sampling point .
If a program spends 90% of its time in function foo(), then 90% of the sampling points should fall in the context of function foo(). Luck is unpredictable, but I think as long as the sampling frequency is high enough and the sampling time is long enough, then the above inferences are more reliable. Therefore, by triggering sampling by tick, we can understand which places in the program consume the most time, and then focus on analysis.
summary
Combining the above commonly used performance test commands and contacting the performance analysis tool diagram in the early part of the article, you can initially understand which aspect of the performance in the performance analysis process uses which aspect of the tool (command).
Commonly used performance testing tools
Proficient and proficient in the performance analysis command tool in the second part, introduce several performance testing tools, and briefly understand several performance testing tools before introducing:
-
perf_events: A performance diagnostic tool released and maintained with the Linux kernel code, maintained and developed by the kernel community. Perf can be used not only for performance statistics analysis of applications, but also for performance statistics and analysis of kernel code.
-
eBPF tools: A performance tracking tool using bcc, eBPF map can be widely used in kernel tuning using custom eBPF programs, and can also read user-level asynchronous code. The important thing is that this external data can be managed in user space. The map data body in kv format is managed by invoking bpf system calls in user space to create, add, delete and other operations.
-
perf-tools: A set of Linux performance analysis and tuning tools based on perf_events (perf) and ftrace. Perf-Tools has few dependent libraries and is easy to use. Support Linux 3.2 and above kernel versions.
-
bcc(BPF Compiler Collection): A perf performance analysis tool using eBPF. A toolkit for creating efficient kernel tracing and manipulation programs, including several useful tools and examples. Take advantage of Extended BPF (Berkeley Packet Filter), formally known as eBPF, a new feature that was first added to Linux 3.15. Multi-purpose requires Linux 4.1 or above BCC.
-
ktap: A new dynamic performance tracking tool for linux scripts. Allows users to track Linux kernel dynamics. ktap is designed to be interoperable, allowing users to tune operational insights, troubleshoot and extend kernels and applications. It is similar to Linux and Solaris DTrace SystemTap.
-
Flame Graphs: is a graphical software that uses perf, system tap, and ktap visualizations, allowing the most frequent code paths to be identified quickly and accurately, and can be the program generation of the development
github.com/brendangregg/flamegraphsource code in use.
Linux observability tools | Linux observability tools
-
The Basic Tools to learn first are as follows: uptime, top (htop), mpstat, isstat, vmstat, free, ping, nicstat, dstat.
-
Advanced commands are as follows: sar, netstat, pidstat, strace, tcpdump, blktrace, iotop, slabtop, sysctl, /proc.
Linux benchmarking tools | Linux benchmarking tools
It is a performance evaluation tool. You can use corresponding tools for performance testing of different modules. If you want to know more about it, you can refer to the attached document below.
Linux tuning tools | Linux performance tuning tools
It is a performance tuning tool, which is mainly tuned from the source code layer of the linux kernel. If you want to know more about it, you can refer to the attached document below.
Linux observability sar | Linux performance observation tool
sar (System Activity Reporter) is one of the most comprehensive system performance analysis tools on LINUX at present. It can report system activities from various aspects, including: file read and write, system call usage, disk I/O, CPU efficiency, memory usage, process activity, and IPC-related activities. Common usage of sar:
sar [options] [-A] [-o file] t [n]
Among them: t is the sampling interval, n is the number of sampling, the default value is 1; -o file means to store the command result in a file in binary format, and file is the file name. options are command line options
Link: https://rdc.hundsun.com/portal/article/731.html