Linux common performance analysis tools and methods

View system load uptime

[root@localhost ~]# uptime
 10:18:27 up 6 days, 17:24,  3 users,  load average: 0.00, 0.01, 0.05

The current time, how long the system has been running, how many users have logged in, and the average load of the system in the past 1 minute, 5 minutes, and 15 minutes. The best value for load average is 1, which means that each process can be executed immediately without missing CPU cycles. Single-core CPU, 1-2 is normal, multi-core CPU, the number of cores is n, then n-2n is also normal.

free View free memory and used memory

[root@kafka3 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           7703        2318        3750         281        1635        4802
Swap:          7935           0        7935

Similar to top, htop, htop can be regarded as an enhanced extension of top, but additional installation is required

vmstat

[root@localhost ~]# vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 582132   1260 2738456    0    0     1     2   30   15  0  0 100  0  0
 0  0      0 582116   1260 2738456    0    0     0     0   65  144  0  0 100  0  0
 0  0      0 582116   1260 2738456    0    0     0     0   49  115  0  0 100  0  0
 0  0      0 582116   1260 2738456    0    0     0     0   69  149  0  1 100  0  0
 0  0      0 582116   1260 2738456    0    0     0     0   55  126  0  0 100  0  0
 [root@localhost ~]# vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 580708 748036 2106624    0    0     1     2   30   16  0  0 100  0  0

The meaning of vmstat 15 is to execute once every 1 second, and it is executed 5 times in total.
There are five columns:

procs
- r The number of runnable processes (while running or waiting to run). Show how many processes are waiting for the CPU
- b The number of processes in an uninterruptible sleep state. How many processes are waiting for io
memory
- swpd virtual memory size
- free free memory size
- buff buffer size
- cache cache size
- inact the amount of memory that is not activated
- The amount of memory currently used by active
swap
- si swap in size
- size swapped out so much of the two values 0, if the block is more than ten, when high performance is certainly reduced concurrency
io reflects io efficiency
- bi the number of blocks received from the block device
- bo The number of blocks sent to the block device
system is usually the usage of system functions
- in the number of interrupts per second, including the clock
- cs the number of context switches per second
The following four parameters of cpu add up to 100%, which is a breakdown of cpu usage
- us Non-kernel code running time
- sy kernel code running time
- id free time
- wa io waiting time
- st The time stolen from the virtual machine.
  Using vmstat, we can clearly know the status of the CPU, IO devices, and memory. Provide a basis for us to find system bottlenecks.

mpstat view cpu information

[root@localhost ~]# mpstat -P ALL 1 2
Linux 3.10.0-327.el7.x86_64 (localhost.localdomain) 	01/11/2021 	_x86_64_	(2 CPU)

10:51:17 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
10:51:18 AM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:51:18 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:51:18 AM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

10:51:18 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
10:51:19 AM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:51:19 AM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:51:19 AM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
Average:       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

There are two CPUs as shown above, and mpstat will output their indicators separately

% usr User mode process CPU usage percentage
% nice specifies the priority of the user mode process CPU usage ratio
% sys CPU usage percentage of kernel mode process (excluding software and hardware interrupts)
% iowait During the period of outstanding disk I/O requests, the percentage of CPU idle
% irq CPU service hardware interrupt usage percentage
% soft CPU service software interrupt usage percentage
% steal virtual CPU or the percentage of involuntary waiting time spent by the CPU
% guest CPU running virtual processor percentage
% The percentage of time that gnice CPU runs a nice gurst
% idle CPU idle ratio

The statistics of pidstat Linux task are similar to top

[root@localhost ~]# pidstat 1 3
Linux 3.10.0-327.el7.x86_64 (localhost.localdomain) 	01/11/2021 	_x86_64_	(2 CPU)

11:10:12 AM   UID       PID    %usr %system  %guest    %CPU   CPU  Command
11:10:13 AM     0     11997    0.98    0.98    0.00    1.96     0  pidstat

11:10:13 AM   UID       PID    %usr %system  %guest    %CPU   CPU  Command
11:10:14 AM     0     11997    1.00    2.00    0.00    3.00     0  pidstat

11:10:14 AM   UID       PID    %usr %system  %guest    %CPU   CPU  Command
11:10:15 AM     0     11997    0.00    0.99    0.00    0.99     0  pidstat

Average:      UID       PID    %usr %system  %guest    %CPU   CPU  Command
Average:        0     11997    0.66    1.32    0.00    1.98     -  pidstat

Statistical analysis of iostat IO devices (usually disks, partitions)

[root@kafka3 ~]# iostat -xz 1 5
Linux 3.10.0-1127.19.1.el7.x86_64 (kafka3.sd.cn) 	01/10/2021 	_x86_64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.35    0.00    0.20    0.00    0.00   99.44

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.02    0.01    1.12     0.22     5.35     9.88     0.01    8.58    7.13    8.59   7.58   0.85
sdb               0.00     0.00    0.00    0.00     0.00     0.00    58.72     0.00    0.33    0.33    0.00   0.14   0.00
dm-0              0.00     0.00    0.01    0.87     0.20     5.35    12.67     0.01   11.39    7.11   11.44   9.75   0.85
dm-1              0.00     0.00    0.00    0.00     0.00     0.00    50.09     0.00   14.48   14.48    0.00  11.82   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00    48.19     0.00    0.56    0.56    0.00   0.30   0.00
dm-3              0.00     0.00    0.00    0.00     0.00     0.00    48.19     0.00    0.40    0.40    0.00   0.23   0.00
dm-4              0.00     0.00    0.00    0.00     0.00     0.00    48.19     0.00    0.42    0.42    0.00   0.30   0.00
dm-5              0.00     0.00    0.00    0.00     0.00     0.00    83.98     0.00   13.35   16.40    7.48  11.57   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.25    0.50    0.00   99.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    1.00    3.00     8.00    12.00    10.00     0.04    9.25   14.00    7.67   9.25   3.70
dm-0              0.00     0.00    1.00    2.00     8.00    12.00    13.33     0.04   12.33   14.00   11.50  12.33   3.70

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.00    0.00    0.00   99.75

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.25    0.00    0.00   99.50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.00    0.00    0.00   99.75

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

There are a lot of parameters, you can refer to the man manual for details, here are the important points. Parameters such as
rrqm/s wrqm/s
mean the number of combined reads (r) and writes (r) per second. That is to say how many logical requests are combined into one request to write to the actual disk
r/sw/s rkB/s wkB/s
read and write requests sent to the device per second, sector.
avgrq-sz avgqu-sz The average (avg) size (sz) sent to the device. qu represents the request queue, rq represents the size of the request, and the unit is a sector.
await r_await w_await The waiting time sent to the io device, in milliseconds.
7. sar network detection
7.1 check network card throughput

[root@localhost ~]# sar -n DEV 1 3
Linux 3.10.0-327.el7.x86_64 (localhost.localdomain) 	01/11/2021 	_x86_64_	(2 CPU)

11:33:07 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
11:33:08 AM     ens32      2.00      0.00      0.14      0.00      0.00      0.00      0.00
11:33:08 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:33:08 AM virbr0-nic      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:33:08 AM    virbr0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:33:08 AM     ens33      2.00      0.00      0.14      0.00      0.00      0.00      0.00

11:33:08 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
11:33:09 AM     ens32      2.00      1.00      0.12      0.65      0.00      0.00      0.00
11:33:09 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:33:09 AM virbr0-nic      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:33:09 AM    virbr0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:33:09 AM     ens33      1.00      0.00      0.06      0.00      0.00      0.00      0.00

11:33:09 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
11:33:10 AM     ens32      1.00      1.00      0.06      0.65      0.00      0.00      0.00
11:33:10 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:33:10 AM virbr0-nic      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:33:10 AM    virbr0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:33:10 AM     ens33      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:        ens32      1.67      0.67      0.10      0.43      0.00      0.00      0.00
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:    virbr0-nic      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:       virbr0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:        ens33      1.00      0.00      0.07      0.00      0.00      0.00      0.00

Rxpck/s: Total number of packets received per second.
txpck/s: Total number of packets transmitted per second.
rxcmp/s txcmp/s cm means compressed compression
7.2

[root@localhost ~]#  sar -n TCP 1 3
Linux 3.10.0-327.el7.x86_64 (localhost.localdomain) 	01/11/2021 	_x86_64_	(2 CPU)

11:39:10 AM  active/s passive/s    iseg/s    oseg/s
11:39:11 AM      0.00      0.00      0.00      0.00
11:39:12 AM      0.00      0.00      1.00      1.00
11:39:13 AM      0.00      0.00      1.00      1.00
Average:         0.00      0.00      0.67      0.67
[root@localhost ~]#  sar -n TCP,ETCP 1 3
Linux 3.10.0-327.el7.x86_64 (localhost.localdomain) 	01/11/2021 	_x86_64_	(2 CPU)

11:40:04 AM  active/s passive/s    iseg/s    oseg/s
11:40:05 AM      0.00      0.00      1.00      0.00

11:40:04 AM  atmptf/s  estres/s retrans/s isegerr/s   orsts/s
11:40:05 AM      0.00      0.00      0.00      0.00      0.00

11:40:05 AM  active/s passive/s    iseg/s    oseg/s
11:40:06 AM      0.00      0.00      1.00      1.00

11:40:05 AM  atmptf/s  estres/s retrans/s isegerr/s   orsts/s
11:40:06 AM      0.00      0.00      0.00      0.00      0.00

11:40:06 AM  active/s passive/s    iseg/s    oseg/s
11:40:07 AM      0.00      0.00      1.00      1.00

11:40:06 AM  atmptf/s  estres/s retrans/s isegerr/s   orsts/s
11:40:07 AM      0.00      0.00      0.00      0.00      0.00

Average:     active/s passive/s    iseg/s    oseg/s
Average:         0.00      0.00      1.00      0.67

Average:     atmptf/s  estres/s retrans/s isegerr/s   orsts/s
Average:         0.00      0.00      0.00      0.00      0.00

Three key indicators

active/s: The number of times TCP connections have made a direct transition to the SYN-SENT state from the CLOSED state per second [tcpActiveOpens]. It means the number of TCP connections initiated by this machine/s
passive/s： The number of times TCP connections have made a direct transition to the SYN-RCVD state from the LISTEN state per second [tcpPassiveOpens]. 远程发起的TCP连接数/s
retrans/s: The total number of segments retransmitted per second-that is, the number of TCP segments transmitted containing one or more previously transmitted octets [tcpRetransSegs]. A large number of message retransmissions/s
means packet loss.

netstat
Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships
are mainly used to detect network status, which may not be directly related to "performance", but it can also help us find out a lot of information. For example, what is the current program with more TIME_WAIT.

[root@localhost ~]# netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      16308/mysqld        
tcp        0      0 0.0.0.0:1234            0.0.0.0:*               LISTEN      1584/distccd        
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      2823/dnsmasq        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1577/sshd           
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      1580/cupsd          
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2235/master         
tcp        0      0 10.10.154.22:22        10.10.157.52:51734     ESTABLISHED 27116/sshd: root@no 
tcp        0      0 10.10.154.22:22        10.10.157.52:56956     ESTABLISHED 7862/sshd: root@pts 
tcp        0      0 10.10.154.22:3306      10.10.154.148:58896    ESTABLISHED 16308/mysqld        
tcp6       0      0 :::22                   :::*                    LISTEN      1577/sshd           
tcp6       0      0 ::1:631                 :::*                    LISTEN      1580/cupsd          
tcp6       0      0 ::1:25                  :::*                    LISTEN      2235/master

Common parameters are as follows

-t						tcp
-a, --all                display all sockets (default: connected)
-n, --numeric            don't resolve names
-p, --programs           display PID/Program name for sockets
-l, --listening          display listening server sockets
``


参考  
[0] https://baike.baidu.com/item/uptime/8818329?fr=aladdin  
[1] https://blog.csdn.net/tencent_teg/article/details/106561235  
[2] https://linux.die.net/man/1/sar  
[3] https://linux.die.net/man/1/iostat  
[4] https://linux.die.net/man/1/pmstat
[5] 高性能MySQL

Linux common performance analysis tools and methods

Guess you like