Linux performance optimization (three)-sysstat performance monitoring tool

1. Introduction to sysstat

1. Introduction to sysstat

sysstat provides a collection of Linux performance monitoring tools, including sar, sadf, mpstat, iostat, pidstat, etc., for monitoring the performance and usage of the Linux system.
iostat-Provide CPU statistics, storage I/O statistics (disk devices, partitions and network file systems)
mpstat-Provide single or combined CPU related statistics
pidstat-Provide Linux process level statistics: I/O, CPU, memory, etc.
sar-collection, Report and save system activity information: CPU, memory, disk, interrupt, network interface, TTY, kernel table, etc.
sadc-system activity data collector, use
sa1 as the sar backend -collect daily data of system activity and store it in binary format, it As the front end of sadc tools, you can call
sa2 through cron -generate daily activity reports of the system, and can also be used as the front end of sadc tools, you can call
sadf through cron -can display the performance data collected by sar in CSV, XML format, etc. , So it is very convenient to import system data into the database, or import it into Excel to generate a chart.
nfsiostat-sysstat: Provide NFS I/O statistics
cifsiostat: Provide CIFS statistics
pidstat output result %wait indicator, which represents the time the process waits for the CPU Percentage, systat 11.5.5 version only introduced new indicators.

2. sysstat installation

sysstat can be installed through yum to
yum install sysstat
download the source code sysstat-11.6.6.tar.xz installation:
http://sebastien.godard.pagesperso-orange.fr/download.html
compile option configuration:
configure
compile:
make
install:
sudo make install

Two, mpstat

1. Introduction to mpstat

mpstat is a commonly used multi-core CPU performance analysis tool, used to view the performance indicators of each CPU and the average indicators of all CPUs in real time, the data file /proc/stat.
mpstat is the abbreviation of Multiprocessor Statistics, a real-time system monitoring tool.

2. mpstat command

mpstat [-P {cpu|ALL}] [internal [count]]
-P {cpu l ALL}: indicates the monitored CPU, cpu takes the value in [0, number of cpu-1]
internal: the interval time between two adjacent samples
count: the number of times of sampling, count can only be combined with delay use

3. mpstat indicator display

CPU: Processor ID
%usr: In the internal time period, user mode CPU time, excluding processes with a negative nice value
%nice: In the internal time period, nice value is the CPU time of a negative process.
%sys: In the internal time period, the kernel is too CPU time.
%iowait: The hard disk IO wait time in the internal time period.
%irq: In the internal time period, the hard interrupt time.
%soft: soft interrupt time in the internal time period.
%steal: Shows the percentage of time the virtual CPU spends involuntarily waiting while the virtual machine manager is serving another virtual processor.
%guest: Shows the percentage of CPU time spent running virtual processors.
%idle: In the internal time period, the CPU idle time except waiting for disk IO operations and idle time for any reason.

Three, sar

1. Introduction to sar

sar (System Activity Reporter, system activity report) is one of Linux's most comprehensive system performance analysis tools, which can report system activities from many aspects, including: file read and write, system call usage, disk IO , CPU efficiency, memory usage, process activities and IPC-related activities.

2. The sar command

sar [options] [-A] [-o file] t [n]
-A: the sum of all reports
-u: output the statistics of CPU usage
-v: output the statistics of inodes, files and other kernel tables
-d: output the activity information of each block device
-r: output the memory and swap space
-B: Display statistics of IO and transfer rate
-a: File read and write status
-c: Output process statistics, the number of processes created per second
-R: Output statistics of memory pages
-y: Terminal device activity Situation-
w: output system exchange activity information

3. Common sar commands

(1) CPU resource monitoring

CPU: all means that the statistical information is the average value of all CPUs.
%user: Displays the percentage of the total CPU time used for running at the user level (application).
%nice: displayed at the user level, used for nice operations, the percentage of the total CPU time occupied.
%system: The percentage of the total CPU time used in kernel mode operation.
%iowait: Displays the percentage of the total CPU time used to wait for IO operations.
%steal: The percentage of the hypervisor waiting for the virtual CPU to provide services for another virtual process.
%idle: Displays the percentage of the total CPU time occupied by the CPU idle time.
sar -u -f test
View the contents of the binary file test
(2) File monitoring is
sampled every 10 seconds, continuously sampling 3 times, and observe the state of the core table
sar -v 10 3

dentunusd: the number of unused entries in the directory cache
file-nr: the use of file handles Quantity
inode-nr: the number of inode handles used
pty-nr: the number of pty used
(3) memory and swap space monitoring
sar -r 10 3
sampling once every 10 seconds, continuous sampling 3 times, monitoring memory paging

kbmemfree: available memory size, Does not include buffer and cache space.
kbmemused: used memory size, including buffer and cache space.
%memused: kbmemused and a percentage of total memory (excluding swap). kbbuffers:
kbcached
:
kbcommit: Guarantee the memory needed by the current system, that is, the memory (RAM+swap) needed to ensure that it does not overflow.
%commit: A percentage of kbcommit and total memory (including swap).
(4) Memory paging monitoring is
sar -B 10 3
sampled once every 10 seconds, continuous sampling 3 times, monitoring memory paging

pgpgin/s: indicates the number of bytes replaced from disk or SWAP to memory per second (KB)
pgpgout/s: indicates from memory per second The number of bytes replaced to disk or SWAP (KB)
fault/s: The number of page faults generated by the system per second, that is, the sum of major page faults and minor page faults (major + minor)
majflt/s: generated per second The number of main page faults.
pgfree/s: the number of pages put into the free queue
per second
pgscank/s: the number of pages scanned by kswapd per second pgscand/s: the number of pages directly scanned per second
pgsteal/s : The number of pages that are cleared from the cache every second to meet memory requirements
% vmeff: The percentage of pages cleared per second (pgsteal) to the total scanned pages (pgscank+pgscand)
(5) IO monitoring is
sar -b 10 3
sampled every 10 seconds, Sampling 3 times in a row, reporting the usage of the buffer

tps: total I/O transfer of the
physical device per second rtps: total data read from the physical device
per second wtps: written to the physical device per second Total data
bread/s: The amount of data read in from the physical device per second, in blocks/s
bwrtn/s: The amount of data written to the physical device per second, in blocks/s
(6) Process queue And load average monitoring
sar -q 10 3
Sampling every 10 seconds, continuous sampling 3 times, monitor the process queue length and average load status

runq-sz: the length of the run queue (the number of processes waiting to run)
plist-sz: the processes and threads in the process list The number of
ldavg-1: the system load average in the last 1 minute (System load average)
ldavg-5: the system load average in the past 5 minutes
ldavg-15: the system load average in the past 15 minutes
(7) System exchange information monitoring
sar -w 10 3
every 10 seconds Sampling once, continuously sampling 3 times, the monitoring system exchanges activity information

(8) Device usage monitoring
sar -d -p 10 3
samples once every 10 seconds, continuously sampling 3 times, reporting device usage

DEV: device name
tps: I/O from physical disk per second frequency. Multiple logical requests will be combined into one IO disk request, and the size of one transfer is uncertain.
rd_sec/s: The number of read sectors per second.
wr_sec/s: The number of write sectors per second.
avgrq-sz: The average data size (sector) of each device IO operation.
avgqu-sz: The average length of the disk request queue.
await: From requesting the disk operation to the completion of the system processing, the average consumption time of each request, including the waiting time of the request queue, in milliseconds.
svctm: The average time the system processes each request, excluding the time spent in the request queue.
%util: The percentage of IO requests to CPU. The larger the ratio, the more saturated.
(9) Network card monitoring
sar –n DEV 1 4
-n parameter options: DEV | EDEV | NFS | NFSD | SOCK | ALL. DEV displays network interface information, EDEV displays statistics about network errors, NFS client information for NFS statistics activities, NFSD statistics NFS server information, SOCK displays socket information, and ALL displays all.

IFACE: LAN interface
rxpck/s: data packets received per second
txpck/s: data packets sent per second
rxbyt/s: bytes received per second
txbyt/s: bytes sent per second
rxcmp/s: compressed data packets received every second txcmp/s: compressed data packets
sent every second
rxmcst/s: multicast data packets received every second

Four, pidstat

1. Introduction to pidstat

Pidstat is a command of sysstat, used to monitor the occupancy of system resources such as CPU, memory, thread, device IO, etc. of all or specified processes. The first sampling of Pidstat displays various statistical information since the start of the system, and subsequent sampling will display the statistical information since the last time the command was run. Users can obtain the required statistical information by specifying the number and time of statistics.

2. The pidstat command

pidstat [ 选项 ] [ <时间间隔> ] [ <次数> ]
-u: default parameter, display the CPU usage statistics of
each process -r: display the memory usage statistics of
each process -d: display the IO usage of each process
-p: specify the process number
-w: display the context switch of each process
-t: Display additional information other than the statistical information of the thread of the selected task
-T {TASK | CHILD | ALL }: Specify the content monitored by pidstat. TASK means to report an independent task, CHILD means to report all thread statistics under the process, and ALL means to report an independent task and all threads under the task.
-V: version number
-h: display all activities on one line.
-I: In the SMP environment, it indicates the CPU usage/number of cores of the task
-l: Displays the command name and all parameters

3. Common commands for pidstat

(1) View the CPU usage of all processes

pidstat
pidstat -u -p ALL

PID: Process ID
%usr: The percentage of CPU occupied by the process in user space
%system: The percentage of CPU occupied by the process in kernel space
%guest: The percentage of
CPU occupied by the process in the virtual machine %CPU: The percentage of
CPU occupied by the process CPU: The percentage of running processes CPU
Command: the command corresponding to the current process
(2) View the CPU usage of all processes.
pidstat -r

PID: process identifier
minflt/s: minor errors that occur in the task per second, no need to load pages from disk
majflt/s: task per second The main error occurred, the page needs to be loaded from the disk
VSZ: virtual address size, virtual memory usage KB
RSS: resident collection size, non-swap area Wuli memory use KB
Command: task command name
(3) View the IO of all processes Usage
pidstat -d

PID: Process id
kB_rd/s: KB read from the disk per second
kB_wr/s: KB written to the disk per second
kB_ccwr/s: KB written to the disk for which the task is cancelled. This happens when the task truncates the dirty pagecache.
COMMAND: task command name
(4) view the context switching situation of all processes
pidstat -w

PID: process id
Cswch/s: number of active task context switches
per second Nvcswch/s: number of passive task context switches per second
Command: command name
(5) view Thread statistics information of the process
pidstat -t

TGID: ID of the main thread
TID: thread id
%usr: the percentage of CPU occupied by the process in user space
%system: the percentage of CPU occupied by the process in kernel space
%guest: the percentage of
CPU occupied by the process in the virtual machine %CPU: the percentage of
CPU occupied by the process CPU: the running process CPU
Command: the command corresponding to the current process
pidstat -T ALL -p 13671

PID: process id
msr-ms: the number of milliseconds used by tasks and subthreads at the user level.
system-ms: The number of milliseconds used by tasks and subthreads at the system level.
guest-ms: The number of milliseconds used by tasks and sub-threads in a virtual machine (running a virtual processor).
Command: Command name

Five, iostat

1. Introduction to iostat

iostat is the abbreviation of IO statistics (input/output statistics). System-level IO monitoring is used to monitor system disk operation activities. It is characterized by reporting disk activity statistics and CPU usage. The disadvantage is that it cannot perform in-depth analysis of a process , Can only analyze the overall situation of the system.

2. The iostat command

iostat [ options ] [ <interval> [ <count> ] ]
-c: display only the CPU usage;
-d: display only the device utilization;
-k: display the status in kilobytes per second, not using blocks per second;
-m: display the status in megabytes Unit per second;
-p: display the status of the block device and all other partitions used;
-t: display the time when each report is generated;
-V: display the version number and exit;
-x: display the extended status.

3. iostat indicator display

%user: The percentage of time that the CPU is in user mode.
%nice: The percentage of time that the CPU is in user mode with NICE value.
%system: The percentage of time that the CPU is in system mode.
%iowait: The percentage of time the CPU waits for input and output to complete.
%steal: The percentage of unconscious wait time of the virtual CPU when the hypervisor maintains another virtual processor.
%idle: CPU idle time percentage.
tps: The number of transmissions per second of the device.
kB_read/s: the amount of data read from the device (drive expressed)
per second ; kB_wrtn/s: the amount of data written to the device (drive expressed) per second;
kB_read: the total amount of data read;
kB_wrtn: written The total amount of data;
iostat -d -x -k 1 2
the detailed information of the disk IO is output every 1 second, with a total of 2 samples.

rrqm/s: The number of times that read requests to the device are merged per second. The file system will merge requests that read the same block.
wrqm/s: The number of times that write requests to the device are merged per second
r/s: Number of reads completed per second
w/s: Number of writes completed per second
rkB/s: Volume of data read per second (in kB)
wkB/s: Volume of data written per second (in kB)
avgrq-sz: Average per second The amount of data for each IO operation (in the number of sectors)
avgqu-sz: the average queue length of IO requests waiting to be processed
await: the average waiting time for each IO request (including waiting time and processing time, in milliseconds)
svctm: average per Processing time of IO requests (in milliseconds)
%util: The ratio of time used for IO operations in the cycle, that is, the ratio of time when the IO queue is not empty

Six, vmstat

1. Introduction to vmstat

vmstat (Virtual Meomory Statistics) is a common tool for monitoring memory in Linux, which can monitor the overall situation of the operating system's virtual memory, processes, and CPU.

2, vmstat command

vmstat [options] [delay [count]]
-a: display active and inactive memory
-f: display the number of forks since the system started.
-m: display slabinfo
-n: display each field name only once at the beginning.
-s: Display memory-related statistics and the number of various system activities.
delay: refresh interval. If not specified, only one result will be displayed.
count: the number of refreshes. If you do not specify the number of refreshes, but specify the refresh interval, it will always refresh.
-d: Display statistics about the disk.
-p: Display statistics of the specified disk partition.
-S: Display in the specified unit, k, K, m, and M represent 1000, 1024, 1000000, 1048576 bytes, and the default unit is K.
-V: Display vmstat version information.
vmstat [-a] [-n] [-S unit] [delay [count]]
vmstat [-s] [-n] [-S unit]
vmstat [-m] [-n] [delay [count]]
vmstat [-d] [-n] [delay [count]]
vmstat [-p disk partition] [-n] [delay [count]]
vmstat [-f]
vmstat [-V]

vmstat [options] [delay [count]]
delay: Data sampling interval. If not specified, it returns the average value since the startup.
count: the number of refresh data. If delay is specified but connt is not specified, it will always refresh by default.
vmstat interval times
Sampling is performed every interval seconds for a total of times.

3. Common commands of vmstat

procs: The r column shows the number of processes waiting for the CPU, and the b column shows the number of processes that are in uninterruptible sleep (waiting for IO).
memory: The swapd column shows the number of blocks that are swapped out of the disk (page swap), the free column shows the number of free (unused) blocks, the buff shows the number of buffers being used, and the cache column shows the number of blocks being used. The number of operating system caches.
swap: Shows swap activity: how many blocks are being swapped in (from disk) and swapped out (to disk) per second.
io: Shows how many blocks are read (bi) and written (bo) from the block device, usually reflecting the hard disk IO.
system displays the number of interrupts (in) and context switches (cs) per second.
cpu shows the percentage of all cpu time spent in various operations, including executing user code (non-kernel), executing system code (kernel), idle and waiting for IO.