sar (System Activity Reporter system activity report) is one of the most comprehensive system performance analysis tools on Linux . It can report system activities from many aspects, including: file read and write, system call usage, disk I/O , CPU efficiency, memory usage, process activity, and IPC-related activities. This article mainly takes the CentOS 6.3 x64 system as an example to introduce the sar command.
Common format of sar command
sar [options] [-A] [-o file] t [n]
in:
t is the sampling interval, n is the number of sampling times, and the default value is 1;
-o file means to store the command result in a file in binary format, where file is the file name.
options is a command line option. The common options of the sar command are as follows:
-A: sum of all reports
-u: output statistics of CPU usage
-v: output statistics for inodes, files, and other kernel tables
-d: output activity information for each block device
-r: output statistics of memory and swap space
-b: Display I/O and transfer rate statistics
-a: file read and write conditions
-c: output process statistics, the number of processes created per second
-R: output statistics of memory pages
-y: terminal device activity
-w: output system exchange activity information
1. CPU resource monitoring
For example, to sample once every 10 seconds, continuously sample 3 times, observe the CPU usage, and store the sampling results in binary form in the file test in the current directory, you need to type the following command:
sar -u -o test 10 3
The screen displays as follows:
17:06:16 CPU %user %nice %system %iowait %steal %idle
17:06:26 all 0.00 0.00 0.20 0.00 0.00 99.80
17:06:36 all 0.00 0.00 0.20 0.00 0.00 99.80
17:06:46 all 0.00 0.00 0.10 0.00 0.00 99.90
Average: all 0.00 0.00 0.17 0.00 0.00 99.83
Description of output items:
CPU:all means that the statistics are the average of all CPUs.
%user: Displays the percentage of total CPU time spent running at the user level (application).
%nice: Displayed at the user level, used for nice operations, the percentage of the total CPU time occupied.
%system: The percentage of the total CPU time used to run at the core level (kernel).
%iowait: Displays the percentage of total CPU time spent waiting for I/O operations.
%steal: Percentage of virtual CPUs waiting for the hypervisor to serve another virtual process.
%idle: Displays the percentage of CPU idle time occupied by the total CPU time.
1. If the value of %iowait is too high, it means that the hard disk has an I/O bottleneck
2. If the value of %idle is high but the system response is slow, it is possible that the CPU is waiting to allocate memory, and the memory capacity should be increased.
3. If the value of %idle is continuously lower than 1, the CPU processing power of the system is relatively low, indicating that the most resource that needs to be solved in the system is the CPU.
If you want to view the contents of the binary file test, you need to type the following sar command:
sar -u -f test
2. Inode, file and other kernel table monitoring
For example, to sample once every 10 seconds and continuously sample 3 times to observe the status of the core table, you need to type the following command:
sar -v 10 3
The screen displays as follows:
17:10:49 dentunusd file-nr inode-nr pty-nr
17:10:59 6301 5664 12037 4
17:11:09 6301 5664 12037 4
17:11:19 6301 5664 12037 4
Average: 6301 5664 12037 4
Description of output items:
dentunusd: Number of unused entries in the directory cache
file-nr: the number of file handles used
inode-nr: the number of inode handles used
pty-nr: number of ptys used
3. Memory and swap space monitoring
For example, to sample every 10 seconds, 3 consecutive samples, and monitor memory paging:
sar -r 10 3
The screen displays as follows:
Description of output items:
kbmemfree: This value is basically the same as the free value in the free command, so it does not include buffer and cache space.
kbmemused: This value is basically the same as the used value in the free command, so it includes buffer and cache space.
%memused: This value is a percentage of kbmemused and total memory (excluding swap).
kbbuffers and kbcached: These two values are the buffer and cache in the free command.
kbcommit: Guarantees the memory required by the current system, that is, the memory (RAM+swap) required to ensure that it does not overflow.
%commit: This value is a percentage of kbcommit and total memory (including swap).
4. Memory paging monitoring
For example, to sample every 10 seconds, 3 consecutive samples, and monitor memory paging:
sar -B 10 3
The screen displays as follows:
Description of output items:
pgpgin/s: Indicates the number of bytes (KB) replaced from disk or SWAP to memory per second
pgpgout/s: Indicates the number of bytes (KB) replaced from memory to disk or SWAP per second
fault/s: The number of page faults generated by the system per second, that is, the sum of the main page fault and the secondary page fault (major + minor)
majflt/s: The number of main page faults generated per second.
pgfree/s: the number of pages placed in the free queue per second
pgscank/s: the number of pages scanned by kswapd per second
pgscand/s: the number of pages scanned directly per second
pgsteal/s: The number of pages flushed from the cache to satisfy memory requirements per second
%vmeff: Percentage of pages cleared per second (pgsteal) to total scanned pages (pgscank+pgscand)
5. I/O and transfer rate monitoring
For example, to sample every 10 seconds and continuously sample 3 times to report the usage of the buffer, you need to type the following command:
sar -b 10 3
The screen displays as follows:
18:51:05 tps rtps wtps bread/s bwrtn/s
18:51:15 0.00 0.00 0.00 0.00 0.00
18:51:25 1.92 0.00 1.92 0.00 22.65
18:51:35 0.00 0.00 0.00 0.00 0.00
Average: 0.64 0.00 0.64 0.00 7.59
Description of output items:
tps: Total I/O transfers per second to the physical device
rtps: The total amount of data read from the physical device per second
wtps: The total amount of data written to the physical device per second
bread/s: The amount of data read from the physical device per second, in blocks/s
bwrtn/s: The amount of data written to the physical device per second, in blocks/s
6. Process queue length and average load status monitoring
For example, sampling every 10 seconds, 3 consecutive samplings, and monitoring the process queue length and average load status:
sar -q 10 3
The screen displays as follows:
19:25:50 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
19:26:00 0 259 0.00 0.00 0.00
19:26:10 0 259 0.00 0.00 0.00
19:26:20 0 259 0.00 0.00 0.00
Average: 0 259 0.00 0.00 0.00
Description of output items:
runq-sz: length of the run queue (number of processes waiting to run)
plist-sz: the number of processes and threads in the process list
ldavg-1: System load average for the last minute
ldavg-5: System load average over the past 5 minutes
ldavg-15: System load average over the past 15 minutes
7. System exchange activity information monitoring
For example, to sample every 10 seconds, 3 consecutive samples, the monitoring system exchanges activity information:
sar - W 10 3
The screen displays as follows:
19:39:50 pswpin / s pswpout / s
19:40:00 0.00 0.00
19:40:10 0.00 0.00
19:40:20 0.00 0.00
Average: 0.00 0.00
Description of output items:
pswpin/s: The number of swap pages swapped in by the system per second
pswpout/s: The number of swap pages (swap pages) swapped out by the system per second
8. Device usage monitoring
For example, to sample once every 10 seconds, continuously sample 3 times, and report the device usage, you need to type the following command:
# sar -d 10 3 –p
The screen displays as follows:
17:45:54 DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util
17:46:04 scd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
17:46:04 sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
17:46:04 vg_livedvd-lv_root 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
17:46:04 vg_livedvd-lv_swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
in:
The parameter -p can print out the names of disk devices such as sda, hdc, etc. If the parameter -p is not used, the device node may be dev8-0, dev22-0
tps: The number of I/Os from the physical disk per second. Multiple logical requests will be combined into one I/O disk request, and the size of a transfer is indeterminate.
rd_sec/s: The number of read sectors per second.
wr_sec/s: The number of write sectors per second.
avgrq-sz: Average data size (sectors) per device I/O operation.
avgqu-sz: The average length of the disk request queue.
await: The average consumption time of each request, including the waiting time of the request queue, from the time the disk operation is requested to the completion of the processing by the system, in milliseconds (1 second = 1000 milliseconds).
svctm: The average time the system takes to process each request, excluding time spent in the request queue.
%util: The percentage of I/O requests to CPU, the larger the ratio, the more saturated it is.
1. When the value of avgqu-sz is low, the utilization of the device is high.
2. When the value of %util is close to 1%, it means that the device bandwidth is full.
To determine system bottlenecks, it is sometimes necessary to combine several sar command options
It is suspected that there is a bottleneck in the CPU, which can be viewed with sar -u and sar -q, etc.
It is suspected that there is a bottleneck in memory, which can be viewed by sar -B, sar -r and sar -W, etc.
Suspected that there is a bottleneck in I/O, you can use sar -b, sar -u and sar -d to check