An online server alarms, disk utilization disk.util> 90, and keeps alarming. After logging in to the server, use iostat -x 1 10 to view related disk usage information.
View related disk usage information through iostat -x 1 10
The relevant screenshots are as follows:
1 # If there is no iostat command , use yum install sysstat to install 2 # iostat -x 1 10
As can be seen from the above figure, the %util [IO] of the vdb disk is almost 100%, which is caused by frequent data reading.
Other field description
Device: Device name tps: The number of IO read and write requests per second. Multiple logical requests can be combined into a single I/O request to the device. Blk_read/s (kB_read/s, MB_read/s): The amount of data read from the device, expressed in blocks (kilobytes, megabytes) per second. A block is equivalent to a sector, so the block size is 512 bytes. Blk_wrtn/s (kB_wrtn/s, MB_wrtn/s): The amount of data written to the device, expressed in blocks (kilobytes, megabytes) per second. A block is equivalent to a sector, so the block size is 512 bytes. Blk_read (kB_read, MB_read): The total number of read blocks (kilobytes, megabytes). Blk_wrtn (kB_wrtn, MB_wrtn): The total number of written blocks (kilobytes, megabytes). rrqm/s: The number of read requests merged into the device per second. That is, delta(rmerge)/s wrqm/s: The number of write requests merged into the device per second. That is, delta(wmerge)/s r/s: The number of read I/O devices completed per second. That is, delta(rio)/s w/s: The number of write I/0 devices completed per second. That is, delta(wio)/s rsec/s (rkB/s, rMB/s): The number of sectors (kilobytes, megabytes) read from the device per second. The size of each sector is 512 bytes wsec/s (wkB/s, wMB/s): the number of sectors (kilobytes, megabytes) written to the device per second. The size of each sector is 512 bytes avgrq-sz: the average amount of data per device I/O operation (in sectors). That is, delta(rsec+wsec)/delta(rio+wio) avgqu-sz: The average length of the I/O queue sent to the device each time. await: The average waiting time for each IO request. (Including waiting queue time and processing time, in milliseconds) r_await: The average wait time for each IO read request. (Including waiting queue time and processing time, in milliseconds) w_await: average waiting time for each IO write request. (Including waiting queue time and processing time, in milliseconds) svctm: average processing time (in milliseconds) of each device I/O operation. caveat! Don't trust the value of this field anymore, this field will be removed in a future version of sysstat. %util: What percentage of a second is used for I/O operations, or how much time in a second the I/O queue is not empty. When the value is close to 100%, device saturation occurs.
Find the process with high IO usage
Via iotop
If there is no such command, please install it through yum install iotop.
# iotop -oP
Through this command, you can see more detailed information, such as: process number, disk read volume, disk write volume, IO percentage, what is the command involved "both are the large IO read volume caused by the grep command".
Via pidstat
1 # Meaning of the command: display I/O statistics, updated every second 2 # pidstat -d 1
It can be seen that the grep command occupies a lot of read IO, and then the relevant process information can be viewed according to the PID.