|
1
2
3
4
5
|
top - 16:15:05 up 6 days, 6:25, 2 users, load average: 1.45, 1.77, 2.14
Tasks: 147 total, 1 running, 146 sleeping, 0 stopped, 0 zombie
Cpu (s): 0.2% us, 0.2% sy, 0.0% ni, 86.9% id, 12.6% wa, 0.0% hi, 0.0% si
Mem: 4037872k total, 4003648k used, 34224k free, 5512k buffers
Swap: 7164948k total, 629192k used, 6535756k free, 3511184k cached
|
|
1
2
3
4
5
6
|
avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.25 33.46 66.29
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 1122 17.00 9.00 192.00 9216.00 96.00 4608.00 123.79 137.23 1033.43 13.17 100.10
sdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
|
|
1
|
vmstat 1
|
Location of sources with high IO load
Foreword:
In general operation and maintenance work, we often encounter such a scenario. The IO load of the server is very high (util in iostat), but the source process and source file of the IO load cannot be quickly located, resulting in the inability to implement corresponding strategies to solve the problem. .
This phenomenon is more common on MySQL. Before 5.6 (performance_schema provides io instrument), we can usually only guess that the high IO caused by MySQL is caused, but it is impossible to locate the load caused by which file.
For example, is ibdata flashing? Or the random read of unpopular ibd?
This article will introduce a relatively simple process for locating high IO load.
Tool preparation:
iotop: http://guichaz.free.fr/iotop/
pt-ioprofile:http://www.percona.com/downloads/percona-toolkit/2.2.1/
Step1: iostat to view the IO situation
iostat -x 1 Check the IO situation. From the figure below, we can see that the IO load of the dfa disk is relatively high. Next, we will locate the specific load source.
Step2: iotop locates the load source process
The essence of iotop is a python script that obtains thread IO information from proc and summarizes it.
As can be seen from the figure below, most of the IO sources come from the mysqld process. Therefore, it can be determined that the load source of dfa is the database
Step3 pt-ioprofile locates the load source file
The principle of pt-ioprofile is to attach a strace process to a pid for IO analysis.
The following is a warning from the official website:
However, it works by attaching strace to the process using ptrace(), which will make it run very slowly until strace detaches. In addition to freezing the server, there is also some risk of the process crashing or performing badly after strace detaches from it, or indeed of strace not detaching cleanly and leaving the process in a sleeping state. As a result, this should be considered an intrusive tool, and should not be used on production servers unless you are comfortable with that.
Use ps aux|grep mysqld to find the process number corresponding to the mysqld process, and use pt-ioprofile to check which file takes the most IO time.
By default, the tool displays the time occupied by IO.
More useful for localization problems is to locate by IO throughput. Use the parameter --cell=sizes, which displays the result in B/s
As can be seen from the above figure, the main source of IO load is sbtest (sysbench's IO bound OLTP test).
And the pressure is mostly on reading.
Foreword:
In general operation and maintenance work, we often encounter such a scenario. The IO load of the server is very high (util in iostat), but the source process and source file of the IO load cannot be quickly located, resulting in the inability to implement corresponding strategies to solve the problem. .
This phenomenon is more common on MySQL. Before 5.6 (performance_schema provides io instrument), we can usually only guess that the high IO caused by MySQL is caused, but it is impossible to locate the load caused by which file.
For example, is ibdata flashing? Or the random read of unpopular ibd?
This article will introduce a relatively simple process for locating high IO load.
Tool preparation:
iotop: http://guichaz.free.fr/iotop/
pt-ioprofile:http://www.percona.com/downloads/percona-toolkit/2.2.1/
Step1: iostat to view the IO situation
iostat -x 1 Check the IO situation. From the figure below, we can see that the IO load of the dfa disk is relatively high. Next, we will locate the specific load source.
Step2: iotop locates the load source process
The essence of iotop is a python script that obtains thread IO information from proc and summarizes it.
As can be seen from the figure below, most of the IO sources come from the mysqld process. Therefore, it can be determined that the load source of dfa is the database
Step3 pt-ioprofile locates the load source file
The principle of pt-ioprofile is to attach a strace process to a pid for IO analysis.
The following is a warning from the official website:
However, it works by attaching strace to the process using ptrace(), which will make it run very slowly until strace detaches. In addition to freezing the server, there is also some risk of the process crashing or performing badly after strace detaches from it, or indeed of strace not detaching cleanly and leaving the process in a sleeping state. As a result, this should be considered an intrusive tool, and should not be used on production servers unless you are comfortable with that.
Use ps aux|grep mysqld to find the process number corresponding to the mysqld process, and use pt-ioprofile to check which file takes the most IO time.
By default, the tool displays the time occupied by IO.
More useful for localization problems is to locate by IO throughput. Use the parameter --cell=sizes, which displays the result in B/s
As can be seen from the above figure, the main source of IO load is sbtest (sysbench's IO bound OLTP test).
And the pressure is mostly on reading.