一.云硬盘的性能衡量指标
云硬盘的性能指标一般通过以下几个指标进行衡量
- IOPS：每秒的读写次数，单位为次（计数）。存储设备的底层驱动类型决定了不同的IOPS
总IOPS：每秒执行的I/O操作总次数
随机读IOPS：每秒指定的随机读I/O操作的平均次数
随机写IOPS 每秒指定的随机写I/O操作的平均次数
顺序读IOPS 每秒指定的顺序读I/O操作的平均次数
顺序写IOPS 每秒指定的顺序写I/O操作的平均次数 - 吞吐量：每秒的读写数据量，单位为MB/S
吞吐量市值单位时间内可以成功传输的数据数量。
如果需要部署大量顺序读写的应用，典型场景比如hadoop离线计算型业务，需要关注吞吐量 -
时延：IO操作的发送时间到接收确认所经过的时间，单位为秒
如果应用对时延比较敏感，比如数据库（过高时延会导致应用性能下降或报错），建议使用SSD存储二.具体参数说明
[root@host-10-0-1-36 ~]# fio --bs=4k --ioengine=libaio --iodepth=1 --direct=1 --rw=read --time_based --runtime=600 --refill_buffers --norandommap --randrepeat=0 --group_reporting --name=fio-read --size=100G --filename=/dev/vdb
fio-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=9784KiB/s,w=0KiB/s][r=2446,w=0 IOPS][eta 00m:00s]
fio-read: (groupid=0, jobs=1): err= 0: pid=22004: Wed Oct 10 21:35:42 2018
read: IOPS=2593, BW=10.1MiB/s (10.6MB/s)(6078MiB/600001msec)
slat (usec): min=4, max=1532, avg=11.98, stdev= 8.10
clat (nsec): min=1021, max=66079k, avg=370367.39, stdev=395393.29
lat (usec): min=44, max=66086, avg=382.88, stdev=399.21
clat percentiles (usec):
| 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 46],
| 30.00th=[ 48], 40.00th=[ 51], 50.00th=[ 383], 60.00th=[ 578],
| 70.00th=[ 644], 80.00th=[ 701], 90.00th=[ 783], 95.00th=[ 865],
| 99.00th=[ 988], 99.50th=[ 1020], 99.90th=[ 1336], 99.95th=[ 2057],
| 99.99th=[11338]
bw ( KiB/s): min= 2000, max=68272, per=99.98%, avg=10370.55, stdev=8123.28, samples=1199
iops : min= 500, max=17068, avg=2592.62, stdev=2030.82, samples=1199
lat (usec) : 2=0.01%, 20=0.01%, 50=37.97%, 100=10.63%, 250=0.39%
lat (usec) : 500=6.09%, 750=31.54%, 1000=12.65%
lat (msec) : 2=0.67%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=1.43%, sys=5.12%, ctx=1555984, majf=0, minf=35
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=1556003,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
READ: bw=10.1MiB/s (10.6MB/s), 10.1MiB/s-10.1MiB/s (10.6MB/s-10.6MB/s), io=6078MiB (6373MB), run=600001-600001msec
Disk stats (read/write):
vdb: ios=1555778/0, merge=0/0, ticks=570518/0, in_queue=569945, util=95.04%| 参数详情 | 含义 |
|---|---|
| io | 执行了多少M的IO |
| bw | 平均IO带宽 |
| iops | IOPS |
| runt | 线程运行时间 |
| slat | 提交延迟 |
| clat | 完成延迟 |
| lat | 响应时间 |
| cpu | cpu利用率 |
| IO depths | io队列 |
| IO submit | 单个IO提交要提交的IO数 |
| IO complete | |
| IO issued | |
| IO latency |
三.FIO测试工具参数说明
[root@host-10-0-1-36 ~]# fio -direct=1 -iodepth=64 -rw=read -ioengine=libaio -bs=4096k -size=100G -numjobs=1 -runtime=300 -group_reporting -filename=/dev/vdb -name=Write_PPS_Testing| 参数 | 说明 |
|---|---|
| -direct=1 | 表示测试时忽略I/O缓存，数据直写 |
| -iodepth=128 | 表示使用AIO时，同时发出I/O数的上限为128 |
| -rw=randwrite | 表示测试时的读写策略为随机写（random writes）。作其它测试时可以设置为：randread（随机读random reads）read（顺序读sequential reads） write（顺序写sequential writes）randrw（混合随机读写mixed random reads and writes） |
| -ioengine=libaio | 表示测试方式为libaio（Linux AIO，异步I/O）。应用程序使用I/O通常有两种方式： 同步：同步的I/O一次只能发出一个I/O请求，等待内核完成才返回。这样对于单个线程iodepth总是小于1，但是可以透过多个线程并发执行来解决。通常会用16−32根线程同时工作将iodepth塞满。异步：异步的I/O通常使用libaio这样的方式一次提交一批I/O请求，然后等待一批的完成，减少交互的次数，会更有效率。 |
| -bs=4k | 表示单次I/O的块文件大小为4 KB。未指定该参数时的默认大小也是4 KB，测试IOPS时，建议将bs设置为一个比较小的值，如本示例中的4k。测试吞吐量时，建议将bs设置为一个较大的值，如本示例中的1024k |
| -size=1G | 表示测试文件大小为1 GiB |
| -numjobs=1 | 表示测试线程数为1 |
| -runtime=1000 | 表示测试时间为1000秒。如果未配置，则持续将前述-size指定大小的文件，以每次-bs值为分块大小写完 |
| -group_reporting | 表示测试结果里汇总每个进程的统计信息，而非以不同job汇总展示信息 |
| -filename=iotest | 指定测试文件的名称，比如iotest。测试裸盘可以获得真实的硬盘性能，但直接测试裸盘会破坏文件系统结构，请在测试前提前做好数据备份 |
| -name=Rand_Write_Testing | 表示测试任务名称为Rand_Write_Testing，可以随意设定 |
四.测试结果
这里我们以4K的数据块测试随机读和随机写来测试最大的IOPS，4M的块测试顺序读和顺序写来测试最大的吞吐量
- 测试4K块的随机写
[root@host-10-0-1-36 ~]# fio --bs=4k --ioengine=libaio --iodepth=128 --direct=1 --rw=randwrite --time_based --runtime=300 --refill_buffers --norandommap --randrepeat=0 --group_reporting --name=fio-write --size=1G --filename=/dev/vdb
fio-write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=3996KiB/s][r=0,w=999 IOPS][eta 00m:00s]
fio-write: (groupid=0, jobs=1): err= 0: pid=22050: Wed Oct 10 22:29:32 2018
write: IOPS=2819, BW=11.0MiB/s (11.5MB/s)(3309MiB/300484msec)
slat (usec): min=2, max=2399, avg= 9.54, stdev=10.19
clat (usec): min=1180, max=3604.0k, avg=45387.25, stdev=168013.09
lat (usec): min=1201, max=3604.0k, avg=45397.35, stdev=168013.57
clat percentiles (usec):
| 1.00th=[ 1713], 5.00th=[ 2212], 10.00th=[ 2835],
| 20.00th=[ 4015], 30.00th=[ 5211], 40.00th=[ 6849],
| 50.00th=[ 8979], 60.00th=[ 11994], 70.00th=[ 17695],
| 80.00th=[ 33162], 90.00th=[ 61604], 95.00th=[ 137364],
| 99.00th=[ 893387], 99.50th=[1266680], 99.90th=[2122318],
| 99.95th=[2432697], 99.99th=[2969568]
bw ( KiB/s): min= 8, max=49120, per=100.00%, avg=11603.11, stdev=10950.79, samples=584
iops : min= 2, max=12280, avg=2900.77, stdev=2737.69, samples=584
lat (msec) : 2=3.16%, 4=16.75%, 10=33.83%, 20=18.49%, 50=14.82%
lat (msec) : 100=6.56%, 250=2.86%, 500=1.49%, 750=0.70%, 1000=0.53%
lat (msec) : 2000=0.69%, >=2000=0.12%
cpu : usr=1.84%, sys=4.16%, ctx=323739, majf=0, minf=28
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwt: total=0,847133,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
WRITE: bw=11.0MiB/s (11.5MB/s), 11.0MiB/s-11.0MiB/s (11.5MB/s-11.5MB/s), io=3309MiB (3470MB), run=300484-300484msec
Disk stats (read/write):
vdb: ios=91/847074, merge=0/0, ticks=3/38321566, in_queue=38360706, util=100.00%
我们可以看到4K的数据块，随机写的最大IOPS为：12280
- 测试4K的随机读
[root@host-10-0-1-36 ~]# fio --bs=4k --ioengine=libaio --iodepth=128 --direct=1 --rw=randread --time_based --runtime=300 --refill_buffers --norandommap --randrepeat=0 --group_reporting --name=fio-write --size=1G --filename=/dev/vdb
fio-write: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=87.1MiB/s,w=0KiB/s][r=22.3k,w=0 IOPS][eta 00m:00s]
fio-write: (groupid=0, jobs=1): err= 0: pid=22055: Wed Oct 10 22:51:54 2018
read: IOPS=22.3k, BW=87.0MiB/s (91.2MB/s)(25.5GiB/300004msec)
slat (usec): min=2, max=8626, avg= 7.91, stdev=11.85
clat (usec): min=36, max=71810, avg=5735.17, stdev=1405.20
lat (usec): min=45, max=71826, avg=5743.59, stdev=1405.30
clat percentiles (usec):
| 1.00th=[ 1958], 5.00th=[ 3556], 10.00th=[ 4424], 20.00th=[ 5145],
| 30.00th=[ 5407], 40.00th=[ 5604], 50.00th=[ 5735], 60.00th=[ 5866],
| 70.00th=[ 6063], 80.00th=[ 6259], 90.00th=[ 6783], 95.00th=[ 7504],
| 99.00th=[10290], 99.50th=[12256], 99.90th=[16712], 99.95th=[18482],
| 99.99th=[25035]
bw ( KiB/s): min=74872, max=93240, per=100.00%, avg=89121.48, stdev=2687.55, samples=600
iops : min=18718, max=23310, avg=22280.35, stdev=671.89, samples=600
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.02%, 750=0.06%
lat (usec) : 1000=0.12%
lat (msec) : 2=0.86%, 4=6.02%, 10=91.77%, 20=1.11%, 50=0.03%
lat (msec) : 100=0.01%
cpu : usr=6.56%, sys=28.57%, ctx=3473136, majf=0, minf=160
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwt: total=6683408,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=128
Run status group 0 (all jobs):
READ: bw=87.0MiB/s (91.2MB/s), 87.0MiB/s-87.0MiB/s (91.2MB/s-91.2MB/s), io=25.5GiB (27.4GB), run=300004-300004msec
Disk stats (read/write):
vdb: ios=6680955/0, merge=0/0, ticks=37981396/0, in_queue=37983491, util=100.00%我们可以看到4K的数据块，随机读的最大IOPS为：23310
- 测试4M的顺序写
[root@host-10-0-1-36 ~]# fio -direct=1 -iodepth=64 -rw=write -ioengine=libaio -bs=4096k -size=100G -numjobs=1 -runtime=300 -group_reporting -filename=/dev/vdb -name=Write_PPS_Testing
Write_PPS_Testing: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=108MiB/s][r=0,w=27 IOPS][eta 00m:00s]
Write_PPS_Testing: (groupid=0, jobs=1): err= 0: pid=22098: Wed Oct 10 23:20:07 2018
write: IOPS=28, BW=115MiB/s (120MB/s)(33.7GiB/300466msec)
slat (usec): min=564, max=2554.1k, avg=34767.87, stdev=65067.32
clat (msec): min=400, max=12179, avg=2188.04, stdev=1037.46
lat (msec): min=473, max=12231, avg=2222.81, stdev=1047.45
clat percentiles (msec):
| 1.00th=[ 1435], 5.00th=[ 1586], 10.00th=[ 1653], 20.00th=[ 1754],
| 30.00th=[ 1838], 40.00th=[ 1921], 50.00th=[ 2005], 60.00th=[ 2089],
| 70.00th=[ 2198], 80.00th=[ 2333], 90.00th=[ 2534], 95.00th=[ 2802],
| 99.00th=[ 8490], 99.50th=[ 9731], 99.90th=[11745], 99.95th=[12013],
| 99.99th=[12147]
bw ( KiB/s): min= 8192, max=196608, per=100.00%, avg=120954.04, stdev=31456.39, samples=580
iops : min= 2, max= 48, avg=29.53, stdev= 7.68, samples=580
lat (msec) : 500=0.02%, 750=0.03%, 1000=0.13%, 2000=49.25%, >=2000=50.56%
cpu : usr=1.04%, sys=2.16%, ctx=6387, majf=0, minf=29
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.3%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwt: total=0,8627,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=115MiB/s (120MB/s), 115MiB/s-115MiB/s (120MB/s-120MB/s), io=33.7GiB (36.2GB), run=300466-300466msec
Disk stats (read/write):
vdb: ios=91/77596, merge=0/0, ticks=6/37685137, in_queue=37717833, util=99.99%我们可以看到4M的数据块，顺序写的最大吞吐量为：196M
- 测试4M的顺序读
[root@host-10-0-1-36 ~]# fio -direct=1 -iodepth=64 -rw=read -ioengine=libaio -bs=4096k -size=100G -numjobs=1 -runtime=300 -group_reporting -filename=/dev/vdb -name=Write_PPS_Testing
Write_PPS_Testing: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=312MiB/s,w=0KiB/s][r=78,w=0 IOPS][eta 00m:00s]
Write_PPS_Testing: (groupid=0, jobs=1): err= 0: pid=22103: Wed Oct 10 23:26:31 2018
read: IOPS=35, BW=142MiB/s (149MB/s)(41.6GiB/300201msec)
slat (usec): min=469, max=95190, avg=28166.40, stdev=15966.53
clat (msec): min=185, max=4070, avg=1772.98, stdev=551.21
lat (msec): min=205, max=4107, avg=1801.15, stdev=558.25
clat percentiles (msec):
| 1.00th=[ 518], 5.00th=[ 634], 10.00th=[ 751], 20.00th=[ 1536],
| 30.00th=[ 1770], 40.00th=[ 1854], 50.00th=[ 1921], 60.00th=[ 1989],
| 70.00th=[ 2039], 80.00th=[ 2140], 90.00th=[ 2299], 95.00th=[ 2433],
| 99.00th=[ 2802], 99.50th=[ 2970], 99.90th=[ 3473], 99.95th=[ 3641],
| 99.99th=[ 3775]
bw ( KiB/s): min=106496, max=466944, per=99.97%, avg=145216.28, stdev=60136.24, samples=597
iops : min= 26, max= 114, avg=35.45, stdev=14.68, samples=597
lat (msec) : 250=0.10%, 500=0.66%, 750=9.21%, 1000=7.02%, 2000=46.51%
lat (msec) : >=2000=36.51%
cpu : usr=0.05%, sys=2.61%, ctx=10959, majf=0, minf=672
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.4%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwt: total=10646,0,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=41.6GiB (44.7GB), run=300201-300201msec
Disk stats (read/write):
vdb: ios=95740/0, merge=0/0, ticks=37360788/0, in_queue=37382226, util=100.00%我们可以看到4M的数据块，顺序读的最大吞吐量为：466M