在RHEL7上 使用 Intel NVMe+LVM+XFS时大块读的性能问题

RHEL7: Large block read performance issue with XFS+LVM with Intel NVMe

https://access.redhat.com/solutions/3406851

 SOLUTION 已验证 - 已更新 2019年二月28日23:06 - 

English 

环境

• Red Hat Enterprise Linux 7
• XFS on a LVM Logical Volume
• NVMe storage

问题

• Large block reads from a XFS file system backed by a LVM Logical Volume on a Intel NVMe device are under performing.
• This behavior is not observed on a XFS file system directly on the NVMe device.
• Reads from a EXT4 file system on a Logical Volume do not exhibit this behavior.

决议

Red Hat Enterprise Linux 7

• This issue has been resolved with the errata RHSA-2018:3083 for the package(s) kernel-3.10.0-957.el7 or later.

根源

• File systems were not being created with optimal strip width at time of creation. This is because # mkfs.xfs was not receiving accurate information from opt_io hint.
• chunk_sectors are now properly reported to io_min and io_opt.
• This now resolves the performance degradation.

诊断步骤

• Results from testing

Raw

XFS+LVM:

   READ: bw=319MiB/s (335MB/s), 319MiB/s-319MiB/s (335MB/s-335MB/s), io=37.4GiB (40.2GB), run=120001-120001msec
   READ: bw=321MiB/s (336MB/s), 321MiB/s-321MiB/s (336MB/s-336MB/s), io=37.6GiB (40.3GB), run=120001-120001msec

XFS+LVM+MQ

# cat /sys/module/dm_mod/parameters/use_blk_mq 
Y

   READ: bw=321MiB/s (337MB/s), 321MiB/s-321MiB/s (337MB/s-337MB/s), io=37.6GiB (40.4GB), run=120001-120001msec


XFS:  Without LVM

   READ: bw=1178MiB/s (1235MB/s), 1178MiB/s-1178MiB/s (1235MB/s-1235MB/s), io=138GiB (148GB), run=120001-120001msec
   READ: bw=1067MiB/s (1119MB/s), 1067MiB/s-1067MiB/s (1119MB/s-1119MB/s), io=125GiB (134GB), run=120001-120001msec

EXT4: Without LVM

   READ: bw=1156MiB/s (1212MB/s), 1156MiB/s-1156MiB/s (1212MB/s-1212MB/s), io=135GiB (145GB), run=120001-120001msec

EXT4+LVM:

   READ: bw=1026MiB/s (1076MB/s), 1026MiB/s-1026MiB/s (1076MB/s-1076MB/s), io=120GiB (129GB), run=120002-120002msec