1、带宽测试
在server端执行
[ibtests]# ib_send_bw -a -c UD -d mlx4_0 -i 1
------------------------------------------------------------------
Send BW Test
Connection type : UD
Inline data is used up to 1 bytes message
local address: LID 0x0b, QPN 0x28004c, PSN 0xfaa100
remote address: LID 0x02, QPN 0x70004b, PSN 0xc14da8
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
------------------------------------------------------------------
在client端执行
[ibtests]# ib_send_bw -a -c UD -d mlx4_0 -i 1 10.10.11.8
------------------------------------------------------------------
Send BW Test
Connection type : UD
Inline data is used up to 1 bytes message
local address: LID 0x02, QPN 0x70004b, PSN 0xc14da8
remote address: LID 0x0b, QPN 0x28004c, PSN 0xfaa100
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
2 1000 7.51 7.21
4 1000 15.29 14.19
8 1000 30.66 30.45
16 1000 60.33 59.95
32 1000 119.53 113.20
64 1000 233.75 233.16
128 1000 414.95 413.64
256 1000 794.90 698.20
512 1000 1600.46 774.67
1024 1000 2011.81 804.29
2048 1000 2923.29 2919.91
------------------------------------------------------------------
2、延时测试
在server端执行
[ibtests]# ib_send_lat -a -c UD -d mlx4_0 -i 1
------------------------------------------------------------------
Send Latency Test
Inline data is used up to 400 bytes message
Connection type : UD
local address: LID 0x0b QPN 0x2c004c PSN 0xa1be86
remote address: LID 0x02 QPN 0x74004b PSN 0x6ea837
------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec]
2 1000 1.41 4.45 1.43
4 1000 1.41 3.84 1.43
8 1000 1.41 2.75 1.43
16 1000 1.41 3.01 1.42
32 1000 1.49 3.92 1.50
64 1000 1.55 3.96 1.57
128 1000 1.70 2.58 1.71
256 1000 2.41 5.73 2.45
512 1000 2.82 4.07 2.90
1024 1000 3.28 4.95 3.31
2048 1000 4.11 11.74 4.14
------------------------------------------------------------------
在client端执行
[ibtests]# ib_send_lat -a -c UD -d mlx4_0 -i 2 10.10.11.8
------------------------------------------------------------------
Send Latency Test
Inline data is used up to 400 bytes message
Connection type : UD
local address: LID 0x02 QPN 0x74004b PSN 0x6ea837
remote address: LID 0x0b QPN 0x2c004c PSN 0xa1be86
------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec]
2 1000 1.41 9.97 1.43
4 1000 1.38 5.31 1.43
8 1000 1.41 2.78 1.43
16 1000 1.40 4.01 1.42
32 1000 1.49 3.67 1.50
64 1000 1.55 5.20 1.56
128 1000 1.69 3.13 1.71
256 1000 2.40 5.72 2.45
512 1000 2.83 4.13 2.90
1024 1000 3.28 4.95 3.31
2048 1000 4.11 11.68 4.14
------------------------------------------------------------------
2、网络调优
#启用connected模式（默认是datagram模式，datagram模式下网络延时更低，connected模式下网络带宽更高），接口带宽提高一倍左右 echo connected > /sys/class/net/ib0/mode or sed -i 's/SET_IPOIB_CM=.*/SET_IPOIB_CM=yes' /etc/infiniband/openib.conf /etc/init.d/openibd restart
#系统参数调优（centos7） systemctl status tuned.service #看看是否启用了tuned服务 tuned-adm profile network-throughput #优化网络带宽 tuned-adm profile network-latency #优化网络延时 tuned-adm active #查看当前配置