1, the installation dependencies

yum install epel-release -y  
yum install gcc gcc-c++ bc openssl-devel automake ncurses-devel libibverbs -y  
yum install libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils perl-Switch elfutils-libelf-devel  -y

2, librxe-dev and rxe-dev Download

download link

Github: https://github.com/SoftRoCE/rxe-dev.git  
Github: https://github.com/SoftRoCE/librxe-dev.git

Note: rxe-dev Download v18 version, that is-dev-RXE RXE submission v18

3, mounting rxe-dev

unzip rxe-dev-rxe_submission_v18.zip
cd rxe-dev-rxe_submission_v18/
cp /boot/config-3.10.0-514.el7.x86_64 .config

Note: Use the root user, execute the following command

make menuconfig

Selection screen will appear (if you do not appear, you need to install ncurse-devel)
input "/", and then enter rxe, press enter, will look for options related to the rxe.
Enter the number 1, will choose the "Software RDMA over Ethernet (ROCE) driver" setting input "M", RDMA selected configuration, if M can not lose, then enter a space.
Save button to move, transport, loading saved to the .config exit installation interface (exit).
Vi .config then confirm 
the CONFIG the RDMA RXE is m
the CONFIG INFINIBAND ADDR TRANS and the CONFIG INFINIBAND ADDR TRANS_CONFIGFS as y

-j 4 the make   
the make modules_install, may be prompted to lose some during execution module, this does not matter, it does not matter.  
the install the make   
the make headers_install INSTALL_HDR_PATH = / usr

Confirm whether the new kernel boot in grub. View /etc/grub.cfg can see. At boot time you can choose to boot the new kernel

4, the installation librxe-dev

cd librxe-dev  
./configure --libdir=/usr/lib64/ --prefix=  
make  
make install

Reboot the operating system, at the time of start-up, select 4.7.0-rc3 kernel
starts, core edition

uname -r

5. Verify rdma

[root@aboss ~]# rxe_cfg start 
  Name        Link  Driver  Speed  NMTU  IPv4_addr  RDEV  RMTU  
  ens33       yes   e1000                                       
  virbr0      no    bridge                                      
  virbr0-nic  no    tun                                         
[root@aboss ~]# rxe_cfg add ens33
[root@aboss ~]# rxe_cfg status 
  Name        Link  Driver  Speed  NMTU  IPv4_addr  RDEV  RMTU          
  ens33       yes   e1000                           rxe0  1024  (3)  
  virbr0      no    bridge                                              
  virbr0-nic  no    tun

View rxe equipment
IBV Devices program displays all devices currently in the system, while the IBV the devinfo command gives specific information for each device

[root@aboss ~]# ibv_devices
    device          	   node GUID
    ------          	----------------
    rxe0            	020c29fffe55c818
[root@aboss ~]# ibv_devinfo  rxe0
hca_id:	rxe0
transport:	InfiniBand (0)
fw_ver:	0.0.0
node_guid:	020c:29ff:fe55:c818
sys_p_w_picpath_guid:	0000:0000:0000:0000
vendor_id:	0x0000
vendor_part_id:	0
hw_ver:	0x0
phys_port_cnt:	1
port:	1
state:	PORT_ACTIVE (4)
max_mtu:	4096 (5)
active_mtu:	1024 (3)
sm_lid:	0
port_lid:	0
port_lmc:	0x00
link_layer:	Ethernet

6, softRoCE connectivity test

Server

rping -s -a 192.168.1.133 -v -C 10

Client

rping -c -a 192.168.1.133 -v -C 10

7, compilation instructions on librdmacm

git clone https://github.com/ofiwg/librdmacm.git
cd librdmacm
yum install autoconf automake gettext gettext-devel libtool -y
./autogen.sh 
./configure 
make 
make install

8. Frequently Asked Questions

(1) If you clone a virtual machine, network cards need to address the problem
(2) use rdma, please turn off the firewall and selinx