rdma centos 7.3 install
1, the installation dependencies
yum install epel-release -y yum install gcc gcc-c++ bc openssl-devel automake ncurses-devel libibverbs -y yum install libibverbs-devel libibverbs-utils librdmacm librdmacm-devel librdmacm-utils perl-Switch elfutils-libelf-devel -y
2, librxe-dev and rxe-dev Download
download link
Github: https://github.com/SoftRoCE/rxe-dev.git Github: https://github.com/SoftRoCE/librxe-dev.git
Note: rxe-dev Download v18 version, that is-dev-RXE RXE submission v18
3, mounting rxe-dev
unzip rxe-dev-rxe_submission_v18.zip cd rxe-dev-rxe_submission_v18/ cp /boot/config-3.10.0-514.el7.x86_64 .config
Note: Use the root user, execute the following command
make menuconfig
Selection screen will appear (if you do not appear, you need to install ncurse-devel)
input "/", and then enter rxe, press enter, will look for options related to the rxe.
Enter the number 1, will choose the "Software RDMA over Ethernet (ROCE) driver" setting input "M", RDMA selected configuration, if M can not lose, then enter a space.
Save button to move, transport, loading saved to the .config exit installation interface (exit).
Vi .config then confirm
the CONFIG the RDMA RXE is m
the CONFIG INFINIBAND ADDR TRANS and the CONFIG INFINIBAND ADDR TRANS_CONFIGFS as y
-j 4 the make the make modules_install, may be prompted to lose some during execution module, this does not matter, it does not matter. the install the make the make headers_install INSTALL_HDR_PATH = / usr
Confirm whether the new kernel boot in grub. View /etc/grub.cfg can see. At boot time you can choose to boot the new kernel
4, the installation librxe-dev
cd librxe-dev ./configure --libdir=/usr/lib64/ --prefix= make make install
Reboot the operating system, at the time of start-up, select 4.7.0-rc3 kernel
starts, core edition
uname -r
5. Verify rdma
[root@aboss ~]# rxe_cfg start Name Link Driver Speed NMTU IPv4_addr RDEV RMTU ens33 yes e1000 virbr0 no bridge virbr0-nic no tun [root@aboss ~]# rxe_cfg add ens33 [root@aboss ~]# rxe_cfg status Name Link Driver Speed NMTU IPv4_addr RDEV RMTU ens33 yes e1000 rxe0 1024 (3) virbr0 no bridge virbr0-nic no tun
View rxe equipment
IBV Devices program displays all devices currently in the system, while the IBV the devinfo command gives specific information for each device
[root@aboss ~]# ibv_devices device node GUID ------ ---------------- rxe0 020c29fffe55c818 [root@aboss ~]# ibv_devinfo rxe0 hca_id: rxe0 transport: InfiniBand (0) fw_ver: 0.0.0 node_guid: 020c:29ff:fe55:c818 sys_p_w_picpath_guid: 0000:0000:0000:0000 vendor_id: 0x0000 vendor_part_id: 0 hw_ver: 0x0 phys_port_cnt: 1 port: 1 state: PORT_ACTIVE (4) max_mtu: 4096 (5) active_mtu: 1024 (3) sm_lid: 0 port_lid: 0 port_lmc: 0x00 link_layer: Ethernet
6, softRoCE connectivity test
Server
rping -s -a 192.168.1.133 -v -C 10
Client
rping -c -a 192.168.1.133 -v -C 10
7, compilation instructions on librdmacm
git clone https://github.com/ofiwg/librdmacm.git cd librdmacm yum install autoconf automake gettext gettext-devel libtool -y ./autogen.sh ./configure make make install
8. Frequently Asked Questions
(1) If you clone a virtual machine, network cards need to address the problem
(2) use rdma, please turn off the firewall and selinx