After we have prepared the source code of librxe-dev and rxe-dev, we can proceed to install rdma. Saying to install rdma is actually recompiling the linux kernel to support the soft-Roce protocol to support rdma.
First post the download path:
- Github: https://github.com/SoftRoCE/rxe-dev.git (this repository)
- Active Branch:
rxe_submission_v18. It should be noted here that after switching to version 18, it is best to use git pull origin rxe_submission_v18 to pull it again to ensure that it is correct. If there are two files aux.c and aux.h under the directory drivers/gpu/drm/nouveau/nvkm/subdev/i2c/, then it is correct. - Github: https://github.com/SoftRoCE/librxe-dev.git
- Current Version:
librxe-1.0.0
Step 1 of the installation of rdma under centos
, prepare several software:
gcc
gcc-c++
bc
pciutils
openssl-devel
automake
ncurses-devel
libibverbs
libibverbs-devel
libibverbs-utils
librdmacm
librdmacm-devel
librdmacm-utils
perl-Switch
Note: You
need to use centos source to install perl-Switch, it is useless to use domestic 163 source.
2, Go to the rxe-dev directory,
cp /boot/config-$(uname –r) .config
to switch to root user authority, run make menuconfig
to display the selection interface. (If it doesn't appear, install ncurse-devel.)
Type "/", then RDMA_RXE, press enter, and you'll find an option for RDMA_RXE.
Enter the number 1, you will select the RDMA_RXE setting, enter "M", and select the RDMA configuration. If you can't lose M, enter a space. In short, you must add RDMA to the compilation options.
Then press the arrow keys to look down a few lines, select all options with the word RDMA, press "M" or space, just in case. For example "IP-over-InfiniBand data path debugging".
To be on the safe side, we also enter INFINIBAND to select several configurations. It depends on which ones are like rdma. Just look at the first few.
Then vi.config to confirm that CONFIG_RDMA_RXE is m, CONFIG_INFINIBAND_ADDR_TRANS and CONFIG_INFINIBAND_ADDR_TRANS_CONFIGFS are y
3, enter make to compile. make -j 32 means to compile with 32 cores, generally make -j will automatically determine the number of cpu cores. My virtual machine is 4 cores, run make -j 4
4, make modules_install, maybe some modules will be lost in the middle of execution, it doesn't matter, it doesn't matter.
5, make install
6, make headers_install INSTALL_HDR_PATH=/usr
7, confirm whether the new kernel is in the grub boot. Check /etc/grub.cfg to see it. At boot time, you can choose to start the new kernel.
8, Start installing librxe-dev. Since librxe-dev directory: cd librxe-dev
9, ./configure --libdir=/usr/lib64/ --prefix=
10, make
11, make install
12, 验证 rdma 。
rxe_cfg start
rxe_cfg add eth0
rxe_cfg status
------------------------------------------
------------------------------------------
------------------------------------------
------------------------------------------
If you are using a cloned virtual machine, then you need to pay attention, because the network card information of the cloned virtual machine is exactly the same as before, including the uuid of the network card, so we need to remove the old network card, add a new network card, and Modify the configuration file manually.
Specific methods:
1. Remove the hardware of the network adapter in the settings of vmware, then add new hardware, and select "custom-VMnet8 (NAT mode)".
2. There is no configuration file for the newly added network card under centos. All we have to manually add the configuration file, first look at the information of the newly added network card: nmcli con . (If the nmcli command cannot be executed, then go to
yum -y install NetworkManager and start it, service NetworkManager start, refer to the URL http://www.linuxidc.com/Linux/2016-08/134149.htm )
Here my information is :
NAME UUID TYPE DEVICE
eno33554984 fb67dbad-ec81-39b4-42b1-ebf975c3ff13 802-3-ethernet eno33554984
3, Create a new configuration file in the /etc/sysconfig/network-scripts directory, I named it ifcfg-eno33554984, and then added Configuration information:
TYPE="Ethernet"
# IP address fixed
BOOTPROTO="static"
BROADCAST= 192.168.3.255 #Manually
configure IP address
IPADDR=192.168.3.144
NETMASK=255.255.255.0
NETWORK=192.168.3.0
DEFROUTE="yes"
PEERDNS="yes"
PEERROUTES="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="no"
IPV6_AUTOCONF="no"
IPV6_DEFROUTE="yes"
IPV6_PEERDNS="yes"
IPV6_PEERROUTES="yes"
IPV6_FAILURE_FATAL="no " #This
is modified to the information viewed above
NAME="eno33554984" #This
is modified to be viewed above infoUUID
="fb67dbad-ec81-39b4-42b1-ebf975c3ff13"
#HWADDR=00:0c:29:81:af:02 #This
is modified to the information viewed above
DEVICE="eno33554984"
ONBOOT="yes"
4. After the configuration is modified, restart the network: service network restart
5. At this time, input rxe_cfg start to start. We can check the status with rxe_cfg status.
Name Link Driver Speed NMTU IPv4_addr RDEV RMTU
eno33554984 yes e1000 rxe0 1024 (3)
If rxe0 appears, it means success. If not, then we can perform the following steps:
rxe_cfg add eno33554984
and then rxe_cfg start again.
6. At this point, we can use the rdma protocol. However, since we are a cloned virtual machine, the information of a network card eno16777736 in rdma will remain, and sometimes a prompt like "sh: line 0: echo: write error: Invalid argument" , "interface eno16777736 not found" will appear. In order to avoid affecting our judgment, we can modify the content of /var/rxe/rxe and remove the line eno16777736.
7. Finally, since the kernel that supports rdma is often selected when booting, we can modify /etc/grub.cfg or /etc/grub2.cfg, and comment out other kernels, leaving only "CentOS Linux (4.7.0 -rc3+) 7 (Core)" This kernel, you don't need to manually select it every time you boot.
8, When testing rdma, remember to close the firewall sudo iptables -F
#chkconfig iptables off #开机不启动
or # setenforce 0 #调整到宽松模式
# vim /etc/selinux/config
SELINUX=disabled #以后开机selinux都禁用