【RDMA】RDMA network card communication

I am a novice and am learning RDMA from scratch. Please correct me if there are any mistakes. I would be very grateful.

In the Unbuntu18.04 environment, the kernel comes with rdma-core. Enter the following command to view the system version information.

cat /proc/version

Check whether the RDMA network card is installed

lspci | grep -i mellanox

If it exists, relevant information similar to the following will be output.

Check the working status of the network card. ibv_devices is a tool included in the libibverbs-utils.rpm package and is used to display the RDMA devices on the local machine:

ibv_devices

 

 For some other information, there must be PORT_ACTIVE in the state to indicate that RDMA can run.

ibv_devinfo

 

Next, use rdma_server and rdma_client to test the connectivity of the network card.

rdma_server and rdma_client is a simple ping-pong test to establish an RDMA connection between two nodes

First check the ip address on the server side

ifconfig

Execute again

rdma_server

Run on another virtual machine

rdma_client -s 10.5.1.3

 If the above characters appear on the client side, it indicates that the RDMA connection is normal.

The following uses rping to establish a set of reliable RDMA connections between two nodes and transfer data between the nodes. 

Enter the following command on the server side, -s indicates the server, -C indicates the number of pings, and -v outputs the information.

rping -s -C 10 -v

On the client side

rping  -c -a 10.5.1.3 -C 10 -v

Output the following information

Let’s conduct some RDMA performance tests using the ib_* series of commands.

The three functions ib_send_*, ib_write_*, and ib_read_* are all functions in the InfiniBand API and are used for data transmission on the InfiniBand network. Their main differences are as follows:

1. ib_send_*: Used to send messages to remote endpoints. Send operations can be blocking or non-blocking, depending on the function's arguments. It needs to specify parameters such as the buffer for sending data, data size, and notification method after the sending is completed.

2. ib_write_*: Used to write data to the receive buffer of the remote endpoint. The write operation is usually blocking, and it requires specifying parameters such as the buffer for writing data, data size, and writing location.

3. ib_read_*: Used to read data from the send buffer of the remote endpoint. Read operations are usually blocking and require specifying parameters such as the buffer to receive data, data size, and reading position.

In short, ib_send_*, ib_write_*and ib_read_*are all functions used for data transmission, but their functions and usage are slightly different. Among them, ib_send_*it is to send data, ib_write_*to write data to the receive buffer, and ib_read_*to read data from the send buffer.

Bandwidth test, on the server side

ib_send_bw -d mlx5_0

On the client side

ib_send_bw -d mlx5_0 10.5.1.3

After running client

 The bandwidth this time is about 85Gb/s

The delay test is basically the same as the bandwidth test. Change bw to lat. After running