RDMA study notes

RDMA WRITE, the requesting end actively writes data to the responding end

1. The requesting APP delivers WQE to SQ

2. The requesting network card takes out the WQE from the SQ, and the WQE provides the virtual address VA of the local data.

3. The requesting end network card converts the physical address PA according to VA and assembles the data packet.

4. The requesting network card sends the data packet to the responding network card

5. The responder network card converts VA into PA and places it in the specified location.

6. The responding network card replies with ACK

7. After receiving the ACK, the requesting network card generates a CQE and puts it into the CQ.

8.APP obtains completion information

READ: Actively reads remote memory data, and the remote data is carried when the network card replies.

1. The requesting APP issues WQE

2. The network card takes out WQE from SQ and obtains relevant information.

3. The requesting network card sends the READ request to the responding network card.

4. After receiving the request, the responding network card converts VA to PA and reads the data from the memory.

5. The responding network card replies to the request and carries the requesting data

6. After receiving the data, the requesting hardware places the data into the designated memory area.

7. Generate CQE and put it into CQ

8.APP obtains completion information

In the process of RDMA READ and WRITE, the entire process does not require the participation of the CPU, but before the READ or WRITE operation, permissions need to be obtained through other means, such as socket.

MR，Memory Region

The RDMA software layer plans an area in the memory for storing and sending and receiving data. This area associates the host memory with RDMA to ensure that RDMA can access the memory area correctly and efficiently during data transmission. This memory area is MR. Registering MR can Using the interface ibv_reg_mr() provided by verbs, during the registration process, the memory will create a mapping table between VA and PA. When the RDMA network card needs to obtain the PA, the PA can be obtained by looking up the table.

When registering MR, two keys L_KEY and R_KEY will be generated, corresponding to local and remote respectively, to ensure access to the memory area.

After registering the MR, in order to prevent the mapping relationship from changing due to page changes, the memory where the MR is located will be locked until the communication is completed and the MR is actively logged out.

PD，Protection Domain

The IB protocol stipulates that each node must have at least one PD, each QP must belong to a PD, and each MR must also belong to a PD.

The resources accommodated by each PD are isolated from each other. The QP and MR between the two PDs on Node 0 cannot access each other. The QP of Node1 cannot access the MR1 of Node0.

ibv_alloc_pd is used to allocate a PD object and return the PD handle

code part

Register PD: Call the ibv_alloc_pd function to allocate a PD object. This function will return a pointer of type struct ibv_pd* pointing to the allocated PD object.

struct ibv_pd *ibv_alloc_pd(struct ibv_context *context);

Unregister PD: The ibv_dealloc_pd() function accepts a pointer of type struct ibv_pd* as a parameter and releases the PD object.

int ibv_dealloc_pd(struct ibv_pd *pd);

Register MR: Return a pointer to ibv_mr

struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr, 
                          size_t length, int access);

pd: pointer to the protection domain associated with this memory area

addr: pointer points to the memory area to be registered

length: the size of the MR to be registered, unit B

access: describes access permissions

Log out MR:

int ibv_dereg_mr(struct ibv_mr *mr);

Find relevant example code in rdma -example

git clone https://github.com/animeshtrivedi/rdma-example
cd rdma-example/src
vi rdma_server.c

In line 50, there is code about protection domain application. The RDMA device where the incoming client connection is located allocates a protection domain. If it fails, relevant error information is returned.

        pd = ibv_alloc_pd(cm_client_id->verbs
                        /* verbs defines a verb's provider,
                         * i.e an RDMA device where the incoming
                         * client connection came */);
        if (!pd) {
                rdma_error("Failed to allocate a protection domain errno: %d\n",
                                -errno);
                return -errno;
        }

When data transmission is required, MR needs to be registered. rdma-example is only a simple communication function and does not use this interface.