RDMA - libibverbs code analysis (2) - Device Discovery

    Based on an article https://www.cnblogs.com/xingmuxin/p/11057845.html

    We now analyze libibverbs from the code, jump to the kernel code analysis, code location in ./drivers/infiniband/core/uverbs_main.c, here we mainly explore, create infiniband_verbs the file directory. To generate this directory, we need to load ib_uverbs.ko. Let's look at how this ko is compiled.

infiniband-$(CONFIG_INFINIBAND_ADDR_TRANS)      := rdma_cm.o
user_access-$(CONFIG_INFINIBAND_ADDR_TRANS)     := rdma_ucm.o

obj-$(CONFIG_INFINIBAND) +=             ib_core.o ib_cm.o iw_cm.o \
                                        $(infiniband-y)
obj-$(CONFIG_INFINIBAND_USER_MAD) +=    ib_umad.o
obj-$(CONFIG_INFINIBAND_USER_ACCESS) += ib_uverbs.o $(user_access-y)
obj-$(CONFIG_INFINIBAND_USER_ACCESS_UCM) += ib_ucm.o $(user_access-y)
……

CONFIG_INFINIBAND_USER_ACCESS explain this compilation options are:

Userspace InfiniBand access support. This enables the kernel side of userspace verbs and the userspace communication manager (CM). This allows userspace processes to set up connections and directly access InfiniBand hardware for fast-path operations. You will also need libibverbs, libibcm and a hardware driver library from rdma-core https://github.com/linux-rdma/rdma-core.

That ib_uverbs.ko for direct user-mode access infiniband hardware support, we in addition to libibverbs, libibcm, but also need hardware drivers Library .

    Generally know ib_uverbs.ko role (where u, I understand it should be userspace), we look at its implementation.

static  int __init ib_uverbs_init ( void ) 
{ 
    int RET; 
    // before the system is registered to a character device, you should first call the function, the system application device number 
    RET = register_chrdev_region (IB_UVERBS_BASE_DEV, 
                     IB_UVERBS_NUM_FIXED_MINOR, 
                     " infiniband_verbs " );
 ...... 
    RET = alloc_chrdev_region ( dynamic_uverbs_dev &, 0 , 
                  IB_UVERBS_NUM_DYNAMIC_MINOR, 
                  " infiniband_verbs " );
 ......   
    // infiniband_verbs create a directory under / sys / class / directory 
    uverbs_class = class_create (THIS_MODULE, "infiniband_verbs");
    if (IS_ERR(uverbs_class)) {
        ret = PTR_ERR(uverbs_class);
        pr_err("user_verbs: couldn't create class infiniband_verbs\n");
        goto out_chrdev;
    }

    uverbs_class->devnode = uverbs_devnode;

    ret = class_create_file(uverbs_class, &class_attr_abi_version.attr);
    if (ret) {
        pr_err("user_verbs: couldn't create abi_version attribute\n");
        goto out_class;
    }

    ret = ib_register_client(&uverbs_client);
    if (ret) {
        pr_err("user_verbs: couldn't register client\n");
        goto out_class;
    }

    return 0;

……
}

   Udev above seems to create some fixed operations, to create a corresponding folder directory under / sys / class directory.

   Next, we go back to libibverbs code, after find_sysfs_devs, will traverse sysfs_dev_list, execution try_drivers, try_drivers code is as follows:

static struct ibv_device *try_drivers(struct ibv_sysfs_dev *sysfs_dev)
{
    struct ibv_driver *driver;
    struct ibv_device *dev;

    for (driver = head_driver; driver; driver = driver->next) {
        dev = try_driver(driver, sysfs_dev);
        if (dev)
            return dev;
    }

    return NULL;
}

The key point here is not assigned head_driver before it is empty, we will look at where head_driver assignment from the code to see when you call ibv_register_driver give head_driver assignment, this ibv_register_driver search function can not call at libibverbs code, in driver.h file, define a macro:

#define PROVIDER_DRIVER(drv)                                                   \
    static __attribute__((constructor)) void drv##__register_driver(void)  \
    {                                                                      \
        verbs_register_driver(&drv);                                   \
    }

__attribute __ ((constructor)) set the priority attribute constructorparameter allows the system to perform main()a function call function before. 
Search PROVIDER_DRIVER, find all rdma drive, we are using this function to initialize their user-mode library. We mlx5 example. In libmlx5 code, mlx5.c (can see rdma-core / providers / mlx5.c) , is called to initialize PROVIDER_DRIVER user mode driver. 

In summary, we can conclude that, libibverbs and libmlx5 user mode driver close cooperation, find mlx5 drive device, rather than through pcie, and ib_uverbs.ko, mlx5 kernel mode drivers, is to provide support. So the question now is, user-mode drivers operate like it, how ordinary device drivers for device discovery is it? 
The next section we discuss this issue.