RDMA - libibverbs code analysis (1)

Libibverbs download the latest code, https://downloads.openfabrics.org/verbs/README.html to version 1.2.0. Back gradually analysis libibverbs source.

1、ibv_get_device_list：

This particular function is implemented in libibverbs-1.2.0 / src / devices.c file.

struct ibv_device ** __ ibv_get_device_list ( int * NUM) 
{ 
...... 
    pthread_once ( & device_list_once, count_devices); // count_devices This function is performed only once in this process. See specific analysis 1.1
 ...... 
    L = calloc (+ NUM_DEVICES . 1 , the sizeof ( struct ibv_device * )); // allocate a length of n consecutive space size, and clears the continuous space
 ...... 

    for (I = 0 ; I < NUM_DEVICES; ++ I) 
        L [I] = device_list [I];
     IF (NUM)
         * NUM = NUM_DEVICES; 

    return L; 
}

1.1 pthread_once：

In a multithreaded environment, some things need only be done once. Usually when the application is initialized, it can be relatively easily placed in the main function. But when you write a library, which can not be initialized in the main, you can use static initialization, but once initialized (pthread_once) would be more easier.
int pthread_once (pthread_once_t * once_control, void (* init_routine) (void));
Function : This function uses the initial value PTHREAD_ONCE_INIT once_control variable guaranteed init_routine () function is performed only once in the present process execution sequence.
Linux Threads use mutexes and condition variables guaranteed by pthread_once () function performs a specified and performed only once, and once_control indicates whether executed. If once_control initial value is not PTHREAD_ONCE_INIT (Linux Threads defined as 0), pthread_once () will not normal behavior. In LinuxThreads, the actual "disposable function", there are three execution state: NEVER (0), IN_PROGRESS ( 1), DONE (2), if the initial value is set to 1 once, since all pthread_once is () must wait for one stimulate "executed a" signal, so all pthread_once () will fall into perpetual waiting; if set to 2, it indicates that the function has been performed once, so all pthread_once () will return 0 immediately.

Then proceed to analyze, focusing on function count_devices, count_devices-> ibverbs_init (init.c) . In ibverbs_init function, the system will first get the path is usually / sys, and then check the version of the kernel ABI, ABI for lower or higher than a certain version, ibverbs not supported. About ABI can refer https://blog.csdn.net/juS3Ve/article/details/82782987 , specifically ABI and cpu architecture and OS related.

Next we will check the resource limit, as follows:

static  void check_memlock_limit ( void ) 
{ 
    struct of rlimit RLIM; 

    IF (! geteuid ())
         return ; 

    IF (Getrlimit (RLIMIT_MEMLOCK, & RLIM)) {     the maximum amount of data that can be locked // process in memory, in bytes. 
        fprintf (stderr, PFX " Warning: Getrlimit (RLIMIT_MEMLOCK) failed. " );
         return ; 
    } 

    IF (rlim.rlim_cur <= 32768 ) 
        fprintf (stderr, PFX " Warning: IS% RLIMIT_MEMLOCK LU bytes \ n-. " 
            "     This Will Severely limit memory registrations. \ n",
            rlim.rlim_cur);
}

The maximum amount of data required in the process here can be locked in memory, the soft limit is greater than 32K. Otherwise it will affect the memory register. In use, you can set ulimit the lifting of restrictions on the system.

Then call the function reads the configuration file read_config (), the function reads the configuration file from the /sysocnfdir/libibverbs.d directory, profile path on my system for /etc/libibverbs.d, the content in this directory as follows:

linux-MgXfWk:/etc/libibverbs.d # ls
bnxt_re.driver  cxgb4.driver      hns.driver    ipathverbs.driver  mlx5.driver   nes.driver     qedr.driver  vmw_pvrdma.driver
cxgb3.driver    hfi1verbs.driver  i40iw.driver  mlx4.driver        mthca.driver  ocrdma.driver  rxe.driver
linux-MgXfWk:/etc/libibverbs.d # cat cxgb4.driver
driver cxgb4
linux-MgXfWk:/etc/libibverbs.d # cat mlx5.driver
driver mlx5

These files are read from the driver name, and then add them to a list, the list name is driver_name_list.

Then call find_sysfs_devs function, obtain / sys / class / device file under infiniband_verbs catalog, view the kernel code, in uverbs_main.c, you create the corresponding device. So my understanding is that find_sysfs_devs this function is actually looking for some device device drivers installed. These devices were then added to the list, sysfs_dev_list.

After traversing the sysfs_dev_list list, call try_drivers, to see if there are registration drive, there is then added to device_list the list, then try from driver_name_list in drvier_name traverse to load_driver, and then see if the sysfs_dev_list match, sysfs_dev_list in there, then join device_list if there sysfs_dev_list, but the driver is not, then the device will sysfs_dev_list corresponding to the removed, is not placed in the device_list.

This piece of code was later made changes finishing high version, focusing on the comment section.

/* Remove entries from the sysfs_list that are already preset in the
     * device_list, and remove entries from the device_list that are not
     * present in the sysfs_list.
     */
    list_for_each_safe(device_list, vdev, tmp, entry) {
        struct verbs_sysfs_dev *old_sysfs = NULL;
……
            }
        }

So far this part of the analysis is complete, we later analysis using high version of the code. Use libibverbs-17.

RDMA - libibverbs code analysis (1)

Guess you like