glusterfs 4.0.1 iobuf analysis notes

1. The buffer used by iobuf to send and receive data is managed and used by iobuf_pool.

The following institutions are defined in iobuf.h:

struct iobuf is a block of memory used to store data;
struct iobuf_arena is a large memory block, divided into N pieces, each small block is used for the ptr pointer of iobuf;
iobuf_init_config is used to indicate the size and number of each type of iobuf

struct iobuf_init_config {
        size_t   pagesize;
        int32_t  num_pages;
};

An array is defined at the beginning of iobuf.c. This array is used as a global variable to define iobuf with 8 sizes.

Each record represents the page size of a certain type of iobuf and the number in anena,

That is to say, the pagesize and page_count of a certain type of iobuf_arena correspond to a certain row in this table.

struct iobuf_init_config gf_iobuf_init_config[] = {
        /* { pagesize, num_pages }, */
        {128, 1024},
        {512, 512},
        {2 * 1024, 512},
        {8 * 1024, 128},
        {32 * 1024, 64},
        {128 * 1024, 32},
        {256 * 1024, 8},
        {1 * 1024 * 1024, 2},
};

There is no hard-coded value for this global variable, but it can be calculated to be 8.
In iobuf.c, the iobuf_pool_new function is used to initialize the entire buffer pool:

struct iobuf_pool *iobuf_pool_new (void)

In the process of initializing the iobuf_pool pool, 8 iobuf_arena will be allocated according to this table and established as a linked list. As shown below:

With reference to global variables, call iobuf_pool_add_arena (), and internally call __iobuf_arena_init_iobufs (struct iobuf_arena *iobuf_arena)
function to initialize 8 different arena:

The allocation methods for each arena are as follows:
1) The __iobuf_arena_alloc() function will allocate a large chunk of memory for arena:

// The size and number correspond to a record in the global variable
iobuf_arena->arena_size = rounded_size * num_iobufs;     
                        
// use memory map to get chunk of memory
iobuf_arena->mem_base = mmap (NULL, iobuf_arena->arena_size,
                                      PROT_READ|PROT_WRITE,
                                      MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);

2) __iobuf_arena_alloc() function internally calls __iobuf_arena_init_iobufs()

Initialize iobufs:

A block of memory of size page_count * sizeof(struct iobuf) is allocated for iobuf_arena->iobufs to store the iobuf header structure in chunks.
The allocated memory block is split.

Then cut the large memory block pointed to by iobuf_arena->mem_base into small pieces, the size is iobuf_arena->page_size, and the
number is iobuf_arena->page_count.

After the 8 types of arena are allocated, the logical structure is as follows:

3) Add a larger arena to the iobuf_pool to meet the needs of the upper layer:

iobuf_create_stdalloc_arena (iobuf_pool);

This function does not allocate a large chunk of memory for arena buffers. If an iobuf of a suitable size cannot be found when it is used, it will be created dynamically.

2. The use of iobuf_pool is mainly two functions:

struct iobuf * iobuf_get2 (struct iobuf_pool *iobuf_pool, size_t page_size)；

void iobuf_put (struct iobuf *iobuf)

glusterfs 4.0.1 iobuf analysis notes

Guess you like