I have always felt that the most difficult part of C/C++ is the management of memory, which is far more than just new/delete and malloc/free. As the amount of code increases, the complexity of the program structure increases. Various memory problems quietly breed. And as a platform, later plug-in extensions are inevitable. The characteristics of the long-running acquisition platform put forward high requirements for stability. Not c#, java, no virtual machine to manage memory for you, everything depends on yourself. So I want to see how the classics of C like nginx, python, and lua deal with memory management.
Let's take a look at nginx first, because it is said on the Internet that the design of nginx's memory pool is very delicate:
1. Basic structure
先来学习一下nginx内存池的几个主要数据结构：
ngx_pool_data_t（内存池数据块结构）
typedef struct {
u_char *last;
u_char *end;
ngx_pool_t *next;
ngx_uint_t failed;
} ngx_pool_data_t;
ngx_pool_s (memory pool header structure)
struct ngx_pool_s {
ngx_pool_data_t d;
size_t max;
ngx_pool_t *current;
ngx_chain_t *chain;
ngx_pool_large_t *large;
ngx_pool_cleanup_t *cleanup;
ngx_log_t *log;
};
It can be said that ngx_pool_data_t and ngx_pool_s basically constitute the main structure of the nginx memory pool. The following describes the main structure of the nginx memory pool in detail:
as shown above, the nginx memory pool is actually a linked list composed of ngx_pool_data_t and ngx_pool_s, where:
ngx_pool_data_t ： ：
last: It is an unsigned char type pointer, which saves the last address of the current memory pool allocation, that is, the next allocation starts from here.
end: the end position of the memory pool;
Next: There are many blocks of memory in the memory pool. These memory blocks are linked into a linked list through this pointer, and next points to the next block of memory.
failed: The number of memory pool allocation failures.
ngx_pool_s
d: The data block of the memory pool;
max: the maximum value of data blocks in the memory pool;
current: points to the current memory pool;
chain: The pointer is linked to an ngx_chain_t structure;
large: Large memory linked list, which is used when the allocated space exceeds max;
cleanup: release the callback of the memory pool
log: log information The
above is the main data structure involved in the memory pool. In order to simplify as much as possible, other data structures involved will be introduced below when they are actually used.
(More free C/C++, Linux, Nginx, ZeroMQ, MySQL, Redis, fastdfs, MongoDB, ZK, streaming media, CDN, P2P, K8S, Docker, TCP/IP, coroutine, DPDK, etc. multiple knowledge points dry goods Learning materials plus group 960994558)
2. Basic operation of memory pool
The main external methods of the memory pool are:
| Create memory pool | ngx_pool_t * ngx_create_pool (size_t size, ngx_log_t * log); |
|---|---|
| Destroy the memory pool | void ngx_destroy_pool(ngx_pool_t *pool); |
| Reset memory pool | void ngx_reset_pool(ngx_pool_t *pool); |
| Memory application (alignment) | void * ngx_palloc(ngx_pool_t *pool, size_t size); |
| Memory application (not aligned) | void * ngx_pnalloc(ngx_pool_t *pool, size_t size); |
| Memory clear | ngx_int_t ngx_pfree (ngx_pool_t * pool, void * p); |
Note:
Before analyzing the memory pool method, you need to introduce several main memory-related functions.
Here is only an introduction to Win32:
ngx_alloc: (just a simple encapsulation of malloc)
```cpp
```cpp
void *ngx_alloc(size_t size, ngx_log_t *log)
{
void *p;
p = malloc(size);
if (p == NULL) {
ngx_log_error(NGX_LOG_EMERG, log, ngx_errno,
"malloc(%uz) failed", size);
}
ngx_log_debug2(NGX_LOG_DEBUG_ALLOC, log, 0, "malloc: %p:%uz", p, size);
return p;
}
ngx_calloc: (call malloc and initialize to 0)
void *ngx_calloc(size_t size, ngx_log_t *log)
{
void *p;
p = ngx_alloc(size, log);
if (p) {
ngx_memzero(p, size);
}
return p;
}
ngx_memzero ：
#define ngx_memzero(buf, n) (void) memset(buf, 0, n)
ngx_free ：
#define ngx_free free
ngx_memalign ：
#define ngx_memalign(alignment, size, log) ngx_alloc(size, log)
Alignment here is mainly aimed at the need for dynamic alignment of some unix platforms, and encapsulates the posix_memalign() provided by POSIX 1003.1d. In most cases, the compiler and the C library transparently help you deal with the alignment problem. Through the macro NGX_HAVE_POSIX_MEMALIGN to control in nginx;
2.1. Memory pool creation (ngx_create_pool)
ngx_create_pool is used to create a memory pool. When we create it, we pass in our initial size:
ngx_pool_t *
ngx_create_pool(size_t size, ngx_log_t *log)
{
ngx_pool_t *p;
p = ngx_memalign(NGX_POOL_ALIGNMENT, size, log);
if (p == NULL) {
return NULL;
}
p->d.last = (u_char *) p + sizeof(ngx_pool_t);//初始状态：last指向ngx_pool_t结构体之后数据取起始位置
p->d.end = (u_char *) p + size;//end指向分配的整个size大小的内存的末尾
p->d.next = NULL;
p->d.failed = 0;
//#define NGX_MAX_ALLOC_FROM_POOL (ngx_pagesize - 1)
//内存池最大不超过4095，x86中页的大小为4K
size = size - sizeof(ngx_pool_t);
p->max = (size < NGX_MAX_ALLOC_FROM_POOL) ? size : NGX_MAX_ALLOC_FROM_POOL;
p->current = p;
p->chain = NULL;
p->large = NULL;
p->cleanup = NULL;
p->log = log;
return p;
}
Nginx's memory management is divided into large memory and small memory. When a certain requested memory is greater than a certain value, it needs to allocate space from large memory, otherwise, allocate space from small memory.
The size of the memory pool in nginx is set when it is created. When a small block of memory is allocated later, if the memory is not enough, it is to recreate a memory string into the memory pool instead of the original memory pool Expand. When a large block of memory is to be allocated, it is managed by redistributing space outside the memory pool, which is called a large memory pool.
2.2, memory application
ngx_palloc
void *
ngx_palloc(ngx_pool_t *pool, size_t size)
{
u_char *m;
ngx_pool_t *p;
if (size <= pool->max) {
//如果申请的内存大小大于内存池的max值，则走另一条路，申请大内存
p = pool->current;
do {
m = ngx_align_ptr(p->d.last, NGX_ALIGNMENT);//对内存地址进行对齐处理
if ((size_t) (p->d.end - m) >= size) {
//如果在当前内存块有效范围内，进行内存指针的移动
p->d.last = m + size;
return m;
}
p = p->d.next;//如果当前内存块有效容量不够分配，则移动到下一个内存块进行分配
} while (p);
return ngx_palloc_block(pool, size);
}
return ngx_palloc_large(pool, size);
}
Points to note here:
1. ngx_align_ptr, this is a macro used to round memory addresses. It is very delicate and can be done in one sentence. The role is self-evident, rounding can reduce the number of times the CPU reads the memory and improve performance. Because it doesn't really make sense to call malloc and other functions to apply for memory here, but just move the pointer mark, so the memory alignment can't help you, you have to do it yourself.
#define ngx_align_ptr(p, a) \
(u_char *) (((uintptr_t) (p) + ((uintptr_t) a - 1)) & ~((uintptr_t) a - 1))
2、ngx_palloc_block(ngx_pool_t *pool, size_t size)
This function is used to allocate a new memory block, open a new memory block for the pool memory pool, and apply for the use of size memory;
static void *
ngx_palloc_block(ngx_pool_t *pool, size_t size)
{
u_char *m;
size_t psize;
ngx_pool_t *p, *new, *current;
psize = (size_t) (pool->d.end - (u_char *) pool);//计算内存池第一个内存块的大小
m = ngx_memalign(NGX_POOL_ALIGNMENT, psize, pool->log);//分配和第一个内存块同样大小的内存块
if (m == NULL) {
return NULL;
}
new = (ngx_pool_t *) m;
new->d.end = m + psize;//设置新内存块的end
new->d.next = NULL;
new->d.failed = 0;
m += sizeof(ngx_pool_data_t);//将指针m移动到d后面的一个位置，作为起始位置
m = ngx_align_ptr(m, NGX_ALIGNMENT);//对m指针按4字节对齐处理
new->d.last = m + size;//设置新内存块的last，即申请使用size大小的内存
current = pool->current;
//这里的循环用来找最后一个链表节点，这里failed用来控制循环的长度，如果分配失败次数达到5次，
//就忽略，不需要每次都从头找起
for (p = current; p->d.next; p = p->d.next) {
if (p->d.failed++ > 4) {
current = p->d.next;
}
}
p->d.next = new;
pool->current = current ? current : new;
return m;
}
3. ngx_palloc_large(ngx_pool_t *pool, size_t size)
will first determine whether the requested memory size exceeds the maximum limit of the memory block in ngx_palloc. If it exceeds, then directly call ngx_palloc_large to enter the large memory block allocation process;
static void *
ngx_palloc_large(ngx_pool_t *pool, size_t size)
{
void *p;
ngx_uint_t n;
ngx_pool_large_t *large;
// 直接在系统堆中分配一块空间
p = ngx_alloc(size, pool->log);
if (p == NULL) {
return NULL;
}
n = 0;
// 查找到一个空的large区，如果有，则将刚才分配的空间交由它管理
for (large = pool->large; large; large = large->next) {
if (large->alloc == NULL) {
large->alloc = p;
return p;
}
if (n++ > 3) {
break;
}
}
//为了提高效率， 如果在三次内没有找到空的large结构体，则创建一个
large = ngx_palloc(pool, sizeof(ngx_pool_large_t));
if (large == NULL) {
ngx_free(p);
return NULL;
}
large->alloc = p;
large->next = pool->large;
pool->large = large;
return p;
}
2.3, memory pool reset
ngx_reset_pool
void
ngx_reset_pool(ngx_pool_t *pool)
{
ngx_pool_t *p;
ngx_pool_large_t *l;
//释放所有大块内存
for (l = pool->large; l; l = l->next) {
if (l->alloc) {
ngx_free(l->alloc);
}
}
pool->large = NULL;
// 重置所有小块内存区
for (p = pool; p; p = p->d.next) {
p->d.last = (u_char *) p + sizeof(ngx_pool_t);
}
}
2.4, memory pool cleaning
ngx_pfree
ngx_int_t
ngx_pfree(ngx_pool_t *pool, void *p)
{
ngx_pool_large_t *l;
//只检查是否是大内存块，如果是大内存块则释放
for (l = pool->large; l; l = l->next) {
if (p == l->alloc) {
ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, pool->log, 0,
"free: %p", l->alloc);
ngx_free(l->alloc);
l->alloc = NULL;
return NGX_OK;
}
}
return NGX_DECLINED;
}
Therefore, the allocation and release of large memory blocks and small memory blocks in the Nginx memory pool are different. When we use the memory pool, we can use ngx_palloc for allocation and ngx_pfree to release. For large memory, this is no problem, but for small memory, the small memory allocated will not be released. Because the allocation of a large memory block only checks the first 3 memory blocks, otherwise the memory is allocated directly, so the release of the large memory block must be timely.
ngx_pool_cleanup_s
The Nginx memory pool supports the cleaning of external resources through callback functions. ngx_pool_cleanup_t is a callback function structure, which is stored in the memory pool as a linked list. When the memory pool is destroyed, these callback functions are called cyclically to clean up the data.
struct ngx_pool_cleanup_s {
ngx_pool_cleanup_pt handler;
void *data;
ngx_pool_cleanup_t *next;
};
among them
handler: is the callback function pointer;
data: When calling back, pass this data into the callback function;
next: //Point to the next callback function structure;
If we need to add our own callback function, we need to call ngx_pool_cleanup_add to get a ngx_pool_cleanup_t, then set the handler as our cleanup function, and set data as the data we want to clean. In this way, the handler will be called cyclically to clean up the data in ngx_destroy_pool;
For example: we can mount an open file descriptor as a resource to the memory pool, and provide a function to close the file description to register on the handler, then when the memory pool is released, it will call the close file function we provide to Process file descriptor resources. 
2.5, memory pool destruction
ngx_destroy_pool
The ngx_destroy_pool function is used to destroy a memory pool:
void
ngx_destroy_pool(ngx_pool_t *pool)
{
ngx_pool_t *p, *n;
ngx_pool_large_t *l;
ngx_pool_cleanup_t *c;
//首先调用所有的数据清理函数
for (c = pool->cleanup; c; c = c->next) {
if (c->handler) {
ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, pool->log, 0,
"run cleanup: %p", c);
c->handler(c->data);
}
}
//释放所有的大块内存
for (l = pool->large; l; l = l->next) {
ngx_log_debug1(NGX_LOG_DEBUG_ALLOC, pool->log, 0, "free: %p", l->alloc);
if (l->alloc) {
ngx_free(l->alloc);
}
}
//最后释放所有内存池中的内存块
for (p = pool, n = pool->d.next; /* void */; p = n, n = n->d.next) {
ngx_free(p);
if (n == NULL) {
break;
}
}
}