Leveldb source code analysis 2.1-the core design of the memory pool implementation
introduction
In order to use memory more reasonably, a custom memory pool is implemented in leveldb.
1 data structure
Leveldb's memory pool implementation class is Arena, and its core data fields are as follows:
First, use a vector type of blocks_ to store the allocated blocks of memory; use alloc_ptr_ to point to the first address of the unused memory in the newly applied memory; use alloc_bytes_remaining_ Indicates the byte size of unused memory in the newly applied memory; in particular, memory_usage_ counts the sum of memory occupied by all bock and blocks_ variables.
2 Source code interpretation
2.1 Source code interpretation
There are Allocate and AllocateAligned methods to apply for memory, the latter has more start address alignment function.
- AllocateAligned function
char* Arena::AllocateAligned(size_t bytes) {
//进行最低8字节的对齐
const int align = (sizeof(void*) > 8) ? sizeof(void*) : 8;
static_assert((align & (align - 1)) == 0,
"Pointer size should be a power of 2");
size_t current_mod = reinterpret_cast<uintptr_t>(alloc_ptr_) & (align - 1);
size_t slop = (current_mod == 0 ? 0 : align - current_mod);
size_t needed = bytes + slop;
char* result;
if (needed <= alloc_bytes_remaining_) {
//如果最新内存块有空余空间
result = alloc_ptr_ + slop; //移动对齐补全字节
alloc_ptr_ += needed; //重置未使用内存指针
alloc_bytes_remaining_ -= needed; //重置未使用内存大小
} else {
//如果最新内存块没有空余空间
result = AllocateFallback(bytes);
}
assert((reinterpret_cast<uintptr_t>(result) & (align - 1)) == 0);
return result;
}
- AllocateFallback function
char* Arena::AllocateFallback(size_t bytes) {
if (bytes > kBlockSize / 4) {
// 如果申请内存大小大于1K,那么进入下面,申请bytes大小然后直接使用
char* result = AllocateNewBlock(bytes);
return result;
}
// 如果申请内存大小大于1K,直接申请4K大内存后,还需要重置alloc_ptr_和alloc_bytes_remaining_
alloc_ptr_ = AllocateNewBlock(kBlockSize);
alloc_bytes_remaining_ = kBlockSize;
char* result = alloc_ptr_;
alloc_ptr_ += bytes;
alloc_bytes_remaining_ -= bytes;
return result;
}
- AllocateNewBlock function
char* Arena::AllocateNewBlock(size_t block_bytes) {
char* result = new char[block_bytes];
blocks_.push_back(result);
// 加sizeof(char*)是因为blocks_还占用一个指针大小的存储空间
memory_usage_.fetch_add(block_bytes + sizeof(char*),
std::memory_order_relaxed);
return result;
}
2.2 Found the problem
By reading the Arena source code, memory will be wasted in the following scenarios:
- First, call AllocateAligned to apply for 512Byte memory, then a large 4K memory will be applied, and the remaining 3.5K memory can be allocated;
- Then, call AllocateAligned to apply for 512Byte memory 6 times, and finally the remaining 512Byte memory can be allocated;
- Continue to apply for 600Byte of memory. At this time, a new piece of 4K of large memory will be new. At this time, alloc_ptr_ and alloc_bytes_remaining_ record the free space of this 4K new large memory. The remaining 512Bytes in the previous 4K will be wasted .