memory allocation process
When an application applies for memory through the malloc function, it actually applies for virtual memory, and does not allocate physical memory at this time.
When the application program reads and writes this virtual memory, the CPU will access the virtual memory. At this time, it will be found that the virtual memory is not mapped to the physical memory, and the CPU will generate a page fault interrupt , and the process will switch from the user mode to the kernel mode. , and hand over the page fault interrupt to the Page Fault Handler (page fault interrupt function) of the kernel for processing.
The page fault interrupt processing function will check whether there is space for physical memory, if so, directly allocate physical memory, and establish a mapping relationship between virtual memory and physical memory.
If there is no free physical memory, the kernel will start to reclaim memory. There are two main ways of recycling: direct memory reclaim and background memory reclaim.
- Background memory reclamation : When the physical memory is tight, the kswapd kernel thread will be awakened to reclaim memory. This process of reclaiming memory is asynchronous and will not block the execution of the process.
- Direct memory reclamation : If the background asynchronous reclamation cannot keep up with the speed of process memory application, it will start to reclaim directly. This process of reclaiming memory is synchronous and will block the execution of the process.
If the free physical memory still cannot satisfy the application for physical memory after direct memory recovery, the kernel will trigger the OOM (out of memory) mechanism
The OOM Killer mechanism will select a process that occupies high physical memory according to the algorithm, and then kill it to release memory resources. If the physical memory is still insufficient, OOM Killer will continue to kill the process that occupies high physical memory until enough is released. memory location.
The process of applying for physical memory is as follows:
What memory can be reclaimed?
When the system memory is tight, the work of reclaiming memory will be carried out. What specific memory will be reclaimed?
There are two main types of memory that can be reclaimed, and they are reclaimed in different ways.
- File page : Both the disk data cached by the kernel and the file data cached by the kernel are called file pages. Most of the file pages can be directly released from the memory, and then re-read from the disk when needed in the future. And those data that have been modified by the application and have not yet been written to the disk (that is, dirty pages) must be written to the disk before the memory can be released. Therefore, the way to reclaim clean pages is to release memory directly, and the way to reclaim dirty pages is to write them back to disk first and then release memory.
- Anonymous page : This part of memory has no actual carrier, unlike the file cache that has a carrier such as hard disk files, such as heap and stack data. This part of the memory is likely to be accessed again, so the memory cannot be released directly. They are recycled through the Swap mechanism of Linux . Swap will first write the infrequently accessed memory to the disk, and then release the memory for other More demanding processes use. When accessing these memories again, it is enough to re-read the memory from the disk.
The recovery of file pages and anonymous pages is based on the LRU algorithm, that is, memory that is not frequently accessed is preferentially recovered. The LRU recycling algorithm actually maintains two doubly linked lists, active and inactive, where:
- active_list Active memory page linked list, where the most recently accessed (active) memory pages are stored
- inactive_list Inactive memory linked list, here are rarely accessed (inactive) memory pages
The closer to the end of the linked list, the less frequently the memory page is accessed. In this way, when reclaiming memory, the system can give priority to reclaiming inactive memory according to the degree of activity.
Performance Impact of Reclaiming Memory
There are two ways to reclaim memory:
- One is background memory recovery, which is to wake up the kswapd kernel thread. This method is asynchronous and will not block the process
- One is direct memory reclamation. This method is synchronous reclaiming, which will block the process, which will cause a long delay, and the CPU utilization of the system will increase, which will eventually cause the system load to soar.
The types of memory that can be reclaimed include file pages and anonymous pages:
- Recycling of file pages: For clean pages, the memory is directly released. This operation will not affect performance. For dirty pages, it will be written back to the disk first and then the memory will be released. This operation will cause disk I/O, and this operation will affect system performance. .
- Recycling of anonymous pages: If the Swap mechanism is enabled, the Swap mechanism will swap out the anonymous pages that are not frequently accessed to the disk, and then swap them from the disk into the memory the next time they are accessed. This operation will affect system performance. .
It can be seen that the operation of reclaiming memory will basically cause disk I/O. If the operation of reclaiming memory is very frequent, it means that there will be a lot of disk I/O times. This process will inevitably affect the performance of the system. The whole system feels It is very stuck.
How to protect a process from being killed by OOM?
In the case of insufficient free memory in the system, the process has applied for a large amount of memory. If there is not enough memory to reclaim the memory directly, the OOM mechanism will be triggered, and the kernel will select a process to kill according to the algorithm.
The result of the process score is affected by the following two aspects:
- First, the number of physical memory pages used by the process
- Second, the OOM calibration value oom_score_adj for each process. It is configurable via /proc/[pid]/oom_score_adj . We can set any value between -1000 and 1000 to adjust the probability of the process being OOM Killed.
The final calculation method in the function oom_badness() is:
// points 代表打分的结果
// process_pages 代表进程已经使用的物理内存页面数
// oom_score_adj 代表 OOM 校准值
// totalpages 代表系统总的可用页面数
points = process_pages + oom_score_adj*totalpages/1000Multiply [the total number of available pages in the system] by [OOM calibration value oom_score_adj] and divide by 1000, and finally add the number of physical pages used by the process. The larger the calculated value, the higher the chance of the process being OOM Killed bigger .
The default value of oom_score_adj of each process is 0, so the final score is related to the memory consumed by the process itself. The larger the consumed memory, the easier it is to be killed. We can modify the scoring result of the process by adjusting the value of oom_score_adj:
- If you do not want a process to be killed first, you can adjust the oom_score_adj of the process to change the score of the process and reduce the probability of the process being killed by OOM.
- If you want a program not to be killed anyway, you can configure oom_score_adj to -1000.
We'd better configure the oom_score_adj of some very important system services to -1000, such as sshd, because once these system services are killed, it will be difficult for us to log in to the system again.
However, it is not recommended to configure the oom_score_adj of our own business program to -1000, because once the business program has a memory leak and it cannot be killed, this will cause the OOM killer to stop as its memory overhead increases. It is woken up to kill other processes one by one.
Summarize
When the kernel allocates physical memory to the application, if there is not enough free physical memory, it will perform memory recovery. There are two main methods:
- Background memory reclamation: When the physical memory is tight, the kswapd kernel thread will be awakened to reclaim memory. The process of reclaiming memory is asynchronous and will not block the execution of the process
- Direct memory reclamation: If the background asynchronous reclamation cannot keep up with the speed of process memory application, it will start to reclaim directly. This process of reclaiming memory is synchronous and will block the execution of the process.
The types of memory that can be reclaimed include file pages and anonymous pages:
- Recycling of file pages: For clean pages, the memory is released directly, this operation will not affect performance, but for dirty pages, it will be written back to disk first and then memory will be released. This operation will cause disk I/O, and this operation will affect system performance.
- Recycling of anonymous pages: If the Swap mechanism is enabled, the Swap mechanism will swap out anonymous pages that are not frequently accessed to the disk, and then swap them from the disk into the memory the next time they are accessed. This operation will affect performance
The recovery of file pages and anonymous pages is based on the LRU algorithm, that is, memory that is not frequently accessed is preferentially recovered. The operation of reclaiming memory will basically cause disk I/O. If the operation of reclaiming memory is very frequent, it means that there will be a lot of disk I/O. This process will inevitably affect the performance of the system.
Common solutions to the performance impact caused by reclaiming memory:
- Set /proc/sys/vm/swappiness, adjust the recycling tendency of file pages and anonymous pages, and try to recycle file pages as much as possible;
- Set /proc/sys/vm/min_free_kbytes to adjust the timing of kswapd kernel thread asynchronously reclaiming memory;
- Set /proc/sys/vm/zone_reclaim_mode to adjust the memory recovery strategy under the NUMA architecture. It is recommended to set it to 0, so that before reclaiming the local memory, it will look for free memory in other Nodes, so as to avoid the situation that the system still has a lot of free memory , due to insufficient local memory of the local Node, frequent direct memory reclamation causes performance degradation;
If the free physical memory size is still not enough after direct memory reclamation, the OOM mechanism will be triggered, and the OOM killer will score each process according to the memory usage and oom_score_adj, and the process with the highest score will be killed first Lose.
You can reduce the chance of being killed by the OOM killer by adjusting the value of oom_score_adj.