Recently, in the daily development of linux, to maintain the previously released programs, only one of the several servers is restarting, and it is not regular, and there is no signal signal and the reason for the service to hang up when checking the service log. I have no choice but to suspect that the linux system gave The service is dead. So I want to check the log of the service being killed, and look at the system log to judge.
That is, the messages file in the directory on the way, you can filter the information through commands to check whether it is killed by the system.
Check the system log information with the following command!
egrep -i 'killed process' /var/log/messagesThe command output is as follows:
Parameter analysis:
total-vm: total virtual memory. The total virtual memory used by the process.
rss: resident set size. Resident set size. The resident set is the collection of pages that a process has loaded into memory.
anon-rss: anonymous rss. Anonymous resident set. For example, malloc is anonymous.
file-rss: memory pages mapped to devices and files.
shmem-rss: Probably shared memory rss?
From the above output, it can be seen that it was killed by the system! The time can also be matched, and it was killed at 9:52. So far, the problem has been found out, which is caused by insufficient server memory! ! ! Prepare to modify the service weight of this server, and assign as few tasks as possible to reduce the pressure on this server.
The Linux kernel has a mechanism called OOM killer (Out Of Memory killer), which monitors those processes that occupy too much memory, especially those processes that occupy memory very quickly, and then automatically kill the process to prevent memory exhaustion. The kernel detects that the system memory is insufficient, selects and kills a process, you can refer to the kernel source code linux/mm/oom_kill.c, when the system memory is insufficient, out_of_memory() is triggered, and then call select_bad_process() to select a " bad" process is killed. How to judge and select a "bad" process? Linux selects a "bad" process by calling oom_badness(). The selection algorithm and idea are very simple and simple: the most bad process is the one that occupies the most memory.
basic concept
The Linux kernel has a mechanism called OOM killer (Out-Of-Memory killer). This mechanism will monitor those processes that occupy too much memory, especially those processes that consume a lot of memory in an instant. In order to prevent the memory from running out, the kernel will kill the process. Lose.
Oom killer mechanism analysis
Oom killer calculates which process to choose to kill? Let's take a look at some parameters provided by the kernel to user mode under /proc: /proc/[pid]/oom_adj, the weight of the pid process killed by the oom killer, between [-17,15], the more A high weight means that it is more likely to be selected by the oom killer, and -17 means that it is forbidden to be killed. /proc/[pid]/oom_score, the current kill score of the pid process. The higher the score, the more likely it is to be killed. This value is calculated according to oom_adj and is the main reference of oom_killer.
There are 2 configurable options under sysctl:
- vm.panic_on_oom = 0 #Whether the kernel panics directly when the memory is not enough
- vm.oom_kill_allocating_task = 1 #oom-killer whether to select the process that is currently applying for memory to kill
- If you want to modify the probability of being selected by oom killer, prohibit or give oom_adj the smallest or smaller value, you can also adjust the behavior of oom killer through sysctl
Can you protect important processes from being killed by OOM Killer? Really can!
OOM Killer will check the /proc/$pid/oom_score_adj file to adjust the final score (oom_score). So we can reduce the possibility of the process being selected and terminated by giving a large negative number in this file. oom_score_adj can vary from -1000 to 1000. If you give -1000, the process will not be killed by the OOM Killer even if it uses 100% of the memory. You can modify oom_score_adj (for example, set it to -200) by the following command: