Some time ago, a colleague in the company encountered such a problem. System A calls system B through RPC. Before the call, the document will be placed locally, which is in the same transaction as RPC. Later, application A hung up, which caused the problem that it was found that B returned to system A, and the relevant pipeline could not be found. Finally, it was found that under the condition of tight memory, Linux would kill the process that occupies the most, resulting in the rollback of database transactions. , but the RPC call succeeds, an inconsistency that is easily overlooked.
Linux OOM-killer
There was a 502 error on the front antenna. After investigation, a developer wrote a sexy program that ate up most of the memory, triggering the OOM-killer to kill Nginx
Check the log /var/log/messages and find errors like "Out of Memory: Kill process..."
I didn't know about OOM-killer{out of memory killer} before, but I happened to have a day off today, so I can learn and improve my posture!
OOM_killer is a way for Linux to protect itself. When the memory is insufficient, it will not cause too serious problems. It is a bit like a strong man breaking his wrist.
In kernel 2.6, insufficient memory will wake up oom_killer, pick out the one with the largest /proc/<pid>/oom_score and kill it
In order to protect important processes from being dropped by oom-killer, we can: echo -17 > /proc/<pid>/oom_adj, -17 means to disable OOM
We can also disable OOM for the entire system:
sysctl -w vm.panic_on_oom=1
sysctl -p
It is worth noting that sometimes there is remaining memory when free -m is used, but OOM-killer will still be triggered, probably because the process occupies a special memory address
Usually we should pay attention to the memory usage of the newly incoming process, so as not to be innocently implicated in the important business process of the system
You can use top M to view the processes that consume the most memory, but it is not that the oom_killer will be triggered once the process exceeds
The parameter /proc/sys/vm/overcommit_memory can control the process's strategy for overcommitting memory
When overcommit_memory=0 allows the process to overcommit memory slightly, but not for a large number of overloaded requests
When overcommit_memory=1 always allow the process to overcommit
Overcommit is always prohibited when overcommit_memory=2