[Problem]
Each business system that uses redis suddenly hangs up and reports an error:
MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
Check the redis log for an error:
Can't save in background: fork: Cannot allocate memory
Check the monitoring of redis, the memory space occupied by the process of redis itself has been stable at about 1.4G, and there is no major change
[Reason]
The RDB persistence scheme of redis will fork a process to save the data to the hard disk every time it is persisted; this error is due to insufficient memory, which leads to the failure of the fork, and the failure to save the data to the hard disk, and finally the command used to modify the data is disabled.
[Solutions]
1. The fastest way: Kill other processes on the machine to free up enough memory space for redis 2.
Temporary solution: Change the redis configuration item stop-writes-on-bgsave-error to no, the default value is yes; by default, if there is a problem in the RDB snapshots persistence process, after setting this parameter, Redis does not allow users to perform any update operations 3. Thorough solution: Add vm to the /etc/sysctl.conf file .overcommit_memory=1, execute sysctl -p to make it
effective
About vm.overcommit_memory parameter
1: Indicates that the kernel allows overuse of memory until it runs out
2: Indicates that the kernel will never overuse memory, that is, the entire memory space of the system cannot exceed the RAM value of swap+50%, 50% is the default value of overcommit_ratio, and this parameter supports modification
Linux responds "yes" to most requests for memory, so that more and larger programs can be run. Because after applying for memory, the memory will not be used immediately, and the free memory that will not be used is allocated to other programs to improve memory utilization. This technology is called Overcommit. Under normal circumstances, when all the programs do not use all the memory you have applied for, the system will not have problems. However, if the program runs, the memory required is getting larger and larger. Within the size range of the application, it will continue to occupy more memory until it exceeds the physical memory. When Linux finds that the memory is insufficient, an OOM killer (OOM=out-of-memory) will occur. It will choose to kill some processes (user mode processes, not kernel threads, which occupy more memory and have shorter running time, are more likely to be killed) in order to release memory.
Reference: https://redis.io/topics/admin