The Linux server Cache occupies too much memory, resulting in insufficient system memory and the final java application crash solution

Problem Description

The Linux memory usage exceeds the threshold, leaving the Java application with no available memory and eventually causing the program to crash. Even if the program is stopped when the program is not hung up, the system memory will not be released.

 

process of finding the cause

This problem has been bothering me for several months. After analyzing it many times, I can't find the reason. I checked the problem online and others have encountered it, but there is no good solution. Because the project has not been launched, every time When the program crashes due to insufficient memory, it is enough to restart the service, and it does not take too much time to find the problem. Now that the project is online immediately, there can be no program crash, and it is still a front-end system, let alone any problems.

At first, I always thought that the program caused the memory leak. I used the jmap -F -dump:live,format=b,file=/usr/local/sztFront/logs/heapdump.bin command that comes with jdk to output the dump file several times. , Through MemoryAnalyzer analysis, the application has no variable that consumes too much memory. The background guesses whether the log output is too much. After all, it is a front-end system . The daily packet volume is particularly large, and the log can reach 5 or 6 G. Therefore, the log received message was turned off, and the log size immediately dropped to 200M per day . The time for the program to hang up has become longer. It used to hang in about a week. Now it can take about two weeks to hang up, but it still doesn't work. After searching for various related problems on the Internet, the problem occurs that the value of Cached is too large, resulting in the system having no memory space that can be re-allocated. As long as Cached is used to cache files, frequently read and written files will be cached in Cached , which can increase the efficiency of reading and writing. This function is provided by the Linux system kernel. It is only provided in the core version after 2.6.16 , that is, the old version. operating system , such as Hongqi DC 5.0 , RHEL 4.xNone of the previous versions. This can explain why my project always hangs. My project mainly deals with files, so the received and downloaded files will be cached, and the memory will not be released. Even if the program is stopped, the memory will not be released. . Finally found three executions, which can clean up the cached memory

Three instructions:

sync

echo 1 > /proc/sys/vm/drop_caches
echo 2 > /proc/sys/vm/drop_caches
echo 3 > /proc/sys/vm/drop_caches

After executing these three instructions, check through the free -m command, the free available memory increases immediately, and the value of the buff/cache column becomes smaller, indicating that the memory has been released, but these three instructions cannot always be executed manually, so I finally wrote a shell The script, open the Linux timing task crond , check the free memory every morning , and execute these three commands when it is less than 4G (Note: the system content is 20G ).

Note: Be sure to execute the sync command before executing these three commands ( Description: The sync command runs the sync subroutine. If the system must be stopped, the sync command is run to ensure the integrity of the file system. The sync command removes all unwritten system Buffers are written to disk, containing modified i- Nodes , delayed block I/O , and read and write mapped files )

 

Solution (manual)

1. Modify /proc/sys/vm/drop_caches to release the cache memory space occupied by Slab (refer to the official documentation of drop_caches ):

Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.  
To free pagecache:  
* echo 1 > /proc/sys/vm/drop_caches  
To free dentries and inodes:  
* echo 2 > /proc/sys/vm/drop_caches  
To free pagecache, dentries and inodes:  
* echo 3 > /proc/sys/vm/drop_caches  
As this is a non-destructive operation, and dirty objects are notfreeable, the user should run "sync" first in order to make sure allcached objects are freed.  
This tunable was added in 2.6.16.  

 Note: Execute the sync command before executing these three commands

 

solution (automatic)

1. Write the shell timed task script freemem.sh

 

#! /bin/sh  
used=`free -m | awk 'NR==2' | awk '{print $3}'`  
free=`free -m | awk 'NR==2' | awk '{print $4}'`  
echo "===========================" >> /app/memory/logs/mem.log  
date >> /app/memory/logs/mem.log  
echo "Memory usage before | [Use：${used}MB][Free：${free}MB]" >> /app/memory/logs/mem.log  
if [ $free -le 4000 ] ; then  
                sync && echo 1 > /proc/sys/vm/drop_caches  
                sync && echo 2 > /proc/sys/vm/drop_caches  
                sync && echo 3 > /proc/sys/vm/drop_caches  
                used_ok=`free -m | awk 'NR==2' | awk '{print $3}'`  
                free_ok=`free -m | awk 'NR==2' | awk '{print $4}'`  
                echo "Memory usage after | [Use：${used_ok}MB][Free：${free_ok}MB]" >> /app/memory/logs/mem.log  
                echo "OK" >> /app/memory/logs/mem.log  
else  
                echo "Not required" >> /app/memory/logs/mem.log  
be  
exit 1

 2. Use the crontab -e command to edit the current user's crontab

 

0 6 * * * /usr/local/tomcat/sztFileFront/bin/freemem.sh  

 Timed task writing reference: http://www.jb51.net/article/15008.htm

 

3. Restart the crond service

 

/sbin/service crond restart  

 4. Check whether the crond service is restarted successfully

 

/sbin/service crond status  

 

Finally, problem solved. The scheduled task I set is to execute the freemem.sh script every day at 6 am.

 

 

 

Original reference: http://blog.csdn.net/u014740338/article/details/66975550