Problem Description
The Linux memory usage exceeds the threshold, leaving the Java application with no available memory and eventually causing the program to crash. Even if the program is stopped when the program is not hung up, the system memory will not be released.
process of finding the cause
This problem has been bothering me for several months. After analyzing it many times, I can't find the reason. I checked the problem online and others have encountered it, but there is no good solution. Because the project has not been launched, every time When the program crashes due to insufficient memory, it is enough to restart the service, and it does not take too much time to find the problem. Now that the project is launched immediately, there can be no program crash, and it is still a front-end system, let alone any problems.
At first, I always thought that the program caused the memory leak. I used the jmap -F -dump:live,format=b,file=/usr/local/sztFront/logs/heapdump.bin command that comes with jdk to output the dump file several times. , Through MemoryAnalyzer analysis, the application has no variable that consumes too much memory. The background guesses whether the log output is too much. After all, it is a front-end system . The daily packet volume is particularly large, and the log can reach 5 or 6 G. Therefore, the log received message was turned off, and the log size immediately dropped to 200M per day . The time for the program to hang up has become longer. It used to hang in about a week. Now it can take about two weeks to hang up, but it still doesn't work. After searching for various related problems on the Internet, the problem occurs that the value of Cached is too large, resulting in the system having no memory space that can be re-allocated. As long as Cached is used to cache files, frequently read and written files will be cached in Cached , which can increase the efficiency of reading and writing. This function is provided by the Linux system kernel. It is only provided in the core version after 2.6.16 , that is, the old version. operating system , such as Hongqi DC 5.0 , RHEL 4.xNone of the previous versions. This can explain why my project always hangs. My project mainly deals with files, so the received and downloaded files will be cached, and the memory will not be released. Even if the program is stopped, the memory will not be released. . Finally found three executions, which can clean up the cached memory
Three instructions:
sync
echo 1 > /proc/sys/vm/drop_caches
echo 2 > /proc/sys/vm/drop_caches
echo 3 > /proc/sys/vm/drop_caches
After executing these three instructions, check through the free -m command, the free available memory increases immediately, and the value of the buff/cache column becomes smaller, indicating that the memory has been released, but these three instructions cannot always be executed manually, so I finally wrote a shell The script, open the Linux timing task crond , check the free memory every morning , and execute these three commands when it is less than 4G (Note: the system content is 20G ).
Note: Be sure to execute the sync command before executing these three commands ( Description: The sync command runs the sync subroutine. If the system must be stopped, the sync command is run to ensure the integrity of the file system. The sync command removes all unwritten system Buffers are written to disk, containing modified i- Nodes , delayed block I/O , and read and write mapped files )
Solution (manual)
1. Modify /proc/sys/vm/drop_caches to release the cache memory space occupied by Slab (refer to the official documentation of drop_caches ):
Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free. To free pagecache: * echo 1 > /proc/sys/vm/drop_caches To free dentries and inodes: * echo 2 > /proc/sys/vm/drop_caches To free pagecache, dentries and inodes: * echo 3 > /proc/sys/vm/drop_caches As this is a non-destructive operation, and dirty objects are notfreeable, the user should run "sync" first in order to make sure allcached objects are freed. This tunable was added in 2.6.16.
Note: Execute the sync command before executing these three commands
solution (automatic)
1. Write the shell timed task script freemem.sh
#! /bin/sh used=`free -m | awk 'NR==2' | awk '{print $3}'` free=`free -m | awk 'NR==2' | awk '{print $4}'` echo "===========================" >> /app/memory/logs/mem.log date >> /app/memory/logs/mem.log echo "Memory usage before | [Use:${used}MB][Free:${free}MB]" >> /app/memory/logs/mem.log if [ $free -le 4000 ] ; then sync && echo 1 > /proc/sys/vm/drop_caches sync && echo 2 > /proc/sys/vm/drop_caches sync && echo 3 > /proc/sys/vm/drop_caches used_ok=`free -m | awk 'NR==2' | awk '{print $3}'` free_ok=`free -m | awk 'NR==2' | awk '{print $4}'` echo "Memory usage after | [Use:${used_ok}MB][Free:${free_ok}MB]" >> /app/memory/logs/mem.log echo "OK" >> /app/memory/logs/mem.log else echo "Not required" >> /app/memory/logs/mem.log be exit 1
2. Use the crontab -e command to edit the current user's crontab
0 6 * * * /usr/local/tomcat/sztFileFront/bin/freemem.sh
Timed task writing reference: http://www.jb51.net/article/15008.htm
3. Restart the crond service
/sbin/service crond restart
4. Check whether the crond service is restarted successfully
/sbin/service crond status
Finally, problem solved. The scheduled task I set is to execute the freemem.sh script every day at 6 am.
Original reference: http://blog.csdn.net/u014740338/article/details/66975550