Linux+Docker high memory usage troubleshooting
Check the resource usage of the docker service
docker stats
docker stop all container commands
docker stop $(docker ps -a -q)
docker start all container commands
docker start $(docker ps -a -q)
2.1 df -h View hard disk resource usage
If the server is too slowly occupied by logs or files of other services, it may also lead to failure of persistence, so check the disk usage at the first time.
use command
df -h
Check the hard disk resource usage.
The mounted disk usage is still very healthy, and this reason can be ruled out.
2.2 Top observation resource observation CPU memory
use command
top
Observing the server resource usage
Start to observe, it is obvious that the memory usage is very high,
check the memory ratio:
free -g
The memory is almost full, and the memory occupied by the buffer part is relatively high.
Clear unused page cache first
echo 1 > /proc/sys/vm/drop_caches
Commonly used clear cache commands
To free pagecache:仅清除页面缓存（PageCache）
echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:清除目录项和inode
echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:清除页面缓存，目录项和inode
echo 3 > /proc/sys/vm/drop_caches
After clearing the cache, it is found that not much memory is released, and it is judged that a program is performing a large number of file operations.
You need to locate the memory-occupied service
using the command
ps -aux | sort -k4nr | head -10
View the top ten services with memory
Since the environment deployed in the form of docker is used, the same app.jar is used in the docker file, so it is impossible to determine which service the PID corresponds to, so the corresponding service needs to be determined according to the PID
2.3 Find the corresponding docker container according to the PID
Use docker container top container_name to get the corresponding PID of the container
docker container top container_name
The corresponding command can only query the PID of the container with a known name, which is difficult to troubleshoot. It is thankless to try one by one
to get the container PID list.
docker inspect -f '{
{.State.Pid}}' $(docker ps -aq)
Among them, $(docker ps -aq) can be replaced with a specific container name, so that the effect is equivalent to the previous command
Get the corresponding service container
2.4 Final processing
Contact the development of the corresponding service. After checking, the other party said that the data report generation operation was performed when the error occurred, which caused the memory to soar. Then, the corresponding service was started to limit the memory, and the content was to be optimized later.