Insufficient server space processing and solution ideas - the actual combat docker takes up too much space

foreword

The server Centos operating system, the problem of insufficient space has been dealt with three times, and I decided to record its solution ideas and processing process. Running out of server space is a common problem, especially with large applications and websites. When a server runs out of space, there are steps you should take to handle and resolve the issue. The following records the latest solutions and ideas (reason: docker takes up too much space)

Phenomenon

Insufficient server space will cause the following symptoms:

1. Applications not running : Some applications may not start or function properly when the server is running low on space because they need to read or write data on the hard drive.
2. File upload failure : If the server has run out of space, users may not be able to upload files because there is not enough space to store them.
3. Database errors : When the server runs out of space, database errors may occur because the database needs to write and read data to disk.
4. Log files cannot be written : When the server space is insufficient, the log files of the system and applications may not be written, which will make troubleshooting and system operation monitoring difficult.
5. System Crash : If the server runs out of space and cannot process data and tasks, it may cause a system crash or downtime.

Therefore, maintaining sufficient server space is an important factor to ensure stable and efficient server operation.

resolution process

1. Locating the problem of insufficient server space

1. Log in to the server, and you can check the disk usage on the server by running commands such as df -hand to see which directories or folders are taking up a lot of space.du -h

1   [root@luntek tmp]# df -h
2   Filesystem      Size  Used Avail Use% Mounted on
3   devtmpfs        7.8G     0  7.8G   0% /dev
4   tmpfs           7.8G     0  7.8G   0% /dev/shm
5   tmpfs           7.8G  827M  7.0G  11% /run
6   tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
7   /dev/vda1        40G   38G     0 100% /
8   /dev/vdb        656G  154G  469G  25% /mnt
9   tmpfs           1.6G     0  1.6G   0% /run/user/0
10   overlay          40G   38G     0 100% /var/lib/docker/overlay2/0e4870d22fb 1c79fd3c0b8ad2023475715a5a56c05b1900de76070d0a80e3001/merged
11   overlay          40G   38G     0 100% /var/lib/docker/overlay2/cd5f647aa8a eede41e2642d56dd346632718c8bc9b07e7bbd7f1cc6dcdea23b6/merged
12   overlay          40G   38G     0 100% /var/lib/docker/overlay2/f7eb94f2fd3 c4b37de3538f965ce64833b5b69a59616ef317833c7039ef5da78/merged
13   overlay          40G   38G     0 100% /var/lib/docker/overlay2/da95b27c0e8 25567df0e483f0c5bb2ba58768604c867b4ceb45928beeef015f2/merged
14   overlay          40G   38G     0 100% /var/lib/docker/overlay2/4fea6e878f2 f317a63d124c2a3831cd9b8ffcdd0bb249b73b39d92adc7d555eb/merged
15   overlay          40G   38G     0 100% /var/lib/docker/overlay2/b910801159c 76ffe6ae68c5596aa13994ff5e9d2eebd2e1febb147b3b10cd2ac/merged
16   overlay          40G   38G     0 100% /var/lib/docker/overlay2/ef09bb54935 3ca0af146aa854b48d7757407cb7429deae1e20e107f40b78796e/merged
17   overlay          40G   38G     0 100% /var/lib/docker/overlay2/d1b236f57e4 09d49d0d7ef0ecf21c3d684ed8317bb88ef530907c4c4e5e57799/merged
18   overlay          40G   38G     0 100% /var/lib/docker/overlay2/ae369ee56f0 8206507b53cb29cbc32e88198f9d38ba1331636a7970a338f30ad/merged
19   overlay          40G   38G     0 100% /var/lib/docker/overlay2/3f84613ee8d b82cf00fc2883b9e31b42766a5d7d1cc910f2a7b9fce6cf74c435/merged
20   overlay          40G   38G     0 100% /var/lib/docker/overlay2/6e2272faed5 9954f4a29f60a0ba6618eede5ccaad24ff79385d98cd2e7c3d0da/merged
21   overlay          40G   38G     0 100% /var/lib/docker/overlay2/3c8f75522e7 2d64eb20b4656cbd11358581a61619ad209f788f3eac03ce0ed15/merged
22   overlay          40G   38G     0 100% /var/lib/docker/overlay2/832ed62f362 edd6d0350828aac3aad454a5dc1679a435d3be0e8efd60e6d25b2/merged
23   overlay          40G   38G     0 100% /var/lib/docker/overlay2/d96561964a2 015583c62329eac1587c3ff4b0a962ed9d30a2dd47609c3cfcc67/merged
24   shm              64M     0   64M   0% /var/lib/docker/containers/dd2e19f40 abcfafe515db471ab89735b79e05b1ca5e97ede16e9074cd53d35a6/mounts/shm
25   shm              64M     0   64M   0% /var/lib/docker/containers/23a9c891d cc3696c294fa9b2867ac96eda72b1422ffb6bc1527c2f00d5ec7f15/mounts/shm
26   shm              64M     0   64M   0% /var/lib/docker/containers/833eae2de d2733e5b3daddc1bcae9ea7ba943e1a1a2bc421d1119265db45fa80/mounts/shm
27   shm              64M     0   64M   0% /var/lib/docker/containers/7ae786e73 c5cd21c8a00d531f73902374150cbb8d6a3a10139324b9924375900/mounts/shm
...

2. Grab the main physical device and don't care about other logical devices. Physical devices are generally mounted in the /dev directory

/dev/vda1         40G   38G     0 100% /
/dev/vdb         656G  154G  469G  25% /mnt

It is observed that /the root directory space has been used to 100%. Although there is still 2G space in name, this space may be occupied by the memory swap partition, so don’t stick to the specific number. If it shows 100%, you can confirm it.

注意: If the device space is seriously insufficient, normal command lines may not be able to be executed. At this time, you can try the following ideas for freeing up space first.

2. Free up space

1. First of all, the safest space that can be released is often in /homethe directory. In general, temporary files and installation packages uploaded by users are often stored in the /home directory, so these files can be deleted first or transferred to other places with space. partition to free up space. However, it should be noted that if these files are necessary files for some applications, deletion may cause the application to fail to run normally, so confirmation is required before deletion. In addition, if there are users' personal data (such as documents, music, pictures, etc.) in the /home directory, you should not delete or transfer these files easily, but you should look for other space release solutions.
2. /mdSome files in the directory may be deleted, but some files may record the login and temporary state data of the currently running process. If you are not familiar with these services, it is not recommended to delete these files.
3. You can free up space by cleaning up unused files and directories, and deleting expired logs and backup files. Additionally, you can reduce the disk space your files take up by compressing and archiving them. Before freeing up space, you need to make sure you don't delete important files and directories by mistake.
4. Restarting some processes and common services can release the space occupied by temporary files generated by these processes. Doing so will basically guarantee that some disk space will be reclaimed after a reboot.
5. Clean up useless process images or deleted Dockercontainers .
6. Clean up the process's cache, you can run the process's system commands, and clear all unused objects (including images, containers, networks, and volumes), as well as all unmarked images and build caches. This can help free up more disk space.

Third, analyze the reasons for space occupation

Use ducommands to analyze the space occupied by folders layer by layer. Among them, -d 1means to analyze only the first-level subdirectories under the current directory, -hmeans to display the results in an easy-to-read way (for example, using KB, MB, etc. units), -p Lmeans to only analyze L-level subdirectories, and -umeans to use M to display the results, that is in megabytes.

du -d 1 -h

For example, from the above analysis, it can be seen that /var/docker/overlay2a directory occupies a directory 28G, Docker Overlay2which is a container storage driver that uses a special file system to manage the storage of Docker containers and images. Overlay2The driver stacks multiple mirror layers together so that reads and writes to the layers only need to look for the topmost layer. Therefore, when using Overlay2the driver, Docker creates a separate directory for each container that contains all the filesystem content needed by the container.
分析结果: There is basically no solution. You cannot delete Overlay2the folder to free up space. At this time, all you can do is move the docker installation environment to a /mntpartition with space.

4. Docker Migration

Moving the Docker directory (/var/lib/docker by default) to another disk can be achieved by following steps:

1. Stop all running Docker containers: Run docker stop $(docker ps -aq)the command to stop all running containers.

docker stop $(docker ps -aq)

2. Stop the Docker service: Run systemctl stop dockerthe command to stop the Docker service.

systemctl stop docker

3. Copy the Docker directory to a new location: Use tools such as cp or rsync to copy the Docker directory to a new location. For example, to copy the Docker directory from /var/lib/docker to /new/path/docker, you can run sudo rsync -aqxP /var/lib/docker/ /mnt/var/lib/docker the command.

sudo rsync -aqxP /var/lib/docker/ /mnt/var/lib/docker

4. Update Docker configuration file: Modify the Docker service configuration file /etc/docker/daemon.jsonand replace the old Docker directory path with the new location. If the file does not exist, create it. A sample configuration file looks like this:

{
    
    
"registry-mirrors": ["https://nrbewqda.mirror.aliyuncs.com"],
"dns": ["100.125.1.250", "100.125.64.250"],
"data-root": "/mnt/var/lib/docker"
}

5. Start the Docker service: Run systemctl start dockerthe command to start the Docker service and check if the new Docker directory has loaded successfully.

systemctl start docker

6. Start Docker Containers: Run docker start <container_id>the command to start all containers that were previously stopped.
   注意: If the docker container is set to restart, the docker itself will automatically start, this step is not necessary.

docker start <container_id>

重点注意: After the migration is complete, do not delete the original docker directory immediately. You can first isolate the original docker directory by modifying the directory name, and then delete it after repeatedly confirming that there is no problem with the startup.

cd /var/lib/docker/
mv docker docker2
mv docker2 /mnt/

Summarize

拓展: If the docker resources are too large, they can be migrated to other servers or external storage devices. Migration can be achieved by exporting and importing docker images . Alternatively, other tools such as rsync or scp can be used to migrate files and directories. It should be noted that data integrity and availability need to be guaranteed during the migration process.

总之, when the server runs out of space, there are steps to be taken to handle and resolve the issue. First, you need to locate the cause of insufficient space and free up space as much as possible. If the problem still exists, you need to further analyze the cause of space occupation, and consider adjusting related settings or migrating. Finally, there is a need to ensure data integrity and availability while addressing issues.