foreword
With more and more pods running services in the K8s cluster, the running time is getting longer and longer, and the disk space occupied by the docker directory is also gradually increasing. Finally, the disk is completely full recently, causing the container to fail to start and prompting "no device space left ". In order to prevent the situation that docker occupies insufficient disk capacity in the future, it is decided to migrate the docker data directory.
1. docker data migration
Docker data is stored in the /var/lib/docker directory by default. The actual storage directory can refer to docker infothe output
Client:
...
Server:
...
Docker Root Dir: var/docker/lib
...
migration steps
1.1. Stop dockerd service
systemctl stop docker
1.2. Create a new directory to migrate data
Using another mount disk, use the cp command to copy data
cp -a /var/lib/docker /mnt/data/
Note that you must add -p or use -a here, otherwise the permissions of the copied files will be incorrect, and some containers will report errors when they start. If the p parameter is not added, the /tmp directory (others && group) of the migrated directory may lose the w (write) permission. Lesson: The mongo container needs to create a ****.sock listening request in the /tmp directory. Since the p parameter is not added to the cp, the /tmp directory changes from drwxrwxrwt permission to drwxr-xr-t permission, and cannot be successfully started. The t permission is formed by x+SBIT.
Here is an example: After changing the docker working directory, harbor starts abnormally, and the error is as follows
rsyslogd: run failed with error -3000 (see rsyslog.h or try https://www.rsyslog.com/e/3000 to learn what that number means) rsyslogd: error writing pid file (creation stage) : Permission denied
-
solution
All containers need to be deleted, and the image has already been downloaded and run again (the data needs to be mounted outside, and the data will not be lost)
1.3. Edit docker configuration file
Edit the configuration file to add a line"data-root": "/mnt/data/docker",
vim /etc/docker/daemon.json
{
"registry-mirrors": [],
"data-root": "/mnt/data/docker"
}
Or soft link the new directory to the original directory (the original directory needs to be deleted or the ln command plus the f parameter is forcibly removed)
ln -s /mnt/data/docker /var/lib/docker
1.4. Restart the dockerd service
systemctl start docker
Don't rush to restart, look at docker log optimization
1.5. Check whether it is successful
- Use the docker info command to see if the Docker Root Dir parameter has changed to a new directory
- Check whether all the containers are started. If not, you can use the ps -a command to check the exit container and execute docker start {container_id} or docker start to restart
docker ps -a --filter 'status=exited' --format '{ {.ID}}'all exited containers (this operation will restart temporary containers that have exited but have not been cleared, please refer to the actual case filter operation)
2. Docker log optimization cutting
If Docker is not rebuilt, the log files will be appended by default, which will slowly fill up the hard disk space of the server after a long time. In fact, the logs printed by our commonly used docker logs command will be printed to the files in this directory.
2.1 View docker default container log driver
[root@xx-xx log]# docker info|grep 'Logging Driver'
Logging Driver: json-file
2.2 View the log driver used by a running docker container
[root@xx-xx log]# docker inspect -f '{
{.HostConfig.LogConfig.Type}}' 96a7b67e2581
json-file
2.3 View the log path used by a running docker container
[root@xx-xx log]# docker inspect -f '{
{.LogPath}}' 4ba762e36fca
/mnt/data/docker/containers/4ba762e36fca30509f6052e3374f6975cf13d4066e8cdbe03f832302c80dd3d0/4ba762e36fca30509f6052e3374f6975cf13d4066e8cdbe03f832302c80dd3d0-json.log
You can see that the files here are the logs printed by the docker terminal. If all the dockers have been running for a long time without restarting, the log will become bigger and bigger. If you don’t clean up the disk, it will definitely explode.
2.4 Use –log-driver when the container starts to specify that it uses a different log driver from the Docker container daemon
docker run -it --log-driver none xxx
2.5 Drivers supported by containers:
| driver name | describe |
|---|---|
| none | Running containers have no logs, docker logs have no output |
| local | Logs are stored in a custom format designed for minimal overhead |
| json-file | The log format is json, the default logging driver for docker |
| syslog | Write log messages to syslog. The syslog daemon must be running on the host |
| journaled | Write log messages to journald. The journald daemon must be running on the host |
| art | Write log messages to a Graylog Extended Log Format (GELF) endpoint, such as Graylog or Logstash. |
| fluentd | Write log messages to fluentd(forward input). This fluentddaemon must be running on the host. |
| awslogs | Write log messages to Amazon CloudWatch Logs. |
| splunk | Log messages are written using the HTTP event collector splunk. |
| etulogs | Write log messages as Event Tracing for Windows (ETW) events. Applies to Windows platforms only. |
| gcplogs | Write log messages to Google Cloud Platform (GCP) Logging. |
| logentries | Write log messages to Rapid7 Logentries. |
2.5.1 Global modification log driver
Modify the configuration file /etc/docker/daemon.json
{
"log-driver": "none"
}
2.5.2 Runtime control, that is, the way to add parameters when the container starts
# max-size 最大数值,必须大于0
# max-file 最大日志数，必须大于0
$ docker run -it --log-opt max-size=10m --log-opt max-file=3 容器名称
2.5.3 Global configuration mode
Create or modify the file /etc/docker/daemon.json and add the following configuration:
{
"log-driver":"json-file",
"log-opts":{
"max-size" :"10m",
"max-file":"20",
"compress": "true"
}
}
Since this configuration does not take effect only for already generated containers, the original container will be destroyed and recreated
Restart the Docker service
sudo systemctl daemon-reload
sudo systemctl restart docker