I recently encountered a problem. When I redeployed a service in docker, after executing the sh start.sh command or docker-compose up -d command, the service failed to start, and a device or resource busy error was reported. .
Then I took a look at docker ps, good guy, not only failed to start, but also the service is gone.
This is because when docker is deployed, it will rm all the original things and then rebuild them again.
Look at the content of the error at that time:
failed to remove root filesystem for 9b13c255d03757c456651506ada5566a5b5e5a87b851e3621dcejf823: remove /var/lib/docker/overlay/9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d/merged: device or resource busy
Mainly, it is reported: device or resource busy, which means that the device or resource is busy, and you cannot delete this resource.
Then I went to /var/lib/docker/overlay/9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d/merged This directory is empty, and manual deletion does not work.
Solution
1. Find the process occupying this directory
Command: grep id /proc/*/mounts
The example is as follows:
grep 9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d /proc/*/mounts
After executing the command, it is obvious that two processes are using it:
/proc/25086/mounts:overlay /data0/docker/overlay2/9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d /merged overlayrw,relatime,lowerdir=/data0/docker/overlay2/l/F4BQHNVPHGRKWXFRUIHXUMAHZ2:/data0/docker/overlay2/l/.....
/proc/25065/mounts:overlay /data0/docker/overlay2/9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d /merged overlayrw,relatime,lowerdir=/data0/docker/overlay2/l/......
2. Kill the process
Command: kill -9 The process id
example is as follows:
kill -9 25086
kill -9 25065
Note: You need to see clearly whether some processes can be deleted directly. If you delete them all at once, it will not be good if it affects other services.
3. Continue to delete the directory, successfully
Command: rm -rf The file path of the error report
Example:
rm -rf /var/lib/docker/overlay/9b13c255d03757c456651506adaa2fb660f75a5b5e5a87b851e3621dce0a368d/merged
After deleting this directory file, re-execute sh start.sh or docker-compose up -d command to start the service, then the startup is successful!