There is a machine that does not get up as usual. The server is a container on docker . Then log in to the docker host and start problem analysis and resolution:
After the intranet was powered off today, there was a machine that did not get up as usual. The server was a container on docker , and then logged in to the docker host to start problem analysis and resolution:
1. Find the problem
1. Start the iframe-test machine
root@ubuntu:~#docker start iframe-test
iframe-test
2. Found that there is no container process
root@ubuntu:~#docker ps |grep iframe-test
3. Check the log and find that there is a problem with the nginx configuration, causing the interruption.
root@ubuntu:~# docker logs iframe-test
Startingnginx: Starting periodic command scheduler: cron.
nginx:[emerg] unexpected end of file, expecting ";" or "}" in/etc/nginx/nginx.conf:21
nginx:configuration file /etc/nginx/nginx.conf test failed
2. Thinking about solutions
The cause of the problem is found, that is, the nginx file detection fails, resulting in interruption.
There are currently two solutions:
Method 1: Submit the problem container to a new image with docker commit, then use docker run -i -d to run a temporary terminal based on the new image to change the configuration file, and then submit the id of the temporary terminal to a new image, Then restart the container based on the new image. (This method has many steps, and a new image is submitted, which adds complexity to subsequent maintenance)
Method 2: Change the configuration file in the container directly without submitting a new image. But this container is down, how to change it? A detailed description will be given below. I would like to thank the netizen Lingx for providing solutions.
3. Modify the configuration of the downtime container
All container data exists in /var/lib/docker/aufs/diff/ . The following container ID directory, ending with init is for configuration files, such as / etc/host, reselv.conf, /dev, etc. The other is the file directory, such as /home , /var/ and services installed by yourself, etc. aufs needs the support of kernel 3.10 or above .
1. View the container id
root@ubuntu:~#docker ps -a|grep iframe-test
fa02f8084b63 debian06-base:latest
2. Find the path of the nginx.conf configuration file
root@ubuntu:~#find / -name 'nginx.conf'
/root/nginx.conf
/var/lib/docker/aufs/diff/7c7b3438586e0653cdca7977a4f889cfdca300f008771462f8a2e6e9d3bc5b84/etc/nginx/nginx.conf
/var/lib/docker/aufs/diff/6bc6a9a5aeb59e19cae8bb78daa481cc465051069c7854528cbfdb3c9c1f2bfb/etc/nginx/nginx.conf
/var/lib/docker/aufs/diff/c7b6b87cfda72701229eebca868eb047aa01c255b62e56ad223dc75396c584e4/etc/nginx/nginx.conf
/var/lib/docker/aufs/diff/fa02f8084b631c371c6c050e5f0315017d327f84746b064246803a6a90a39456/etc/nginx/nginx.conf
3. Enter the directory corresponding to the container id and modify the problem file
root@ubuntu:cd /var/lib/docker/aufs/diff/fa02f8084b631c371c6c050e5f0315017d327f84746b064246803a6a90a39456
Execute the ls command, and the root directory of the container is displayed in front of you. Are you familiar with it?
root@ubuntu:/var/lib/docker/aufs/diff/fa02f8084b631c371c6c050e5f0315017d327f84746b064246803a6a90a39456#ls
etc root run srv tmp usr var
Next, find the syntax error of nginx.conf in this container and modify it.
4. Start the container after modification
root@ubuntu:~# docker start iframe-test
root@ubuntu:~# docker ps |grep iframe-test
fa02f8084b63 debian06-base:latest "/etc/rc.local" 6 weeks ago Up 13 minutes 10.18.103.2:22->22/tcp,10.18.103.2:80->80/tcp, 10.18.103.2:443->443/tcp,10.18.103.2:3306->3306/tcp, 10.18.103.2:6379->6379/tcp,10.18.103.2:6381->6381/tcp, 10.18.103.2:8000->8000/tcp,10.18.103.2:8888->8888/tcp iframe-test
The container starts successfully and the problem is solved. In the future, you don't have to worry about the docker container being broken, resulting in data loss.
My other docker related articles:
The liberation of operation and maintenance personnel - Docker rapid deployment
http://yangrong.blog.51cto.com/6945369/1551327
Docker container port IP planning and port dynamic expansion