问题概况
过了个周末，早上登上服务器，执行
kubectl get svc
等段时间抛出异常
查看docker运行情况
docker ps -a
发现 k8s 相关的容器都消失了
查询service状态
systemctl list-units --type=service |grep kube*
也是正常的，执行命令比较慢，有可能是磁盘比较满
查询磁盘使用情况
df -h
100% 问题
清空message
> message
重启k8s服务
systemctl restart kube-apiserver
systemctl restart kube-controller-manager
systemctl restart kube-scheduler
systemctl restart kubelet
systemctl restart kube-proxy
解决方法
we have a kubernetes cluster running on Centos 7. However all logging is going to /var/log/messages which is making centos system logs hard to read. Is there a way I can tell kubeadm/kubernetes to log to /var/log/kubernetes rather?
We are already sending our application (pod) logs to a mountpoint. We need to move the stderr logs of kubernetes.
No, not exactly, but you can reconfigure Docker to log in a different way.
This might depend on the Docker version you’re running but in my CentOS 7 VM (a couple of weeks old) i’m running Docker version 1.13.1, installed via yum.
When looking through the docs for version 1.13 and the latest stable version of Docker they say more or less the same thing:
If you do not specify a logging driver, the default is json-file.
The version of Docker i installed via yum had the following line in an environment file (/etc/sysconfig/docker) that is loaded when starting Docker:
OPTIONS=’–selinux-enabled --log-driver=journald --signature-verification=false’
As you can see the logging driver are configured as journald, that should be the reason you’re seeing logs from your containers in /var/log/messages. You can check which logging drive are configured with:
docker info | grep ‘Logging Driver’
The logging driver decides where all of the logs, in Docker meaning stderr and stdout from containers are sent. Docker supports a couple of different logging drivers, if you choose to configure e.g. json-file which might be the best choice if you want to relocate the logging from an OS perspective (“changing” the log path). Every Docker container will have it’s own log written to /var/log/pods///, actually the log files are symlinks back to /var/lib/docker/containers//-json.log.
If you do configure json-file then remove the –log-driver=journald flag and instead configure this in the /etc/docker/daemon.json file, mentioned in the docs. With json-file you can configure things like log rotation and log file sizes, please consult the docs for more options.
When configuring via the daemon.json file this becomes a global setting, you can always override the logging driver used for a specific container with docker run … --log-driver.
These logging changes applies for everything running within Docker, to move logging for e.g. kubelet which runs alongside Docker on your host you can look at the configurable options. Default the kubelet stderr logs are logged via journald and ends up in /var/log/messages, to change this behavior can add the –log-dir options and point to another location.
In the end of the day i think it’s good to give log shipping a thought and investigate the other logging drivers if they might fit into your environment.
Docker 日志清理
Docker容器在启动/重启的时候会往 /var/lib/docker 中写东西，如果你在启动docker容器遇到 No space left on device 的问题，可以按照下面的步骤进行清理相关的日志操作。
1、 对 /var/lib/docker/containers 下的文件夹进行排序，看看哪个容器占用了太多的磁盘空间
$ du -d1 -h /var/lib/docker/containers | sort -h
上面的命令会按照升序的方式对于容器文件夹进行排序，并列出容器文件夹的大小:
2、选择你要清理的容器进行清理
$ cat /dev/null > /var/lib/docker/containers/container_id/container_log_name
上述命令会清空对应的日志,如:
cat /dev/null > /var/lib/docker/containers/374aa0ba92b37d829012282ff15c1bb838d95dedb54589874c4285991be2d4aa/374aa0ba92b37d829012282ff15c1bb838d95dedb54589874c4285991be2d4aa-json.log
详解Docker守护进程的配置及日志
docker 日志路径
Configure and troubleshoot the Docker daemon
Configure logging drivers
Docker日志太多导致磁盘占满的处理方法