Prometheus+Grafana monitoring system cooperates with Cadvisor to monitor Docker containers

Prometheus uses cadvisor to monitor docker containers

1. Cadvisor overview

Generally, companies have a lot of docker hosts, so you need to monitor docker. docker monitoring can use docker stats with shell commands to get the value for monitoring, but it cannot be passed to prometheus for collection. Zabbix monitoring docker is more troublesome, so there is Google's cadvisor

Cadvisor can not only collect all the running container information on a machine, but also provides the http interface of the basic query interface, which is convenient for prometheus to capture data.

Cadvisor can perform real-time monitoring and performance data collection on the resources and containers on node machines, including CPU usage, memory usage, network throughput and file system usage. However, cadvisor has a flaw and can only display current monitoring data in real time. , The chronological data is not retained, so cadvisor needs to be connected to prometheus, and the chronological data is stored by the built-in tsdb database of prometheus

Cadvisor uses Linux cgroups to obtain the resource usage of the container, integrated in k8s, and the official standard configuration

Cadvisor needs to be installed on every docker host

Docker mainly monitors container memory, CPU, disk, and network card

Commands to monitor container performance using docker stats

[root@192_168_81_220 docker]# docker stats --no-stream nervous_moser
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
e265e173ac98 nervous_moser 0.00% 5.605MiB / 972.6MiB 0.58% 2.42kB / 0B 27.3MB / 22.5kB 2

Finally, cooperate with grafana to show the effect of monitoring docker containers.
You can clearly see the CPU, memory usage of each container and the number of containers running on each host in the figure.

2. Environmental preparation

IP	CPU name	service
192.168.81.210	prometheus-server	prometheus 、 grafana
192.168.81.220	192_168_81_220	docker、cadvisor
192.168.81.230	192_168_81_230	docker、cadvisor

3. Deploy docker on the docker server

192.168.81.220/92.168.81.230 servers need to be operated

1.获取镜像源
wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
 
sed -i 's#download.docker.com#mirrors.tuna.tsinghua.edu.cn/docker-ce#' /etc/yum.repos.d/docker-ce.repo 

yum makecache fast

2.安装docker
yum -y install docker-ce
systemctl start docker
systemctl enable docker

3.配置镜像加速器
 tee /etc/docker/daemon.json <<-'EOF'
{
   "registry-mirrors": ["https://zggyaen3.mirror.aliyuncs.com"]
}
EOF


4.启动docker
systemctl restart docker

5.简单运行一个容器
docker run -d nginx:latest

4.Docker install cadvisor monitoring

4.1. Install cadvisor

All operations on the docker host, 192.168.81.220/92.168.81.230 servers all need to be operated

docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest
  
注释：   --volume=/:/rootfs:ro 		#将/目录挂载到容器的/rootfs，权限为读
	    --volume=/var/run:/var/run:rw 	#将/var/run挂载到容器的/var/run目录，权限为读写
	    其他的volume都是这个意思，ro为读，rw为读写
	    --volume=/var/lib/docker/:/var/lib/docker:ro			#这个是将宿主机的docker目录挂载到容器，因为cadvisor需要监控所有容器，因此需要挂载
	    --publish=8080:8080 		#开放公共端口

The final effect is that both machines have started cadvisor and nginx mirroring

4.2. Visit cadvisor to view monitoring data

Visit http://host ip:8080j

4.3. View the monitoring information of a certain container

Click on the homepage/docker, and after jumping in, you can see the container list under subcontainers. The container IDs displayed below are all container IDs. You can use docker ps to see the specific container

After opening, you can see a specific container monitoring indicator

You can also see the image

4.4.cadvisor monitoring index page

Just visit http://host ip:8080/metrics

5.prometheus add cadvisor monitoring

5.1.prometheus installation

二进制安装详细看上一篇文章
docker安装
[root@prometheus-server ~]# docker run -d -p9090:9090  -v /data/prom/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus:latest 

5.1. Modify the prometheus configuration file

1.修改配置文件增加docker监控cadvisor
[root@prometheus-server ~]# vim /etc/prometheus/prometheus.yml 
  - job_name: "docker"
    static_configs:
    - targets: ['192.168.81.220:8080','192.168.81.230:8080']
2.刷新配置
[root@prometheus-server ~]# curl -X POST http://127.0.0.1:9090/-/reload

The docker monitoring host is up

Check the monitoring items, the beginning of the container is monitoring

6.prometheus+grafana realizes the monitoring and graphical display of docker containers

6.1. A dressed grafana

[root@prometheus-server ~]# rpm -ivh grafana-6.3.2-1.x86_64.rpm 
[root@prometheus-server ~]# systemctl start grafana-server
[root@prometheus-server ~]# systemctl enable grafana-server

Visit http://server address: 3000

6.1. Add prometheus data source

Click Add data source

Choose prometheus

Fill in the prometheus address

6.2. Import docker monitoring graphic template

6.2.1. Click the + sign—import

6.2.2. Fill in the monitoring template id

Fill in the id number of the monitoring template here, the docker monitoring template No. 193 is better

6.2.3. Select prometheus source

After filling in the id number, the corresponding template will be found automatically

Select the corresponding prometheus source and click import

6.3. View graphics

Choose Docker monitoring

The fly in the ointment of this template is that you can’t see the containers on a specific host, but instead list all the containers of the docker host here.

7. You can choose the template import of the docker host

Finally found a docker monitoring template that can select the host in grafana

Download link https://grafana.com/api/dashboards/8321/revisions/3/download

After importing this face value

8. The final effect