183. docker容器监控

1. 安装cadvisor + prometheus

1. 环境：
docker01 docker02    启动容器cadvisor 采集数据
docker01 docker02    启动容器node-exporter 监控宿主机资源
docker03    安装prometheus

2. 增加node节点[docker01 && docker02]
上传docker_monitor_node.tar.gz
[root@docker01 ~]# docker load -i docker_monitor_node.tar.gz 

3. 启动node-exporter
docker run -d   -p 9100:9100   -v "/:/host:ro,rslave"   --name=node_exporter   quay.io/prometheus/node-exporter   --path.rootfs /host

4. 启动cadvisor
docker run --volume=/:/rootfs:ro  --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro  --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest

5.安装普罗米修斯并配置配置yml文件
[root@docker03 opt]# tar xf prometheus-2.12.0.linux-amd64.tar.gz 
[root@docker03 opt]# mv prometheus-2.12.0.linux-amd64 prometheus
[root@docker03 prometheus]# vim prometheus.yml
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'cadvisor'
    static_configs:
    - targets: ['10.0.0.11:8080','10.0.0.12:8080']
  - job_name: 'node-exporter'
    static_configs:
    - targets: ['10.0.0.11:9100','10.0.0.12:9100']

6.启动普罗米修斯
[root@docker03 prometheus]# ./prometheus --config.file="prometheus.yml"

7. 检测端口
[root@docker03 ~]# lsof -i:9090
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
prometheu 8144 root    7u  IPv6  59275      0t0  TCP *:websm (LISTEN)
prometheu 8144 root    9u  IPv4  59277      0t0  TCP localhost:38986->localhost:websm (ESTABLISHED)
prometheu 8144 root   10u  IPv6  59280      0t0  TCP localhost:websm->localhost:38986 (ESTABLISHED)

2. 安装grafana并启动

[root@docker03 ~]# yum localinstall grafana-6.3.3-1.x86_64.rpm -y

[root@docker03 ~]# systemctl start grafana-server.service 
[root@docker03 ~]# systemctl enable grafana-server.service

grafana:
插件：zabbix
数据源：zabbix
dashboard

3. 登录grafana配置数据源

登录grafana配置数据源--->导入监控docker插件--->查看监控

4. zabbix低级自动发现

docker zabbix监控 低级自动发现 自动创建监控项

1. 启动zabbix容器
[root@docker01 zabbix]# docker-compose up -d 
Creating zabbix_mysql-server_1        ... done
Creating zabbix_zabbix-java-gateway_1 ... done
Creating zabbix_zabbix-java-gateway_1 ... 
Creating zabbix_zabbix-server_1       ... done
Creating zabbix_zabbix-server_1       ... 
Creating zabbix_zabbix-web_1          ... done

2. 监控一台主机

4.1 监控mac地址

[root@docker02 ~]# cat /etc/zabbix/zabbix_agentd.d/mac_status.conf 
UserParameter=net.if.mac_status[*],ifconfig $1 |awk '/ether /{print $$2}'
UserParameter=docker.discovery[*],sh /scripts/docker_discovery.sh

docker01测试：
[root@docker01 ~]# wget https://mirrors.tuna.tsinghua.edu.cn/zabbix/zabbix/3.2/rhel/7/x86_64/zabbix-get-3.2.11-1.el7.x86_64.rpm
[root@docker01 ~]# rpm -ivh zabbix-get-3.2.11-1.el7.x86_64.rpm 
[root@docker01 ~]# zabbix_get -s 10.0.0.12 -k net.if.mac_status[eth0]
00:0c:29:c6:01:ef

4.2 创建自定义监控项



4.3 创建自动发现规则

1.编写脚本
[root@docker02 scripts]# cat docker_discovery.sh 
#!/bin/bash
port=($(/usr/bin/docker ps -a|grep -v "CONTAINER ID"|awk '{print $NF}'))
printf '{\n'
printf '\t"data":[\n'
   for key in ${!port[@]}
       do
           if [[ "${#port[@]}" -gt 1 && "${key}" -ne "$((${#port[@]}-1))" ]];then
              printf '\t {\n'
              printf "\t\t\t\"{#CONTAINERNAME}\":\"${port[${key}]}\"},\n"

         else [[ "${key}" -eq "((${#port[@]}-1))" ]]
              printf '\t {\n'
              printf "\t\t\t\"{#CONTAINERNAME}\":\"${port[${key}]}\"}\n"

           fi
   done
              printf '\t ]\n'
              printf '}\n'

2. 监控容器的存活状态：
[root@docker02 ~]# cat /etc/zabbix/zabbix_agentd.d/mac_status.conf 
UserParameter=net.if.mac_status[*],ifconfig $1 |awk '/ether /{print $$2}'
UserParameter=docker.discovery[*],sh /scripts/docker_discovery.sh
UserParameter=docker_alive[*],/usr/bin/docker ps -a |grep $1|grep -c Up

systemctl restart zabbix-agent

3. 测试取值
[root@docker01 zabbix]# zabbix_get -s 10.0.0.12 -k docker.discovery
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json?all=1: dial unix /var/run/docker.sock: connect: permission denied

解决问题：
[root@docker02 scripts]# chmod u+s /usr/bin/docker
或者：chmod 777 /var/run/docker.sock


4.登录zabbix-server 创建自动发现规则：docker.discovery

基于刚才创建的自动发现规则--->创建监控项原型
名称和键值需要用变量docker_alive[{#CONTAINERNAME}]

过滤器操作：
管理--->一般---》正则表达式---》添加正则--->配置过滤器