Prometheus 之 业务容器指标的监控（即cadvisor）

一、cadvisor的理解使用

cAdvisor 的指标访问路径为: https://10.96.0.1/api/v1/nodes/<node_name>/proxy/metrics ，如：

https://10.96.0.1/api/v1/nodes/k8s-master/proxy/metrics
https://10.96.0.1/api/v1/nodes/k8s-slave1/proxy/metrics
https://10.96.0.1/api/v1/nodes/k8s-slave2/proxy/metrics

curl -k  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Inp6MklMeE1OajFfWlVuVGZvTXFsalh5Y2hsak9kVHpjdGpNaXpjZENSeDAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi10b2tlbi02bHB2cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjYwYTM1NTg2LTI4MTYtNDJkOS1iZjEzLWM2ODZlM2JiMTc3NSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbiJ9.TBwd0LTb1GKfXEKMJJt2IeilcpcYj2fL45M9iGu0TUni1qzyJ-blTzoU4d2qhh2lrSnVloPcfSjH8KfBq1qsvVn-2iUdQNRbntnFHh3NWPBKBykGNzVOwZazGEykKoekk_opc68Jm1TFzvtzZ_LhgsVF9wJZGLBpIreKZ1D4MJT-GRGF_CVDJ0Vwc2nuEP1w5R_ek8MbPDWLH_D-UZTaYsV1o5qcMoypBPORnMecV_QL32eOQ470eFQZfQgie8YfwRl8IdSoE5iVho4pExytV1w6UpXP_42QcFbx9GXz2NCu92kCylmGv62gR8Bcrn44spZd_z5udZI2EVpA8hsJNw" https://10.96.0.1:443/api/v1/nodes/k8s-master/proxy/metrics | more

二、以标准方式添加job

prometuese配置文件添加如下内容：

    - job_name: 'kubernetes-sd-cadvisor'
      kubernetes_sd_configs:
        - role: node

三、使用relabeling的能力，修改请求地址

1、需修改成如下对应的地址：
https://10.96.0.1/api/v1/nodes/k8s-master/proxy/metrics
https://10.96.0.1/api/v1/nodes/k8s-slave1/proxy/metrics
https://10.96.0.1/api/v1/nodes/k8s-slave2/proxy/metrics

2、正则匹配过程

3、修改job

    - job_name: 'kubernetes-sd-cadvisor'
      kubernetes_sd_configs:
        - role: node
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
        - target_label: __address__
          replacement: 10.96.0.1
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

4、运行效果