With prometheus monitoring vessel, apiserver and automatic discovery and monitoring services

Use the built-in monitor cAdvisor container
cAdvisor already built into the assembly kubelet, so we do not need to install a separate, cAdvisor data path / API / V1 / Nodes / <Node> / Proxy / metrics
. 1, Job increases, updates prometheus configuration

- job_name: 'kubernetes-cadvisor'
  kubernetes_sd_configs:
  - role: node
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: kubernetes.default.svc:443
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

Prometheus-kubectl the Apply -f $ cm.yaml
$ kubectl GET svc -n Kube-OPS | grep Prometheus
Prometheus NodePort 10.102.197.83 <none> 9090: 32619 / TCP
$ curl -X POST " http://10.102.197.83:9090 / - / reload "# validate the configuration

Monitoring apiserver
1, increase job, configuration update prometheus

- job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  - role: endpoints
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
    action: keep
    regex: default;kubernetes;https

Prometheus-kubectl the Apply -f $ cm.yaml
$ kubectl GET svc -n Kube-OPS | grep Prometheus
Prometheus NodePort 10.102.197.83 <none> 9090: 32619 / TCP
$ curl -X POST " http://10.102.197.83:9090 / - / reload "# validate the configuration

Configuring general svc automatic discovery and monitoring
1, increase job, configuration update prometheus

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

To automatically discover the cluster Service, we need to add annotation area of Service: prometheus.io/scrape=true statement
$ kubectl the Apply -f Prometheus-cm.yaml
$ kubectl GET svc -n Kube-OPS | grep Prometheus
NodePort 10.102.197.83 Prometheus <none> 9090: 32619 / TCP
$ curl -X POST " http://10.102.197.83:9090/-/reload " to validate the configuration #
2, modify svc redis, and to dynamically discover and monitor ( on the basis of the static discovery)
add annotations

kind: Service
apiVersion: v1
metadata:
  name: redis
  namespace: kube-ops
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9121"
spec:
  selector:
    app: redis
  ports:
  - name: redis
    port: 6379
    targetPort: 6379
  - name: prom
    port: 9121
    targetPort: 9121

Add the prometheus.io/scrape=true in redis this Service created earlier this annotation
because the metrics service interfaces redis redis-exporter in 9121 the services above, we also need to add annotations such a prometheus.io/port=9121
Prome-kubectl the Apply -f $ redis.yaml
3, modify svc trafik, and to dynamically discover and monitor (found on the basis of static)

apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
  annotations:
    prometheus.io/scrape: "true"        #新增
    prometheus.io/port: "8080"        #新增
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

After we have a new service, if the service itself provides / metrics interfaces, we absolutely do not need to configure a static way
after turning services dynamic discovery, the default automatic dynamic discovery and monitoring services as follows: prometheus itself, kube- DNS
. 4, automatic discovery kube-state-metrics
implemented stateful Pod, DaemonSet, Deployment, Job, CronJob other object resources in the cluster Kubernetes
$ Git clone https://github.com/kubernetes/kube-state- metrics.git
$ cd kube-state-metrics / Kubernetes
$ kubectl the Apply -f.
after kube-state-metrics deployed to Kubernetes, you will find Kubernetes cluster service will automatically Prometheus in kubernetes-service-endpoints this job found kube-state-metrics, and began to pull metrics, this is because the deployment manifest definition file kube-state-metrics of kube-state-metrics-service.yaml Service definition includes prometheus.io/scrape: 'true' so one annotation