Prometheus monitoring K8S Node Monitoring
NodeExporter Project Prometheus community can be a host of key metrics to monitor, can be deployed by DeamonSet Kubernetes of nodes on each host one and only one NodeExporter example, to achieve control of the host performance data, but since the container isolate the cause, use containers NodeExporter not get the right to host disk information, therefore this course will NodeExporter deployed to the host.
node_exporter: * NIX systems for monitoring, use the Go language Collector
- Use the document: https: //prometheus.io/docs/guides/node-exporter/
- GitHub：https://github.com/prometheus/node_exporter
- exporter list: https: //prometheus.io/docs/instrumenting/exporters/
The official document: https: //github.com/kubernetes/kube-state-metrics
Index node-exporter collected are:
* Node_cpu_ Node_disk_ * Node_entropy_ * Node_filefd_ * Node_filesystem_ * Node_forks_ * Node_intr_total_ * Node_ipvs_ * Node_load_ * Node_memory_ * Node_netstat_ * Node_network_ * Node_nf_conntrack_ * Node_scrape_ * Node_sockstat_ * Node_time_seconds_ * Node_timex _ * Node_xfs_ *
Profiles
Modify a nice profile
- # Prometheus profile
- prometheus-configmap.yaml
-
Profiles
# Prometheus the format https://prometheus.io/docs/prometheus/latest/configuration/configuration/ Configuration apiVersion: V1 kind: The ConfigMap Metadata: name: Prometheus - config namespace: Kube - System Labels: kubernetes.io / Cluster-Service-: " to true " addonmanager.kubernetes.io / the MODE: EnsureExists the Data: # store prometheus profile prometheus.yml: | # configure acquisition target scrape_configs: - job_name: prometheus static_configs: - targets: # Collection itself - localhost: 9090 prometheus.yml: | # Configure acquisition target scrape_configs: - job_name: kubernetes- Nodes static_configs: - Targets: # collection itself - 192.168.1.110:9100 - 192.168.1.111:9100 # Acquisition: Apiserver survival index # the Job name name created for apiservers-Kubernetes - job_name: kubernetes- apiservers # discovery based k8s service kubernetes_sd_configs: - Role: Endpoints # using communication markup tags relabel_configs: # Reserved regular matching tag - Action: the Keep # already contains regex: default; Kubernetes; HTTPS source_labels: - __meta_kubernetes_namespace - __meta_kubernetes_service_name - __meta_kubernetes_endpoint_port_name # use as https, the default HTTP scheme: HTTPS tls_config: # promethus access Apiserver use authentication ca_file: / var / RUN / Secrets / kubernetes.io / serviceaccount / ca.crt # skip https certification insecure_skip_verify: to true # promethus access Apiserver use certification bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/ token # collection: Kubelet survival index - job_name: Kubernetes-nodes- kubelet kubernetes_sd_configs: # found a cluster of all the Node - Role: the Node relabel_configs: # access to key by regex information - Action: labelmap : regex __meta_kubernetes_node_label_ (. + ) scheme: HTTPS tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt insecure_skip_verify: to true bearer_token_file:/var/run/secrets/kubernetes.io/serviceaccount/token # 采集：nodes-cadvisor 信息 - job_name: kubernetes-nodes-cadvisor kubernetes_sd_configs: - role: node relabel_configs: - action: labelmap regex: __meta_kubernetes_node_label_(.+) # 重命名标签 - target_label: __metrics_path__ replacement: /metrics/cadvisor scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/ token # collection: service-endpoints Information - job_name: Kubernetes-Service- Endpoints # selected indicators kubernetes_sd_configs: - Role: Endpoints relabel_configs: - Action: the Keep regex: to true # specify the source label source_labels: - __meta_kubernetes_service_annotation_prometheus_io_scrape - Action: the replace regex: (HTTPS?) source_labels: - __meta_kubernetes_service_annotation_prometheus_io_scheme # rename tag acquisition target_label: __scheme__ - action: replace regex: (.+) source_labels: - __meta_kubernetes_service_annotation_prometheus_io_path target_label: __metrics_path__ - action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 source_labels: - __address__ - __meta_kubernetes_service_annotation_prometheus_io_port target_label: __address__ - action: labelmap regex: __meta_kubernetes_service_label_ (+. ) - Action: the replace source_labels: - __meta_kubernetes_namespace target_label: kubernetes_namespace - Action: the replace source_labels: - __meta_kubernetes_service_name target_label: kubernetes_name # collection: kubernetes-services service indicators - job_name: kubernetes- Services kubernetes_sd_configs: - Role: Service # Black Box probing, probing IP and port is available metrics_path: / the probe params: Module: - http_2xx relabel_configs: - action: keep regex: true source_labels: - __meta_kubernetes_service_annotation_prometheus_io_probe - source_labels: - __address__ target_label: __param_target # 使用 blackbox进行黑盒探测 - replacement: blackbox target_label: __address__ - source_labels: - __param_target target_label: instance - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: - __meta_kubernetes_namespace target_label: kubernetes_namespace - source_labels: - __meta_kubernetes_service_name target_label: kubernetes_name # 采集： kubernetes-pods 信息 - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod relabel_configs: - action: keep regex: true source_labels: # 只保留采集的信息 - __meta_kubernetes_pod_annotation_prometheus_io_scrape - action: replace regex: (.+) source_labels: - __meta_kubernetes_pod_annotation_prometheus_io_path target_label: __metrics_path__ - action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 source_labels: # 采集地址 - __address__ # 采集端口 - __meta_kubernetes_pod_annotation_prometheus_io_port target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - token action: replace source_labels: - __meta_kubernetes_namespace target_label: kubernetes_namespace - Action: Replace source_labels: - __meta_kubernetes_pod_name target_label: kubernetes_pod_name Alerting: # alert profile alertmanagers: - kubernetes_sd_configs: # dynamic acquisition - Role: POD tls_config: ca_file: / var / RUN / Secrets / Kubernetes. IO / serviceaccount / ca.crt bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/ relabel_configs: - source_labels: [__meta_kubernetes_namespace] regex: kube-system action: keep - source_labels: [__meta_kubernetes_pod_label_k8s_app] regex: alertmanager action: keep - source_labels: [__meta_kubernetes_pod_container_port_number] regex: action: drop
Node department: node_exporter
1, the configuration file to take effect
kubectl apply -f prometheus-configmap.yaml
2, to see if the entry into force
3, using Grafana visualization templates: 9276
4, select the group
5, a node information (trimmable according to their display)
