1. The development history of K8S monitoring system
First edition: Cadvisor+InfluxDB+Grafana
It can only be collected from the host dimension, without the aggregation function of Namespace, Pod and other dimensions
Second edition: Heapster+InfluxDB+Grafana
Heapster is responsible for calling the cadvisor interface in each node, summarizing the data, and then importing it to InfluxDB, which can provide detailed resource usage from all levels of cluster, node, and pod.
The third edition: Metrics-Server + Prometheus
Metrics Server and Cadvsior and Cgroup relationship
Metrics Server is a service that exposes monitoring data as a standard Kubernetes API, such as obtaining monitoring data of a certain Pod; whether it is a heapster or a metric-server, it is only the transfer and aggregation of data, and both are the api interfaces of the kubelet called The data obtained in the kubelet code is the cadvisor module that actually collects indicators; when cadvisor obtains the indicators, it actually calls the runc/libcontainer library, and libcontainer encapsulates the cgroup file, that is, cadvsior is only a forwarder, and its data comes from In the cgroup file.
The value in the cgroup file is the ultimate source of monitoring data.
Such as: the value of mem usage,
For docker containers, it comes from /sys/fs/cgroup/memory/docker/[containerId]/memory.usage_in_bytes.
For pods, /sys/fs/cgroup/memory/kubepods/besteffort/pod[podId]/memory .usage_in_bytes
K8S advanced practice: dynamic expansion and contraction of business applications through HPA