In the previous lesson, we learned how to set requests and limits of resources. In this lesson, we will learn how to monitor resources, and configure requests and limits reasonably according to the monitoring resource usage.
Collect and obtain actual resource usage
Kubelet contains an agent named cAdvisor, which collects the resource consumption of all individual containers running on the entire node. This information can be used to centralize the monitoring information of the entire cluster through an additional component Heapster
Heapster runs on a certain node as a pod, and it exposes the service through a common kubernetes Service, so that the outside world can access it through a stable ip address. It collects information from all cAdvisors in the cluster and exposes it through a single address.
Start Heapster
On Google Container Engine, Heapster is enabled by default
On Minikuber, it is used as an add-on and needs to be enabled with the command: $minikube addons enable heapster
Other types of kubernetes clusters run heapster, refer to: https://github.com/kubernetes/heapers
Note: It takes a few minutes to start the heapster
Display the cpu and memory usage of the cluster nodes
kubectl top node
Show cpu and memory usage of individual pods
kubectl top pod --all -namespaces
Save and analyze historical resource usage statistics
The top command only displays the current resource usage, it does not display the pod and cpu usage from one hour, one day, one week ago to the present. Both cAdvisor and Heapster only store resource usage data for a short time window. To analyze pod and resource usage over time, additional tools must be used. Google container engine can be monitored through google cloud monitoring, but for local kubernetes clusters, InfluxDB can be used to store statistics, and then Grafana can be used to visualize and analyze the data.
InfluxDB and Grafana
InfluxDB is an open-source time-series database for storing application metrics and other monitoring data.
Grafana is a data analysis and visualization suite with a gorgeous web console, also open source
It allows users to visualize data stored in InfluxDB while discovering how an application's resource usage changes over time.
Computing resources chapter summary:
This chapter talks about the need to consider the resource usage of the pod in order to ensure that everything runs smoothly, and configure resource requests and limits for the pod at the same time
Specify resource requests to help kubernetes schedule pods in the cluster
Specify resource limits to prevent a pod from preempting resources from other pods
Idle cpu time is allocated according to the cpu requests of the container
If the container uses too much cpu, the system will not kill the container, but if it uses too much memory, it will be killed
In an overcommited system, containers can also be killed to free up memory for more important pods, based on the pod's QoS level and actual memory usage
You can define minimum, maximum, and default values for resource requests and limits of a single pod through the LimitRange object
You can limit the number of resources available to all pods in a namespace through the ResourcesQuota object
To know how to set appropriate requests and limits for pods, it is necessary to monitor the usage of pod resources for a long enough period of time