[Cloud native • Monitoring] Prometheus-based cloud native cluster monitoring (theory + practice)-02

[Cloud native • Monitoring] Prometheus-based cloud native cluster monitoring (theory + practice)-02

k8s resource object index

kube-state-metrics

cAdvisorIt is mainly the performance indicator of the underlying container runtime, and there is no kubernetesstatus indicator of the cluster resource object. For example, we want to know the running status of the service, whether the Pod has been restarted, whether the scaling has been successful, and what the status of the Pod is. A component is kubernetesprovided kube-state-metricscan be used to expose these indicators.

kube-state-metricsIt is just a simple service that API Servergenerates the latest status indicators of resource objects by listening to and subscribing to the changes of various resource objects, such as Deployment, Daemonset, StatefulSet, Node, Pod, Container, Serviceetc. It does not focus on Kubernetesthe running status of node components, but on the running status of various resource objects within the cluster. It should be noted that it kube-state-metricssimply provides a metricsdata and does not store these indicator data, so we can use it prometheusto grab these data Then store.

cAdvisorIt has been Kubernetesintegrated by default, but kube-state-metricsnot integrated by default, so if we want to monitor the complete data of the cluster, we need to kubernetesdeploy kube-state-metricscomponents separately in , so that the service resource indicator data in the cluster can be exposed, so as to monitor different resources .

kube-state-metricsComponent deployment is relatively simple, but you need to pay attention to version compatibility issues:

My kubernetesversion is v1.21.0, so select kube-state-metricsthe version here v2.3.0.

deploy

1. masterCreate a directory on the node kube-state-metrics, and copy the files under the directory kube-state-metrics-2.3.0.tar.gzin the decompressed package to the directory:examples/standardkube-state-metrics

[root@k8s-01 kube-state-metrics]# ls -alh
总用量 20K
drwxr-xr-x 2 root root  135 6月  19 16:02 .
drwxr-xr-x 4 root root   54 6月  19 15:57 ..
-rw-r--r-- 1 root root  418 6月  19 16:02 cluster-role-binding.yaml
-rw-r--r-- 1 root root 1.7K 6月  19 16:02 cluster-role.yaml
-rw-r--r-- 1 root root 1.2K 6月  19 16:02 deployment.yaml
-rw-r--r-- 1 root root  234 6月  19 16:02 service-account.yaml
-rw-r--r-- 1 root root  447 6月  19 16:02 service.yaml

❝

Since kube-state-metricsthe component needs to connect with kube-apiserverand call the corresponding interface to obtain kubernetesthe cluster data, this process requires kube-state-metricsthe component to have certain permissions to successfully perform these operations. kubernetesBy default, the method is used to manage permissions in RBAC, so we need to create corresponding RBAC resources to provide this component.

❞

2. Modify the image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  labels:
    k8s-app: kube-state-metrics
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: kube-state-metrics
  template:
    metadata:
      labels:
        k8s-app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: bitnami/kube-state-metrics:2.0.0
        securityContext:
          runAsUser: 65534
        ports:
        - name: http-metrics    ##用于公开kubernetes的指标数据的端口
          containerPort: 8080
        - name: telemetry       ##用于公开自身kube-state-metrics的指标数据的端口
          containerPort: 8081

❝

k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0This image is special attention, because the prefix is k8s.gcr.io, so it is pullnot necessary, change it registry.cn-beijing.aliyuncs.com/zhaohongye/kube-state-metrics:v2.3.0.

❞

3. Create a deployment:

[root@master kube-state-metrics]# kubectl  apply -f  ./
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
serviceaccount/kube-state-metrics created
service/kube-state-metrics created

4. Check whether it is allowed to succeed:

[root@k8s-01 ~]# kubectl get pod -n kube-system -l app.kubernetes.io/name=kube-state-metrics -owide
NAME                                 READY   STATUS    RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
kube-state-metrics-5dc49b696-rssx8   1/1     Running   0          11m   192.165.179.4   k8s-02   <none>           <none>
[root@master kube-state-metrics]# kubectl get svc -n kube-system |grep kube-state-metrics
kube-state-metrics   ClusterIP   None           <none>        8080/TCP,8081/TCP        50m

5. Verify /metricsthat the endpoint can collect indicators:

[root@master kube-state-metrics]# curl  192.165.179.4:8080/metrics

❝

Port to access kube-state-metricsthe component pod ip+8080.

❞

prometheus access

/metricsThe endpoint can collect indicator data, indicating that kube-state-metricsthe component deployment is normal, and you can use prometheus to access it.

1. The component creates svca name kube-state-metrics, so the service discovery protocol that can be used is configured as follows service:endpoints

- job_name: 'kube-state-metrics'
   metrics_path: metrics
   kubernetes_sd_configs:
   - role: endpoints
   relabel_configs:
   - source_labels: [__meta_kubernetes_service_name]
     regex: kube-state-metrics
     action: keep
   - source_labels: [__meta_kubernetes_endpoint_port_name]
     regex: http-metrics
     action: keep  
   - action: labelmap
     regex: __meta_kubernetes_pod_label_(.+)

❝

See prometheusthe deployment section, prometheusand put the configuration in prometheus-configthis ConfigMap. And deploy configmap-reloadthe container to the container in sidecarthe mode , we only need to modify this , the container will automatically monitor the change, and then pull the latest configuration and hot update to it .prometheusprometheus-configConfigMapconfigmap-reloadprometheus

❞

2. Check the information prometheuson the web page target, and the access is normal:

3. grafanaImport in 14518 dashboard, and kube-state-metricsthe performance monitoring indicators will be displayed on the template.

a. Such as cluster Node node performance indicators, such as CPU, memory, network IO, disk IO, disk and other monitoring information:

b. Such as the statistics of various resource objects in the cluster, such as the statistics of the number of Pods in various states, the number of namespaces, the number of PVCs, the statistics of pods running on node nodes, the statistics of namespace resource applications, and so on.

c. More resource monitoring is as follows:

metrics-server

There kube-state-metricsis another component that is often confused with components metrics-server. Let's take a look at the role of this component.

From kubernetes 1.8the beginning, resource usage metrics, such as container's CPUand memory, are obtained through Metrics APIin kubernetes, and these data can be directly accessed by users, for example, by using kubectl topcommands to view node, Podor container's CPUand memory usage information, or by the controller in the cluster, For example VPA, HPAetc. are used to make decisions, metrics-serverand components replace the heapsterimplementation of this set of interfaces, which heapsterhave been gradually abandoned since 1.11.

❝

VPA, HPAReading is used for capacity expansion and contraction technology, and whether there is a performance problem in the standard for capacity expansion and contraction judgment metrics-serveris to provide data basis for performance problems.

❞

metrics-serverIt is an aggregator of cluster core monitoring data. In layman's terms, it stores the monitoring data of each node in the cluster and provides APIs for analysis and use. kubeletThe core principle is to collect indicator information from the components on each node Summary API. The following command can obtain the data of the components k8s-02on the node :kubeletsummart

# 获取节点 kubelet summary 数据
[root@k8s-01 metrics-server]# kubectl get --raw=/api/v1/nodes/k8s-02/proxy/stats/summary

The data here is the CPU, memory, network, and file system usage of the node, as well as the more detailed CPU, memory, network, and volume usage of each container on the node metrics-server. The information is collected and gathered together, so that kubectl top podsthe information can be viewed by typing commands in the console.

For example, to view the CPU and memory usage of each node:

[root@k8s-01 metrics-server]# kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-01   164m         8%     2172Mi          59%       
k8s-02   93m          4%     1631Mi          44%       
k8s-03   101m         5%     1489Mi          40%

Or view the CPU and memory usage of a Pod:

[root@k8s-01 metrics-server]# kubectl top pod --all-namespaces
NAMESPACE              NAME                                        CPU(cores)   MEMORY(bytes)   
default                my-nginx-6b74b79f57-gxzx6                   0m           4Mi             
demo01                 nginx                                       0m           3Mi             
demo01                 nginx-demo1-7678bcdf48-cwtxk                0m           2Mi             
demo01                 nginx-demo1-7678bcdf48-f8bhh                0m           2Mi             
demo01                 nginx-deployment-746fbb99df-lxn7b           0m           1Mi             
demo01                 probe-deployment-59dd8bb78d-d5ljg           0m           0Mi             
demo01                 tomcat-deployment-7db86c59b7-bn8zl          2m           129Mi                     
monitoring             grafana-7cfd74ccf5-crcnz                    1m           34Mi            
monitoring             prometheus-7fd7fdb677-4q7d2                 4m           160Mi

You can also view the load of the container in the pod in more detail:

[root@k8s-01 metrics-server]# kubectl top pod prometheus-7fd7fdb677-4q7d2 --containers -n monitoring
POD                           NAME                CPU(cores)   MEMORY(bytes)   
prometheus-7fd7fdb677-4q7d2   prometheus          8m           156Mi           
prometheus-7fd7fdb677-4q7d2   prometheus-reload   0m           4Mi

Behind this function is metric-serverthe support provided by this component. If this component is not deployed, the execution kubectl topcommand will report an error.

" metrics-serverThe focus is on the underlying container CPU, memory and other performance data, which come from kubeletthe components. In fact, they come from kubeletthe container performance indicators provided by the internally integrated cAdvisor component. That is, the performance indicators provided by cAdvisor already include the metrics-serverperformance indicators of the components. prometheusThe collection only needs to be collected. cAdvisorComponents metrics-serverno longer need to be connected to prometheus monitoring.”

deploy

1. metrics-serverIt is an extended dependency on kube-aggregator, because we need to add startup parameters in APIServer --enable-aggregator-routing=true:

# vi /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.31.160
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --enable-aggregator-routing=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379

❝

If you're not masterrunning on node kube-proxy, you'll have to make sure kube-apiserverthat is included in the startup parameters --enable-aggregator-routing=true.

Then, restart kubeletthe component: systemctl restart kubelet.

❞

2. Download metrics-servermedium: https://github.com/kubernetes-sigs/metrics-server/releases

❝

I download v0.6.1the version here.

❞

3. Modify the configuration:

❝

Precautions:

1. If you need to ignore the Kubelet certificate, name only needs to add –kubelet-insecure-tls to the containers.args of the deployment. 2. If you are in China, you need to change the mirror warehouse to a domestic source, such as Alibaba Cloud, for example Change k8s.gcr.io/metrics-server/metrics-server to registry.aliyuncs.com/google_containers/metrics-server.

3. Add configurationhostNetwork: true

❞

4. Create a deployment:

[root@k8s-01 metrics-server]# kubectl apply  -f components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

5. Check whether the operation is successful:

[root@k8s-01 metrics-server]# kubectl get pod -n kube-system -l k8s-app=metrics-server 
NAME                              READY   STATUS    RESTARTS   AGE
metrics-server-5bb7df9c68-x4ttq   1/1     Running   0          3m44s

❝

After a minute or two, you can see that metrics-server-5bb7df9c68-x4ttq runs successfully.

❞

6. Execute kubectl topthe instruction to verify whether it is normal:

[root@k8s-01 metrics-server]# kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-01   164m         8%     2172Mi          59%       
k8s-02   93m          4%     1631Mi          44%       
k8s-03   101m         5%     1489Mi          40%

"So, to sum up: there is still a big difference between kube-state-metrics and kube-state-metrics, which mainly focuses on some business-related metadata, such as Deployment, Pod, replica status, etc.; metrics-service mainly focuses on resource kube-state-metricsmetrics metrics-serverThe implementation of the API, such as CPU, memory, network and other indicators, can be used to view performance information for the kubectl top command, or provide decision-making indicator support for components such as VPA and HPA.”

[For more cloud-native monitoring and operation and maintenance, please pay attention to the WeChat public account: cloud-native ecological laboratory]