Prometheus+Grafana (external) monitors Kubernetes (K8s) cluster (based on containerd)

1. Experimental environment

---

1. k8s environment

Version v1.26.5
binary installation Kubernetes (K8s) cluster (based on containerd)—installation tutorial from scratch (with certificate)

CPU name	IP	system version	install service
master01	10.10.10.21	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master02	10.10.10.22	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master03	10.10.10.23	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
node01	10.10.10.24	rhel7.5	nginx、kubelet、proxy
node02	10.10.10.25	rhel7.5	nginx、kubelet、proxy

2. Prometheus+Grafana environment

Prometheus+Grafana monitoring system

CPU name	IP	system version
jenkins	10.10.10.10	rhel7.5

3. Prometheus deployment method

• Prometheus inside kubernetes monitors k8s cluster
  • Prometheus monitoring internal K8S is to deploy Prometheus in the K8S cluster, such as deploying it under the monitoring namespace of the K8S cluster, because K8S automatically creates a serviceAccount and the corresponding Secret in all namespaces with its own token and ca for accessing the K8S API. So there is no need to manually create serviceAccount and Secret.
• Prometheus monitoring k8s outside kubernetes
  • Prometheus external to kubernetes monitors external K8S by deploying Prometheus on a virtual machine. You need to manually specify the API address, ca and token in Prometheus.yaml.

4. Version correspondence

https://github.com/kubernetes/kube-state-metrics

5. Collection method

• Exporter: It is a tool that converts indicators of third-party components into a format that Prometheus can recognize and exposes them as crawl targets. In Kubernetes, there are many third-party components (such as Etcd, Kube-proxy, Node exporter, etc.) that also generate important monitoring indicators.

• kube-state-metrics: It is an independent component used to expose the status indicators of various resources (such as Pod, Service, Deployment, etc.) in the Kubernetes cluster. It is more convenient to use. Here we use this method

2. Configure kube-state-metrics

---

https://github.com/kubernetes/kube-state-metrics/tree/v2.9.2/examples/standard

1. File download

[root@master01 kube-state-metrics]# ls
cluster-role-binding.yaml  cluster-role.yaml  deployment.yaml  service-account.yaml  service.yaml

[root@master01 kube-state-metrics]# cat cluster-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.9.2
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: kube-system

[root@master01 kube-state-metrics]# cat cluster-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.9.2
  name: kube-state-metrics
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - secrets
  - nodes
  - pods
  - services
  - serviceaccounts
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  - endpoints
  verbs:
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - list
  - watch
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  - volumeattachments
  verbs:
  - list
  - watch
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  - ingressclasses
  - ingresses
  verbs:
  - list
  - watch
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - list
  - watch
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterrolebindings
  - clusterroles
  - rolebindings
  - roles
  verbs:
  - list
  - watch

[root@master01 kube-state-metrics]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.9.2
  name: kube-state-metrics
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  template:
    metadata:
      labels:
        app.kubernetes.io/component: exporter
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: 2.9.2
    spec:
      automountServiceAccountToken: true
      containers:
      - image: bitnami/kube-state-metrics:2.9.2
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
        name: kube-state-metrics
        ports:
        - containerPort: 8080
          name: http-metrics
        - containerPort: 8081
          name: telemetry
        readinessProbe:
          httpGet:
            path: /
            port: 8081
          initialDelaySeconds: 5
          timeoutSeconds: 5
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 65534
          seccompProfile:
            type: RuntimeDefault
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: kube-state-metrics

[root@master01 kube-state-metrics]# cat service-account.yaml
apiVersion: v1
automountServiceAccountToken: false
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.9.2
  name: kube-state-metrics
  namespace: kube-system

[root@master01 kube-state-metrics]# cat service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/version: 2.9.2
  name: kube-state-metrics
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - name: http-metrics
    port: 8080
    targetPort: 8080
    nodePort: 32080
    protocol: TCP
  - name: telemetry
    port: 8081
    targetPort: 8081
    nodePort: 32081
    protocol: TCP
  selector:
    app.kubernetes.io/name: kube-state-metrics

2. Install kube-state-metrics

Exposing ports using NodePort

[root@master01 kube-state-metrics]# kubectl apply -f ./
[root@master01 kube-state-metrics]# kubectl  get po -n kube-system -o wide | grep kube-state-metrics
kube-state-metrics-57ddc8c4ff-krsh2   1/1     Running   0             9m5s   10.0.3.1      master02   <none>           <none>

[root@master01 kube-state-metrics]# kubectl get svc -n kube-system | grep kube-state-metrics
kube-state-metrics   NodePort    10.97.38.90     <none>        8080:32080/TCP,8081:32081/TCP   9m17s

3. Test results

It was found that it was deployed on master02, which is 10.10.10.22

[root@master01 kube-state-metrics]# curl http://10.97.38.90:8080/healthz -w '\n'
OK

3. Configure Prometheus

---

1. Modify prometheus.yml

[root@jenkins ~]# cat Prometheus/prometheus.yml
  - job_name: "kube-state-metrics"
    static_configs:
      - targets: ["10.10.10.22:32080"]

  - job_name: "kube-state-telemetry"
    static_configs:
      - targets: ["10.10.10.22:32081"]

2. Restart Prometheus

[root@jenkins ~]# docker ps -a
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                       PORTS                    NAMES
a0497377cd82        grafana/grafana-enterprise          "/run.sh"                13 days ago         Up 3 minutes                 0.0.0.0:3000->3000/tcp   grafana
3e0e4270bd92        prom/prometheus                     "/bin/prometheus --c…"   13 days ago         Up 3 minutes                 0.0.0.0:9090->9090/tcp   prometheus

[root@jenkins ~]# docker restart 3e0e4270bd92

3. Log in to view results

4. Configure Grafana

---

Recommended templates: 13332, 13824, 14518

1. Import template

2. View the results