Install Prometheus and Grafana monitoring on Kubernetes (k8s)

Environmental preparation

Kubernetes preparation

Of course, the prerequisite environment is that you must first have a Kubernetes cluster with a version between v1.21.*~v1.27.*. Of course, I have already prepared Kubernetes:

You can see that the Kubernetes version I prepared is 1.21.14, which meets the requirements. This article also uses this version for installation. The installation steps for the version mentioned above are generally the same as this version. Just follow the steps.

Installation project

Because it is troublesome to install Prometheus on Kubernetes, some big guys have contributed some projects to install them on Github. The project I recommend this time iskube-prometheus, the address is ：https://github.com/prometheus-operator/kube-prometheus, but you should pay attention to the version of kube-prometheus installed and the corresponding list of versions :

Kubernetes version	Kube-prometheus version
1.21.*	0.9.0
1.22.*	0.9.0、0.10.0
1.23.*	0.10.0、0.11.0
1.24.*	0.11.0、0.12.0
1.25.*	0.12.0
1.26.*	main
1.27.*	main

As shown in the picture:

It should be noted that it is best not to use projects in the main branch, because it has not been officially released yet, so it is recommended to use Kubernetes below 1.25.*

start installation

Download and install the project

Go to this URL:https://github.com/prometheus-operator/kube-prometheus/releases and select the corresponding version Download the release version, because my Kubernetes installation is 1.21.*, so I downloaded version 0.9.0:

After downloading, it looks like this:

Installation project

Let’s unzip this project first:

tar -zxvf kube-prometheus-0.9.0.tar.gz

After decompression, a kube-prometheus-0.9.0 folder will appear as shown in the figure:

Let’s go to the kube-prometheus-0.9.0/manifests directory first:

cd kube-prometheus-0.9.0/manifests/

Replace image

Because some of the mirrors are pulled from k8s.gcr.io, this mirror is on Google and cannot be pulled from China, so we need to modify the pull address. Because someone has already provided a replacement for this image on dockerhub, we can directly replace it here

Replace kube-state-metrics

Modify ./kube-state-metrics-deployment.yaml file (in kube-prometheus-0.9.0/manifests/kube-state-metrics-deployment.yaml):

vim kube-state-metrics-deployment.yaml

There is k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.1.1 image at about line 34 (the version number may be different, which does not affect the installation):

Check the most popular ones on dockerhub:

Select the same version number (⚠️Note that this version number does not have the v prefix):

So we change k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.1.1 to docker.io/bitnami/kube-state-metrics:2.1.1 (plus docker.io prefix):

Replace prometheus-adapter

Modify ./prometheus-adapter-deployment.yaml file (in kube-prometheus-0.9.0/manifests/prometheus-adapter-deployment.yaml):

vim ./prometheus-adapter-deployment.yaml

There is k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.0 image at about line 34 (the version number may be different, which does not affect the installation):

Here we come to dockerhub to search, and we also use this most downloaded and collected one:

Also pay attention to the version number correspondence:

So we change k8s.gcr.io/prometheus-adapter/prometheus-adapter:v0.9.0 to docker.io/willdockerhub/prometheus-adapter:v0.9.0 (plus docker.io prefix):

Modify Service

Because the type of some Service settings that we need to access externally is ClusterIP, we need to change it to NodePort or LoadBalancer.

Modify alertmanager-service.yaml

This file is in kube-prometheus-0.9.0/manifests/alertmanager-service.yaml , we modify it directly:

vim alertmanager-service.yaml

Follow the picture prompts to modify, add a new line under and modify the type to , and then < /span> Port:. This port can be specified by yourself. and modify the nodePort port to Add a new line belowspectype: NodePortNodePortspec.portsnodePort: 300933009330000~32767

Save and exit after modification is completed.

Modify grafana-service.yaml

This file is in kube-prometheus-0.9.0/manifests/grafana-service.yaml , we modify it directly:

vim grafana-service.yaml

The steps are the same, so I won’t go into details here, but be careful not to conflict with the port numbers:

Modify prometheus-service.yaml

This file is in kube-prometheus-0.9.0/manifests/prometheus-service.yaml , we modify it directly:

vim prometheus-service.yaml

The steps are the same, so I won’t go into details here, but be careful not to conflict with the port numbers:

Execute these yaml resource manifest files

Come to kube-prometheus-0.9.0 directory first:

Then execute the following lines of commands:

kubectl apply --server-side -f manifests/setup
kubectl wait \
	--for condition=Established \
	--all CustomResourceDefinition \
	--namespace=monitoring
kubectl apply -f manifests/

The execution result is as shown in the figure, a bunch of resources are created:

We use the following command to check whether the creation is successful:

watch kubectl get pod -o wide -n monitoring

After the installation is completed, the values ​​of STATUS are all Running, and the values ​​of READY are all correct:

Visit Grafana

Use this line of command to view grafana’s port:

kubectl get svc -o wide -n monitoring | grep grafana

You can see that the nodePort port for external access by grafana is 30300, which is also the value we set before:

We can access grafana directly by accessing the IP address of any node through the browser and bringing this port:

The default username and password areadmin/admin. We can log in directly. After logging in, we need to reset the password and set a new password for your admin user:

Next we visit Grafana’s homepage:

Follow my prompts and click Dashboards -> Manage:

Then expand this Default:

You can see that there are many kube-prometheus Dashboards that have been prepared for us:

We select Node Exporter/Nodes’s Dashboard to enter:

Here you can see the CPU and memory information of each node:

point Kubernetes/Networking/Cluster progress:

You can see the network access status under each namespace;

Okay, you can check out more Dashboards yourself.

Access to other systems

Prometheus

Use the following command to view Prometheus’s NodePort:

kubectl get svc -o wide -n monitoring | grep prometheus-k8s

You can see that the NodePort port of Prometheus is 30090 which we set ourselves:

Access can directly go to the home page of Prometheus:

Then you can do whatever you want with Prometheus

AlertManager

We can directly query the NodePort port of alertmanager through the following command:

kubectl get svc -o wide -n monitoring | grep alertmanager-main

You can see that the port is still 30093 we set before:

You can also access the AlertManager homepage by accessing port 30093 of any of our nodes:

Okay, the steps to build Prometheus and Grafana on Kubernetes have been completed. I wish you happy coding.