Kafka monitoring of K8S (Prometheus + Grafana)

For Kafka deployed on K8S, Prometheus + Grafana is a commonly used monitoring solution. Today, we will come to actual combat to monitor Kafka in K8S environment through Prometheus + Grafana;

Ready to work

Today ’s focus is on Kafka monitoring, so K8S, Helm, Kafka, Prometheus, Grafana and other services are all ready. The following provides some links, which can be used as a reference when you make related deployments:

Build K8S: "kubespray2.11 install kubernetes1.15"
Build Helm: "Deploy and Experience Helm (Version 2.16.1)"
Build Prometheus and Grafana: "kubernetes1.15 rapid deployment of prometheus and grafana"
Deploying Kafka: "Rapid deployment of Kafka in K8S environment (accessible outside K8S)"

Version Information

Governors ： 1.15
Kubernetes host: CentOS Linux release 7.7.1908
NFS service: IP address 192.168.50.135, folder / volume1 / nfs-storageclass-test
Helm：2.16.1
Kafka：2.0.1
Zookeeper：3.5.5
Prometheus：2.0.0
Grafana ： 5.0.0

After preparation, you can start actual combat;

Confirm kafka-exporter parameters

The kafka-exporter to be deployed needs to obtain data from kafka, so kafka-exporter parameters must be prepared

Check the Kafka service where the TYPE is ClusterIP, as shown in the red box below:
The service name in the above red box is kafka and the port is 9092 , so the kafka information configured later in kafka-exporter is kafka: 9092

Actual operation

Add the Helm warehouse (the warehouse has the kafka-exporter we need): helm repo add gkarthiks https://gkarthiks.github.io/helm-charts
下载kafka-exporter：helm fetch gkarthiks/prometheus-kafka-exporter
Unzip the downloaded chart file: tar -zxvf prometheus-kafka-exporter-0.1.0.tgz
Enter the unzipped directory: cd prometheus-kafka-exporter
Modify the values.yaml file, as shown in the red box below, kafka: 9092 is the address to access kafka under the same namespace:
Execute in the directory where the values.yaml file is located: helm install --name-template kafka-exporter -f values.yaml. --Namespace kafka-test (kafka's namespace is kafka-test, and the namespace here must be consistent with kafka)
Check whether the service and pod are ready:
At this point, we have deployed kafka-exporter and can provide prometheus with monitoring data, but prometheus does not know to come here to collect data, so next we need to configure prometheus;
Confirm the access address of the kafka-exporter service. Prometheus collects the data to be used. The name and port are shown in the red box in the figure below, so you can stitch the access address across the namespace: kafka-exporter-prometheus-kafka-exporter.kafka-test. svc.cluster.local: 9308
If you are referring to the prometheus deployed by "kubernetes1.15 rapid deployment of prometheus and grafana" , please open the configmap.yaml file downloaded during deployment , which contains the configuration of prometheus, if it is deployed by other means, please find prometheus according to your deployment .yml location;
configmap.yaml adds the content in the red box below, so that prometheus can collect kafka-exporter data:
Make the configuration effective: kubectl apply -f configmap.yaml
At this time, the prometheus container still uses the old configuration. In order for the configuration to take effect, the pod of prometheus must be deleted, so that the new pod automatically created by K8S uses the new configuration to find the pod of prometheus: kubectl get pods -n kube -system
Delete the old pod: kubectl delete pod prometheus-68545d4fd8-f5q2w -n kube-system
Wait for a new pod to be created automatically;
Next, log in to Grafana, configure the monitoring page, as shown below, do template import operation:
Enter the number 7589 on the page of import template:
When choosing a data source, you must choose prometheus, as shown below:
At this time, if your kafka has

sent and received messages, you can immediately see the data: At this point, the kafka monitoring in the K8S environment has been deployed, I hope this article can give you some reference;