High Availability Kubernetes Cluster-14. Deploy Kubernetes Cluster Performance Monitoring Platform

Reference documentation:

1. Github introduction:https://github.com/kubernetes/heapster
2. Github yaml file: https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb

   或者（source code）：https://github.com/kubernetes/heapster/releases

3. cAdvisor：https://github.com/google/cadvisor
4. Influxdb：http://influxdb.com
5. Grafana ： http://grafana.org

 

The open source software cAdvisor ( Container Advisor ) is used to monitor the running status of the container on the node . It has been integrated into the kubelet component by default, and the tcp 4194 port is used by default.

In large-scale container clusters, the Heapster+Influxdb+Grafana platform is generally used to collect, store, and display cluster performance data.

1. environment

1. basic environment

components	Version	Remark
governor	v1.9.2
heapster	v1.5.1
Influxdb	v1.3.3
gravitation	v4.4.3

1. principle

1. Heapster : The data collection and aggregation system of cAdvisor of each node in the cluster, by calling the api of kubelet on the node , and then calling the api of cAdvisor through kubelet to collect the performance data of all containers on the node. Heapster aggregates performance data and saves the results to the back-end storage system. Heapster supports a variety of back-end storage systems, such as memory , Influxdb , etc.
2. Influxdb : Distributed time series database (each record has a timestamp attribute), mainly used for real-time data collection, time tracking records, storage time charts, raw data, etc. Influxdb provides rest api for data storage and query.
3. Grafana : Display the time series data in Influxdb in the form of graphs or curves through the dashboard , which is convenient for viewing the running status of the cluster.
4. Heapster , Influxdb , and Grafana all start and run in the form of Pods .

   1. Deploy Kubernetes cluster performance monitoring

   1. prepare images

When deploying services in kubernetes, in order to avoid the problem of time-out of pull images during deployment, it is recommended to pull related images to all relevant nodes in advance (take kubenode1 as an example below), or build a local image system.

1. The basic environment has been accelerated by mirroring, please refer to: http://www.cnblogs.com/netonline/p/7420188.html
2. The image that needs to be pulled from gcr.io has used the "Create Auto-Build GitHub" function of Docker Hub (Docker Hub uses the Dockerfile file on GitHub to build the image). The build is successful in the personal Docker Hub and can be directly pulled to the local use.

 

# heapster

[root@kubenode1 ~]# docker pull netonline/heapster-amd64:v1.5.1

 

# influxdb

[root@kubenode1 ~]# docker pull netonline/heapster-influxdb-amd64:v1.3.3

 

# grafana

[root@kubenode1 ~]# docker pull netonline/heapster-grafana-amd64:v4.4.3

1. download yaml template

# release download page: https://github.com/kubernetes/heapster/releases

# The yaml template in the release is sometimes larger than https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb

yaml new, but little difference

[root@kubenode1 ~]# cd /usr/local/src/

[root@kubenode1 src]# wget -O heapster-v1.5.1.tar.gz https://github.com/kubernetes/heapster/archive/v1.5.1.tar.gz

 

# The yaml template is in the heapster/deploy/kube-config/influxdb directory, and another heapster-rbac.yaml is in the heapster/deploy/kube-config/rbac directory, both of which have the same directory structure as github

[root@kubenode1 src]# tar -zxvf heapster-v1.5.1.tar.gz -C /usr/local/

[root@kubenode1 src]# mv /usr/local/heapster-1.5.1 /usr/local/heapster

 

This experiment uses the yaml file (modified version): https://github.com/Netonline2016/kubernetes/tree/master/addons/heapster

1. heapster-rbac.yaml

# heapster needs to request the node list from kubernetes-master, and needs to set the corresponding permissions;

# By default, there is no need to modify heapster-rbac.yaml. Use the ClusterRole that comes with the kubernetes cluster : system:heapster as the ClusterRoleBinding to complete the authorization

[root@kubenode1 ~]# cd /usr/local/heapster/deploy/kube-config/rbac/

[root@kubenode1 rbac]# cat heapster-rbac.yaml

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1beta1

metadata:

name: heapster

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: system:heapster

subjects:

- kind: ServiceAccount

name: heapster

namespace: kube-system

1. heapster.yaml

hepster.yaml consists of 3 modules: ServiceAccout, Deployment, Service.

ServiceAccount

By default, there is no need to modify the ServiceAccount part, set the ServiceAccount resource, and obtain the permissions defined in rbac.

Deployment

# Modification: line 23, change the image name;

# --source: Configure the collection source and use the secure port to call the kubernetes cluster api;

# --sink: Configure the backend storage as influxdb; the address uses the service name of influxdb, and the cluster dns needs to work normally. If the dns service is not configured, the service's ClusterIP address can be used

[root@kubenode1 ~]# cd /usr/local/heapster/deploy/kube-config/influxdb/

[root@kubenode1 influxdb]# sed -i 's|gcr.io/google_containers/heapster-amd64:v1.5.1|netonline/heapster-amd64:v1.5.1|g' heapster.yaml

[root @ kubenode1 influxdb] # cat heapster.yaml

……

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: heapster

namespace: kube-system

spec:

replicas: 1

template:

metadata:

labels:

task: monitoring

k8s-app: heapster

spec:

serviceAccountName: heapster

containers:

- name: heapster

image: netonline/heapster-amd64:v1.5.1

imagePullPolicy: IfNotPresent

command:

- /heapster

- --source = kubernetes: https: //kubernetes.default

- --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086

……

Service

By default, the Service section does not need to be modified.

1. influxdb.yaml

influxdb.yaml consists of 2 modules: Deployment, Service.

Deployment

# Modification: Line 16, change the image name;

[root@kubenode1 influxdb]# sed -i 's|gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3|netonline/heapster-influxdb-amd64:v1.3.3|g' influxdb.yaml

Service

There is no need to modify the Service part by default, just pay attention to the correspondence of the Service name.

1. grafana.yaml

grafana.yaml consists of 2 modules: Deployment, Service.

Deployment

# Modification: Line 16, change the image name;

# Modification: Line 43, uncomment; after the value of "GF_SERVER_ROOT_URL" is set, grafana can only be accessed through API Server proxy;

# Modification: line 44, comment this line;

# The value of INFLUXDB_HOST is set to the service name of influxdb, depending on the cluster dns, or directly using ClusterIP

[root @ kubenode1 influxdb] # sed -i 's | gcr.io/google_containers/heapster-grafana-amd64: v4.4.3 | netonline / heapster-grafana-amd64: v4.4.3 | g' grafana.yaml

[root@kubenode1 influxdb]# sed -i '43s|# value:|value:|g' grafana.yaml

[root@kubenode1 influxdb]# sed -i '44s|value:|# value:|g' grafana.yaml

[root @ kubenode1 influxdb] # cat grafana.yaml

……

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: monitoring-grafana

namespace: kube-system

spec:

replicas: 1

template:

metadata:

labels:

task: monitoring

k8s-app: grafana

spec:

containers:

- name: grafana

image: netonline / heapster-grafana-amd64: v4.4.3

ports:

- containerPort: 3000

protocol: TCP

volumeMounts:

- mountPath: /etc/ssl/certs

name: ca-certificates

readOnly: true

- mountPath: /var

name: grafana-storage

env:

- name: INFLUXDB_HOST

value: monitoring-influxdb

- name: GF_SERVER_HTTP_PORT

value: "3000"

# The following env variables are required to make Grafana accessible via

# the kubernetes api-server proxy. On production clusters, we recommend

# removing these env variables, setup auth for grafana, and expose the grafana

# service using a LoadBalancer or a public IP.

- name: GF_AUTH_BASIC_ENABLED

value: "false"

- name: GF_AUTH_ANONYMOUS_ENABLED

value: "true"

- name: GF_AUTH_ANONYMOUS_ORG_ROLE

value: Admin

- name: GF_SERVER_ROOT_URL

# If you're only using the API Server proxy, set this value instead:

value: /api/v1/namespaces/kube-system/services/monitoring-grafana/proxy

# value: /

volumes:

- name: ca-certificates

hostPath:

path: /etc/ssl/certs

- name: grafana-storage

emptyDir: {}

……

Service

There is no need to modify the Service part by default, just pay attention to the correspondence of the Service name.

1. verify

1. Start monitoring related services

# Copy heapster-rbac.yaml to influxdb/ directory;

[root@kubenode1 ~]# cd /usr/local/heapster/deploy/kube-config/influxdb/

[root@kubenode1 influxdb]# cp /usr/local/heapster/deploy/kube-config/rbac/heapster-rbac.yaml .

[root@kubenode1 influxdb]# kubectl create -f .

 

1. View related services

# View deployment and Pod running status

[root@kubenode1 ~]# kubectl get deploy -n kube-system | grep -E 'heapster|monitoring'

[root@kubenode1 ~]# kubectl get pods -n kube-system | grep -E 'heapster|monitoring'

# View service running status

[root@kubenode1 ~]# kubectl get svc -n kube-system | grep -E 'heapster|monitoring'

1. Visit the dashboard

Browser access to access dashboard: https://172.30.200.10:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

Note: When Dasheboard is not configured with hepster monitoring platform, it cannot display metric graphs such as CPU and memory of node and Pod resources.

Node resource CPU/memory metric graph:

Pod resource CPU/memory metric graph:

1. visit grafana

# Access via kube-apiserver

[root@kubenode1 ~]# kubectl cluster-info

 

Browser access to access dashboard: https://172.30.200.10:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy

Cluster node information:

Pod information: