grafana alarm configuration

The alarm function is supported in grafana 4 and above, which makes it more complete for us to use grafana as a monitoring panel, because the alarm is an indispensable link in the monitoring system, and grafana supports many forms of alarm functions, such as email, nails, slack , Webhook, etc., let’s test email and DingTalk.

email alert

To enable email alarms, you need to enable the SMTP service in the startup configuration file /etc/grafana/grafan.ini. Here we also use a ConfigMap resource object to mount to the grafana Pod: (grafana-cm.yaml)

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-config
  namespace: kube-ops
data:
  grafana.ini: |
    [smtp]
    enabled = true
    host = smtp.163.com:25
    user = [email protected]
    password = <邮箱密码>
    skip_verify = true
    from_address = [email protected]
    [alerting]
    enabled = true
    execute_alerts = true

Of course we have to mount this ConfigMap file to the Pod:

        volumeMounts:
        - mountPath: /var/lib/grafana
          subPath: grafana
          name: storage
        - mountPath: "/etc/grafana"
          name: config
      securityContext:
        fsGroup: 472
        runAsUser: 472
      volumes:
      - name: storage
        persistentVolumeClaim:
          claimName: grafana
      - name: config
        configMap:
          name: grafana-config

Create a ConfigMap object and update the Deployment:

$ kubectl create -f grafana-cm.yaml
$ kubectl apply -f grafana-deploy.yaml

After the update is complete, test the email alert in the Alert page of grafana's webui: After
sending the test, you can receive the test alert email under normal circumstances:

Dingding alarm

ignore

Currently only Graph supports the alarm function, so we select Graph related graphs, click Edit, and enter the Graph edit page. You can see that there is an Alert module. Switch over to create an alarm:
Then configure the relevant parameters:

1、Alert 名称，可以自定义。
2、执行的频率，这里我选择每60s检测一次。
3、判断标准，默认是 avg，这里是下拉框，自己按需求选择。
4、query（A,5m,now），字母A代表选择的metrics 中设置的 sql，也可以选择其它在 metrics中设置的，但这里是单选。5m代表从现在起往之前的五分钟，即5m之前的那个点为时间的起始点，now为时间的结束点，此外这里可以自己手动输入时间。
5、设置的预警临界点，这里手动输入，和6是同样功能，6可以手动移动，两种操作是等同的。

Then you need to set the alarm sending information, click on the Notifications on the side:
Send to is the name of the email and DingTalk alarm channel that we configured earlier.

After the configuration is complete, you need to save this graph, otherwise sending the alarm may fail, and then click Test Rule in the Alert area to test the alarm rule, and then the email and Dingding can normally receive the alarm information.

Email alarm information:
This is the end of the use of grafana to display the monitoring chart information and alarm configuration of the Kubernetes cluster, but we can clearly feel that the advantage of grafana lies in the display of the chart, the alarm function is a bit weak, so in general, in the production environment We will not directly use the alarm function of grafana. We will use the more powerful AlertManager. We will introduce it to you in the next class.