Foreword
In a monitoring system, if the data link is her skeleton, then the alarm notification service is his soul! All monitoring service is to be able to notify it, reduce manual query status, to identify problems, avoid unnecessary massive failures, government enterprises save money, and ensure the security and existence.
They are able to find the problem is very important, more important is to find problems quickly let people know, that's to say today, alarm notification service.
An open source project PrometheusAlert
This project can give a lot of third-party service docking, telephone, SMS and other alarm mode, it is what we need to first deploy.
Reference deployment project README.mdof 部署方式the section should be noted that his configuration file must be in the current directory binaries, conf/app.confthat name will be read.
The reason is to use beegothe framework, read the default configuration file in this position, if there is no compliance with the binary file, you can compile your own.
GOPATH=xxxx/monitor_alert CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o xxx/monitor_alert/bin/PrometheusAlertLinuxAmd64 xxx/monitor_alert/src/PrometheusAlert/PrometheusAlert.go
GOPATH=xxxx/monitor_alert CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -o xxx/monitor_alert/bin/PrometheusAlertLinuxArm64 xxx/monitor_alert/src/PrometheusAlert/PrometheusAlert.go
GOPATH=xxxx/monitor_alert CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -o xxx/monitor_alert/bin/PrometheusAlertMacAmd64 xxx/monitor_alert/src/PrometheusAlert/PrometheusAlert.go
According to gothe characteristics of the language, we need to create a project, and then in the srcdirectory decentralization project and then compile. For example, I created a project calledmonitor_alert
After the end of the deployment of the project to save links, here is my dockerstart in the resulting linkhttp://172.17.0.3:8080
Prometheus docking alarm service
First you have to build up a complete set of Prometheus, like on the map, like, if you have not built up before I can look at the article that's Prometheus .
Installing alertmanageralarms service (prometheus common alerting service)
- create
alertmanager.yaml
global:
resolve_timeout: 5m
route:
group_by: ['instance']
group_wait: 10m
group_interval: 10s
repeat_interval: 10m
receiver: 'web.hook.prometheusalert'
receivers:
- name: 'web.hook.prometheusalert'
webhook_configs:
- url: 'http://172.17.0.3:8080/prometheus/alert'
This configuration file open webhookservice, generated when the alarm will automatically call this api, this apican customize, my link is on a service url deployment http://172.17.0.3:8080/prometheus/alert.
- start up
alertmanager
使用docker启动，把刚刚创建配置文件挂载上去，暴露端口9093（默认）
docker run -d -p 9093:9093 -v ~/Documents/code/docker/prometheus/alertmanager.yaml:/etc/alertmanager/alertmanager.yml --name alertmanager prom/alertmanager:latest
启动效果：
可以在Status里看到你刚刚的webhook配置内容
增加prometheus与告警相关的配置文件。
- 创建
prometheus.rules.yml文件
groups:
- name: node_alert
rules:
- alert: 主机CPU告警
expr: node_load1 > 0.01
labels:
name: prometheusalertcenter
level: 3
annotations:
description: "{{ $labels.instance }} CPU load占用过高"
mobile: 176xxxxxxxx
- 启动prometheus
用docker启动
docker run -d -p 9090:9090 -v ~/Documents/code/docker/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml -v ~/Documents/code/docker/prometheus/prometheus.rules.yml:/etc/prometheus/prometheus.rules.yml --name prometheus-server prom/prometheus
注意了prometheus.yml的配置文件是参考这篇文章 这就是Prometheus。
prometheus.rules.yml就是刚刚的文件。
启动效果：
看，这里产生告警，你可以修改自己的告警阀值，我这里设置的大于1%就告警所以立马就告警出来了。
稍后短信就出来了。
基于granfna告警对接
- 新建告警通道
Name任意，Type选webhook，Url配置为 http://172.17.0.3:8080/grafana/txdx，其中的ip就是刚刚安装了prometheus_alert服务的ip。
搞定以后选Save保存。
- 配置各指标的告警规则
进入面板编辑各个指标
可以先把阀值设置低一些，测试短信发送
加号选中你刚刚设置的告警通道，message 可不写，会自动查询grafana得到告警情况。
右上角保存
怎么对接到腾讯云短信平台？
服务器出网调用腾讯云短信服务，采用加密传输方式。
采用SDKAppID 和 AppKey 作为安全密钥，同时需要短信签名和模板，具体加密方式见腾讯云公有云短信服务加密方式
怎么申请
短信服务开通可以访问 腾讯云短信服务，初始会免费赠送100条 。
短信条数不够可以访问购买短信包网页
对接准备
1、申请 SDK AppID 以及 App Key： 在开始本教程之前，您需要先获取 SDK AppID 和 App Key，如您尚未申请，请到 短信控制台 中添加应用。应用添加成功后您将获得 SDK AppID 以及 App Key。
注意： SDK AppID 是以 14xxxxx 开头。
2、申请签名： 下发短信必须携带签名，您可以在 短信控制台 中
注意，假如是政府机关在这里需要选择，然后提交资料，其他就按照自己的实际情况填写就好。
3.、申请模板： 下发短信内容必须经过审核，您可以在短信控制台中申请短信模板
完成以上三项就会拿到appid,appkey,模板 id,你的签名，四个东西，便可开始填入刚刚的开源项目配置文件，或者自己进行代码开发。
如果喜欢用API可以参考 文档
开始对接
进入你部署的prometheus_alert配置 目录，例如xxx/prometheus_alert/conf/目录，编辑app.conf文件
找到defaultphone=176xxxxxxxx,123123123修改为你要的手机号，可以是单个也可以是多个，多个要用英文半角逗号隔开
修改以下配置open-txdx为1表示打开腾讯云短信告警
open-txdx=1
TXY_DX_appkey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TXY_DX_tpl_id=54xxxx
TXY_DX_sdkappid=140032xxxx
TXY_DX_sign=你的腾讯云短信平台签名
重启服务使配置生效即可。
