Prometheus Alarm Rules CRUD automation
Preface:
With the combination of the development of container technology, zabbix monitoring methods and k8s imperfections, resulting had to give up zabbix, and the use of new monitoring tools prometheus on more and more. But after initial use, the original discovery prometheus too many settings to be added manually modify configuration files, this article describes how I solve prometheus "original."
surroundings:
Language: python2.7
web framework: flask
Summary:
Prometheus is provided by alarm setting file provided in prometheus.yml rule_files profile:
- /etc/prometheus/rules.d/*.yml
All alarms rule files in the directory /etc/prometheus/rules.d/, so we controlled alarm as to write and modify files in this directory.
1, adding an alarm rule
Prometheus page style is modeled on the interface to write some color and position size is not exactly the same (front-end tone headache).
To create alarm rules by entering the alarm name, expression, duration, and description, I will set the template is dead, input box set disabled property, alarm content filtering through the alarm script.
Main technique:
This page applications are more basic form submission, there is no difficulty, some of the pages to strengthen the effect of adding the operation is the only treatment for input and select events, input alarm content at the same time, the right side of the alarm inputs can be displayed simultaneously rules preview after the effect of the above FIG.
Js Code:
$("#rules_name").bind('input propertychange',function () {
var content = $("#rules_name").val();
var html = "- name: " + content + '_rules';
$("#name_rules").html(html);
var alerthtml = " - alert: " + content;
$("#alert_rules").html(alerthtml);
});
The input bound propertychange event (rules_name is input id label), enter the value input to produce change, and refresh content name_rules id corresponding to refresh, so you can view real-time content to enter the corresponding configuration file.
Finally, the form is submitted, the information written to the file to be submitted in the background:
[root@localhost rules.d]# cat service_100000.yml
# { 'Name': 'service_100000_rule', 'alert': 'service_100000', 'expr': 'service_count> 100000', '_for': '30s',' level ':' disaster ',' summary ':' Alarm host and port {{$ labels.instance}} alarm value: {{$ value}} ',' description ':' a service is greater than 100,000, the warning must be '}
groups:
- name: service_100000_rule
rules:
- alert: service_100000
expr: service_count>100000
for: 30s
labels:
level: "disaster"
service: "local-234"
annotations:
summary: "Alarm host and port {{$ labels.instance}} alarm value: {{$ value}}"
description: "Business is greater than 100,000, and must be a warning"
More service_100000.yml a document, the contents of the above, the first line is used to record the parameters, modifying the parameters for later use.
After the refresh prometheus configuration discovery, alarm came already loaded
2. Modify the alarm
修改告警规则应该是我们日常最多用到的，经常要修改阈值，首先，在选择框中会加载所有已经设置的rules规则，然后我们选中后（如果告警规则多之后，如何找到我们要修改的规则又是个问题，我是在select的基础上添加一个输入框，可以根据输入内容快速进行搜索，查找或者选择我们要修改的规则，如上图，输入ser就能弹出我们要找的规则），触发select事件：
$("#list_select_rules").change(function(){
var content=$("#list_select_rules").val();
jQuery.ajax({
type: "POST",
url: "/prometheus/getRulesDetail",
dataType: 'json',
data : {
"name": content
},
async: false,
error: function () {
alert("操作失败,请稍等片刻重新尝试,如仍有问题请联系管理员......");
return false;
},
success : function(result){
//{'name': '', 'alert': '', 'expr': '', '_for': '', 'level': '', 'summary': '', 'description': ''}
$("#name_rules").html("- name: " + result.alert + '_rules');
$("#alert_rules").html(" - alert: " + result.alert);
$("#rules_name").val(result.alert);
$("#expr_rules").html(" expr: " + result.expr);
$("#rules_expr").val(result.expr);
$("#time_rules").html(" for: " + result._for);
$("#rules_time").val(result._for);
$("#desc_rules").html(" level: " + result.description);
$("#rules_desc").val(result.description);
$("#select_rules").html(" level: " + result.level);
var add = $('#rules_select').val(result.level);
add.attr('selected',true);
}
})
});
When selected, select the tab content js trigger event to send the selected content to the background, alerting rules to obtain the corresponding information (information that is the first line of the note alarm rule out), and then input the corresponding position on the right side preview to refresh, you can after the realization of the selected rule, all the information directly displayed in the front, then submit, the background file modification, thus achieving modify alarm rules.
3, delete
Delete function is relatively simple, and update the same, when the interface is loaded by js loads all the alarm rules to select tab, and then set the select event, as shown above, I chose service_100000 this rule, put the rear warning alarm rule preview information show came out, js code as the top event of the update code. Then click the Delete button, the selected rule passed in the background, just delete the files.
At this point, add or delete alert rules prometheus the change is complete. Then I will refine them, hoping to achieve the same effect as zabbix.
Code Location: https: //github.com/1182640071/AddRulesWeb
My prometheus management platform Address: https://blog.csdn.net/www199202/article/details/90728824
This switched http://blog.sina.com.cn/s/blog_16eedc9400102yc8h.html