prometheus告警email+企业微信机器人告警一起配置和书写一个k8s的webhook

1.在alertmanager创建同时的告警文件

global:
       resolve_timeout: 5m
       smtp_smarthost: 'smtp.qq.com:465'                                                      
       smtp_from: '[email protected]'                                                          
       smtp_auth_username: '[email protected]'                                                 
       smtp_auth_password: 'lrxxxxxxxxtubdfd'                                                 
       smtp_require_tls: false                                                                
       smtp_hello: 'qq.com'

templates:
        - '/etc/alertmanager/template/*.tmpl'

route:
      group_by: ['alertname']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 30m
      receiver: 'email' 
      routes:
        - receiver: 'web.hook'
          match:
            severity: Warning

receivers:
    - name: 'email'
      email_configs:                                                                          
      - to: '[email protected]'                                                               
        html: '{
    
    { template "email.to.html" . }}'
        headers: { Subject: "告警" }
        send_resolved: true    
    - name: 'web.hook'
      webhook_configs:
      - url: 'http://prometheus-webhook-dingtalk.kube-system.svc.cluster.local:8060'   # 这里是指定的k8s的webhook，下面有写
        send_resolved: true

inhibit_rules:
      - source_match:
           severity: 'critical'
        target_match:
           severity: 'warning'
        equal: ['alertname', 'dev', 'instance']

自定义模板这里有写

(2条消息) prometheus自定义邮件告警和自定义微信机器人告警_kali_yao的博客-CSDN博客

2.书写k8s的webhook

]# vim webhook.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: prometheus-webhook-dingtalk
  name: prometheus-webhook-dingtalk
  namespace: kube-system
spec:
  selector:
    matchLabels:
      run: prometheus-webhook-dingtalk
  template:
    metadata:
      labels:
        run: prometheus-webhook-dingtalk
    spec:
      volumes:
      - name: app                
        hostPath:                     
           path: /data/prometheus/Dockerfile2/webkook/app.py  # app.py下有链接可查看书写
      - name: localtime              
        hostPath:                     
           path: /etc/localtime
      containers:
       - name: prometheus-webhook-dingtalk
         image: webhook:v1
         env:
         - name: ROBOT_TOKEN
           value: "eb25f0c4-69ac-458c-af07-a6999beb05cd"   # 机器人的 key
         volumeMounts:
         - name: app
           mountPath: /src/app.py
         - name: localtime
           mountPath: /etc/localtime
         ports:
         - containerPort: 5000
           protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    run: prometheus-webhook-dingtalk
  name: prometheus-webhook-dingtalk
  namespace: kube-system
spec:
  ports:
  - port: 8060
    protocol: TCP
    targetPort: 5000
  selector:
    run: prometheus-webhook-dingtalk
  type: ClusterIP

# 创建
~]# kubectl apply -f webhook.yaml

这里有app.py的书写和webhook的镜像制作

(2条消息) prometheus自定义邮件告警和自定义微信机器人告警_kali_yao的博客-CSDN博客