Foreword
Container cluster management system Kubernetes (referred K8s), provides for the deployment of the application container operation, container scheduling, load balancing, service discovery and dynamic stretching and a series of full-featured, Prometheus for K8s support is great, it can automatically discover the monitoring target K8s ! Prometheus generated alarm can be forwarded to the Cloud Alert by Alertmanager, noise reduction, dispatching and notification alarms.
Kubernetes
K8s Google is open source cluster management system container. Cloud application platform for managing containerized on multiple hosts, K8s goal is to make application deployment container of simple and effective (powerful), K8s provides application deployment, planning, updating and maintenance of a mechanism. (Choose from as K8s Chinese document)
Master: k8s management node cluster, resource data is responsible for managing cluster, the cluster provides access to the entrance. It comprises four modules: kube-apiserver, kube-controller-manager and kube-scheduler.
Node: k8s serving node cluster architecture running Pod, each running at least Node container runtime (such docker or rkt), kubelet and kube-proxy service.
Installation and use of more K8s, see K8s Chinese documents. K8s to solve the two problems of traditional IT systems to expand and upgrade services and improve the convenience of large-scale container cluster management. K8s usually with promethues monitor.
Prometheus
Prometheus is an open source and community-driven project monitoring & alarm & timing database. Since its launch from providing a complete container-based deployment, developers can quickly build their own container-based monitoring platform.
Prometheus compatible monitor K8s well, mainly to monitor K8s:
Node: If the host CPU, memory, network throughput and bandwidth usage, disk I / O and disk usage and other indicators. node-exporter acquisition.
容器关键指标:集群中容器的 CPU 详细状况，内存详细状况，Network，FileSystem和Subcontainer 等。通过 cadvisor 采集。
K8s 集群上部署的应用：监控部署在 K8s 集群上的应用。主要是 pod，service，ingress 和 endpoint。通过 black-box 和 kube-apiserver 的接口采集。
产生的告警，promethues 支持 Email 推送和 webhook 推送，如果要实现电话、短信、微信的推送，如果不想再造轮子，可以配合 Cloud Alert（原OneAlert）或 pagerduty 使用。本文以Cloud Alert 为例详细描述集成步骤，pagerduty的集成方式类似。
Cloud Alert
Cloud Alert（原OneAlert）是免费 SaaS 云告警监控项目，可以实现告警的降噪、分派、通知。通知方式支持电话、短信、微信、邮件、APP、钉钉、Slack通知等等。
-
注册 Cloud Alert 账号，注册地址。
-
创建 Prometheus 应用集成，集成 -> 监控工具，选择 Prometheus，保存后，即可获取 webhook 回调 url。
- 进入 Prometheus 界面，修改 Alertmanager 模块的配置文件。
receivers:
-name: 'team-X-pager'
webhook_configs:
-url: 'http://api.onealert.com/alert/api/event/prometheus/--'
send_resolved: true
- 重启下 Alertmanager 模块，加载该配置文件。结束。
Cloud Alert default all alarms are telephone, e-mail, text messages notify the user, if you want to set different alarms assigned to different people / team, you can configure -> dispatch policy settings. If more members of the company, it is recommended to use with scheduling function is not necessary for all full alarm response, the human liberation. Micro-channel notification, we need to bind Cloud Alert the micro-channel public number: cloud alarm, after binding, can receive micro-letter notified. Greater use of official documents refer to it.