Prometheus (Prometheus) is a combination of open & alarm monitoring time-series & databases, by starting SoundCloud developed. With the development, more and more companies and organizations to accept the use of Prometheus, society is also very active, they put it into a separate open source projects, and companies have to operate. Google SRE also mentioned in the book with them BorgMon monitoring system similar implementation is Prometheus. Now the most common Kubernetes container management system, usually with Prometheus monitor.

The basic principle is Prometheus HTTP protocol crawl status periodically monitored component, has the advantage that any component as long as the HTTP interface can access control system, or without any other SDK integration process. Doing so is ideal for virtualized environments such as VM or Docker.

Prometheus should be one of the few suitable Docker, Mesos, monitoring system Kubernetes environment.

Output monitored component information HTTP interface called exporter. At present the Internet company most commonly used components are exporter can be used directly, such as Varnish, Haproxy, Nginx, MySQL, Linux system information (including disk, memory, CPU, network, etc.), specific support sources see: https: // github.com/prometheus.

Compared with other monitoring systems, Prometheus main features are:

A multidimensional data model (time series by the set key indicators defined name and / median size).
Very efficient storage, a sample average data accounted ~ 3.5bytes, 3,200,000 time series of sampled every 30 seconds, for 60 days, the consumption of the disk about 228G.
A flexible query language.
Not rely on distributed storage, a single server node.
PULL model time set by the HTTP.
Support Pushed through intermediate gateways.
Static configuration or through service discovery to find the target.
Multi-mode graphics and dashboards support.

Two, Prometheus architectural overview

The figure illustrates the overall architecture of Prometheus (Prometheus) and some of its eco-system components:

Its service process is as Prometheus daemon is responsible for the timing crawl metrics (indicators) data on away goals, goals need to expose each grab an http service interface to its regular crawl.

Prometheus: supported through configuration files, text files, zookeeper, Consul, DNS SRV lookup, etc. specified crawl target. Support many ways chart visualization, such as very fine Grafana, comes Promdash, as well as provide their own template engine and so on, but also provides the HTTP API query, customize the output required.

Alertmanager: Prometheus is independent of a component that can be supported Prometheus query, provide a very flexible alarm mode.

PushGateway: This component is to support the Client metrics active push PushGateway, but the timing to fetch the data Prometheus on Gateway.

If a user is used statsd will think this is very similar, but statsd is sent directly to the server, and Prometheus mainly by the process of the initiative to crawl.

Most Prometheus components are written in Go, they can easily build and deploy static binaries. Access prometheus.io for complete documentation, examples and guidelines.

Three, Prometheus data model

Prometheus fundamentally all are stored in time series to achieve the same metrics (metric name) and label (or a plurality of tags) consisting of a time series, different label represents different time sequences. In order to support some queries, and sometimes produce some temporary time-series storage.

metrics name & label name and label indicators

Each time series is only "Index name" and a set of "tags (key = value)" in the form of a composition.

Index Name: General is to monitor the image from a name, for example http_requests_total so, it has some naming conventions, you can pack alphanumeric _ like the. Usually begins with the application name like _ _ monitoring of numeric types _ such units. For example: push_total, userlogin_mysql_duration_seconds, app_memory_usage_bytes.

Tags: is to identify the different dimensions of a time series, and for example, an http request using a POST or GET, what is its endpoint, this time we should go with a label marked. Such identification is finally formed of: http_requests_total {method = "POST", endpoint = "/ api / tracks"}.

Remember, for http_requests_total the metrics name tag either increase or remove tags will form a new time series.

According to the query can be combined with the above labels to query the results of the polymerization.

If the database to understand the traditional view of this statement, consider http_requests_total is the table name, the tag is a field, and timestamp is the primary key, there is a float64 field is worth it. (Prometheus which all values are by float64 storage).

Four, Prometheus four data types

Counter

Counter for the integrated value, for example, a request to record the number of times, the number of tasks completed, the number of errors. It has been increased, not reduced. After the restart process, it will be reset.

例如：http_response_total{method=”GET”,endpoint=”/api/tracks”} 100，10秒后抓取http_response_total{method=”GET”,endpoint=”/api/tracks”} 100。

Gauge

Gauge conventional value, such as temperature changes, changes in memory usage. Big variable, variable small. After the restart process, it will be reset.

For example: memory_usage_bytes {host = "master-01"} 100 <gripping value, memory_usage_bytes {host = "master-01"} 30, memory_usage_bytes {host = "master-01"} 50, memory_usage_bytes {host = "master-01 "} 80 <gripping value.

Histogram

Histogram (histogram) can be understood as meaning the histogram, commonly used scale for tracking events, such as: time-consuming request, response size. It is so special to group recorded content, providing functionality count and sum of all values.

For example: {less than 10 = 5, 1 = less than 20, less than 30 = 2}, count = 7 times, summation value SUM = 7 times.

Summary

Summary and Histogram are very similar, commonly used scale for tracking events, such as: time-consuming request, response size. Also provides functions count and sum of all values.

For example: count = 7 times, sum = 7 times the value of evaluation.

It provides a quantiles function, you can press% results than divide tracking. For example: quantile value of 0.95, which represents the sample values take 95% of the data.

Fifth, installation and operation Prometheus (docker version)

Here's how to use this machine Grafana Prometheus and server performance monitoring.

Monitoring the machine, only one exporter, node_exporter - data collection system for machine

Grafana is a feature-rich open source data visualization platform, commonly used for visualizing time series data. It built supports the following data sources:

Here is the architecture diagram we use when installing:

注意：本文使用的是ubuntu-18 桌面系统，只需要一台服务器即可！

下载镜像包

apt-get install -y docker.io #安装docker
docker pull prom/node-exporter
docker pull prom/prometheus
docker pull grafana/grafana

启动node-exporter

docker run --name node-exporter --restart=always -d -p 9100:9100 -v /root/docker/prom/proc:/host/proc:ro -v /root/docker/prom/sys:/host/sys:ro -v /root/docker/prom/rootfs/:/rootfs:ro prom/node-exporter

访问url：http://192.168.100.3:9100/metrics 效果如下：

这些都是收集到数据，有了它就可以做数据展示了

启动prometheus

新建目录/root/docker/prometheus，编辑配置文件prometheus.yml( vim prometheus.yml) 内容如下：

global:
  scrape_interval:     60s
  evaluation_interval: 60s
 
scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: prometheus
 
  - job_name: linux
    static_configs:
      - targets: ['192.168.100.3:9100']
        labels:
          instance: localhost

注意：修改IP地址，这里的192.168.91.132就是本机地址

启动prometheus

docker run --name prometheus --restart=always -d -p 9200:9090 -v /root/docker/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

访问url：http://192.168.100.3:9200/graph