Prometheus Introduction

Prometheus is an open source monitoring system, use the Go language development, is Google BorgMon monitoring system similar implementations.

Prometheus is a basic principle of the HTTP protocol crawl status periodically monitored component, any component as long as the corresponding HTTP interface can access the monitoring, is more suitable Docker, one of the environmental monitoring system Kubernetes. Output monitoring information is referred to as HTTP interface Exporter .

Prometheus architecture

Prometheus using a Pullmodel, Prometheus Server to pull the respective target monitoring data via HTTP pull manner.

Retrieval : defines Prometheus Server is necessary to pull data from what place
- Jobs / Exporters : Prometheus monitoring data can be pulled from the Jobs or Exporters. Exporter Web API in the form of external exposure data acquisition interface.
- Prometheus Server : Prometheus can pull data from the other Prometheus Server.
- Pushgateway : For some instances of a component temporary Job runs, Prometheus may not have had time to pull from monitoring data, these Job has ended, you can run the monitoring data pushed to Pushgateway in Job runtime, Prometheus from the Pushgateway pull data, prevent data loss monitoring.
- -Service : Prometheus means can dynamically find some products, pull data monitor, as found from the DNS, Kubernetes, Consul in.
Storage : namely Prometheus storage, use of local storage Prometheus Server.
PromQL : Prometheus is a query, you can use PromQL and some of WEBUI (such as Grafana) Integration
AlertManager : is a separate external component Prometheus for alarm monitoring system can be configured through the configuration file a number of alarm rules, the alarm will push Prometheus AlertManager.

Prometheus features

Multi-dimensional data model, a time sequence of a plurality of tags and a key metric for determining;
Flexible query language, time series data collected restructuring;
Powerful data visualization capabilities, in addition to built-in browser, and also supports integration with Grafana;
Efficient storage memory plus the local disk, and Union slice can be extended by a performance function;
Operation and maintenance simple, depends only on the local disk, Go binary packages that no other packages depend on libraries;
Accurate alarm;
Lot of Client Libraries;
Export offers many common system to collect indicators;
When sequence data push can be performed by a gateway intermediate;
Static configuration or through service discovery to find the target clients.

Key concept

Data Model

All data stored on the Prometheus are basically 时间序列数据(Time Serie Data, referred to as time-series data). The timing data is a time stamp data stream, the data stream belonging to a certain metrics (the Metric) and a plurality of the measurement tag (Label) under the index.

Metrics (Metric): describes the characteristics of a measure to be monitored. Metrics name of ASCII letters, numbers, underscores and colons composition, must match the regular expression [a-zA-Z_:][a-zA-Z0-9_:]*.
Tag (Tag): a metric with respect to different combinations of values will form a timing tag for a particular dimension. Open label multi-dimensional data model of Prometheus. Prometheus query language and can be filtered through a polymerization time series data metrics and labels. Label name can contain ASCII letters, digits, and underscores, it must match the regular expression [a-zA-Z_][a-zA-Z0-9_]*with _ underscore the label name is reserved for internal use. Tag value can contain any Unicode characters, including Chinese.
Sample value (Sample): time-series data is actually a series of samples. Each sample value comprises:
- A 64-bit floating-point data
- A millisecond accurate time stamp
Annotations (Annotation): a note of a set of metrics and a key-tag pair.

Metrics

Prometheus metrics in the following types:

Counter (Counter): one kind of accumulative metrics, it is a only increment value. Counters are used to summarize similar to server requests, so the number of data errors and the number of completed tasks.
Meter (Gauge): represents an increase either, but also can reduce metrics. Meter for measuring mainly similar to the temperature, memory usage data such as instantaneous.
Histogram (Histogram): observations on sampling (typically response to such a request duration or data size), and a configurable statistics bucket. There are several ways to generate a histogram (assuming that metrics as <basename>):
- Press bucket count equivalent<basename>_bucket{le="<upper inclusive bound>"}
- The sum of the sampling values, corresponding to<basename>_sum
- The total number of sample values, corresponding to <basename>_count, but also all equivalent to the sampled values into a bucket counts<basename>_bucket{le="+Inf"}
Summary (Summary): sampling of observations. In addition to statistical sampling and the total sum of the values, but also according to quantile statistics. There are several ways to generate summary (assuming that metrics as <basename>):
- Quantile is the median sample value is less than the number of points the ratio is less than the total number φ, corresponding to<basename>{quantile="<φ>"}
- The sum of the sampling values, corresponding to<basename>_sum
- The total number of sample values, corresponding to<basename>_count

Tasks and Examples

In Prometheus, the endpoint from which grab sample value is referred to as example, multiple instances of performance for extended and formed out of a copying task.

Tasks (Job): grab your task.
Instance (Instance): grab the source instance.

Prometheus monitor real

Prometheus installation and deployment

# 1. 下载
wget https://github.com/prometheus/prometheus/releases/download/v2.10.0/prometheus-2.10.0.linux-amd64.tar.gz

# 2. 解压
tar zxvf prometheus-2.10.0.linux-amd64.tar.gz

# 3. 启动
cd prometheus-2.10.0.linux-amd64
./prometheus --config.file=prometheus.yml
复制代码

exporter installation and deployment

node_exporter

# 1. 下载
wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.darwin-amd64.tar.gz

# 2. 解压
tar zxvf node_exporter-0.18.1.darwin-amd64.tar.gz

# 3. 启动
cd node_exporter-0.18.1.darwin-amd64
./node_exporter
复制代码

nginx-vts-exporter

I'm using here is openresty, if you are using nginx it is the same

Download dependent packages

cd /usr/local/src/openresty-1.15.8.1/bundle

# openssl
wget https://www.openssl.org/source/openssl-1.1.1c.tar.gz

# ngx_cache_purge
wget http://github.com/FRiCKLE/ngx_cache_purge/archive/2.3.zip -O ngx_cache_purge.zip

# nginx-module-vts
wget http://github.com/vozlt/nginx-module-vts/archive/v0.1.18.zip -O nginx-module-vts.zip

# upstream 健康检查
wget https://github.com/yaoweibin/nginx_upstream_check_module/archive/master.zip -O nginx_upstream_check_module.zip
unzip nginx_upstream_check_module.zip
cd nginx-1.15.8
patch -p1 < ../nginx_upstream_check_module-master/check_1.14.0+.patch
复制代码

compile and install nginx

./configure --prefix=/opt/openresty \
--with-http_auth_request_module \
--with-http_realip_module \
--with-http_v2_module \
--with-debug \
--with-http_stub_status_module \
--with-http_ssl_module \
--with-http_gzip_static_module  \
--with-http_gunzip_module \
--with-http_random_index_module \
--with-threads \
--with-pcre \
--with-luajit \
--with-mail \
--with-file-aio \
--with-http_v2_module \
--with-http_flv_module \
--with-http_mp4_module \
--with-http_dav_module \
--with-http_sub_module \
--with-http_addition_module \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module \
--with-http_secure_link_module \
--with-stream_ssl_preread_module \
--with-openssl=./bundle/openssl-1.1.1c \
--add-module=./bundle/ngx_cache_purge-2.3 \
--add-module=./bundle/nginx-module-vts-0.1.18 \
--add-module=./bundle/nginx_upstream_check_module-master \
-j2  

gmake && gmake install
复制代码

nginx configuration

http {
    vhost_traffic_status_zone;
    vhost_traffic_status_filter_by_host on;   #开启此功能，会根据不同的server_name进行流量的统计，否则默认会把流量全部计算到第一个上。
    ...
    server {
        ...
		location /status {
    		vhost_traffic_status_display;
    		vhost_traffic_status_display_format html;
		}
        # vhost_traffic_status off;
    }
}

复制代码

If you do not want to monitor the domain name, which requires the serverconfiguration modulevhost_traffic_status off;

Installation nginx-vts-exporter

# 1. 下载
wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.0/nginx-vts-exporter-0.10.0.linux-amd64.tar.gz
# 2. 解压
tar zxvf nginx-vts-exporter-0.10.0.linux-amd64.tar.gz
# 3. 启动
cd nginx-vts-exporter-0.10.0.linux-amd64
复制代码

redis_exporter

# 1. 下载
wget https://github.com/oliver006/redis_exporter/releases/download/v1.0.3/redis_exporter-v1.0.3.linux-amd64.tar.gz
# 2. 解压
tar zxvf redis_exporter-v1.0.3.linux-amd64.tar.gz
# 3. 启动
cd redis_exporter-v1.0.3.linux-amd64
./redis_exporter -redis.addr 192.168.102.55:7000 -redis.password test --web.listen-address 0.0.0.0:9121
复制代码

Prometheus Adding Exporter Configuration

After the installation finished exporter, the exporter prometheus need to add a configuration, a simple configuration is as follows:

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']
    
  - job_name: "node"
    static_configs:
    - targets: 
      - '192.168.26.18:9100'
      - '192.168.102.51:9100'
      - '192.168.102.58:9100'
      - '192.168.102.59:9100'
      #labels:
      #  instance: "192.168.26.18:9100"
      #  env: "pro"
      #  name: "192.168.26.18"
    
  - job_name: 'nginx'
    static_configs:
    - targets: 
      - '192.168.102.51:9913'
      - '192.168.102.58:9913'
      - '192.168.102.59:9913'
    
  - job_name: 'redis-exporter'
    file_sd_configs:
    - files: ['./redis.json']

复制代码

redis.json configuration file as follows:

[{
        "targets": [
            "192.168.102.53:9121",
            "192.168.102.53:9122",
            "192.168.102.54:9121",
            "192.168.102.54:9122",
            "192.168.102.55:9121",
            "192.168.102.55:9122",
            "192.168.102.70:9121",
            "192.168.102.70:9122",
            "192.168.102.71:9121",
            "192.168.102.71:9122",
            "192.168.102.72:9121",
            "192.168.102.72:9122"
        ],
        "labels": {
            "service": "redis"
        }
    }
]
复制代码

Prometheus can restart. The final step is to configure Grafana visualized.

Grafana installation configuration

Grafana is an open source metric analysis and visualization tool for cross-platform, it can query the data collected and visual display, and timely notification.

Renderings:

Nginx monitoring

Redis monitoring

Node monitoring

-- END -- Welcome to long press drawing public attention No. DigNew

Recommended reading:

Use Prometheus + Grafana quickly create high force grid monitoring platform