Prometheus Grafana quickly build

Prometheus

Prometheus and combinations Grafana substantially standard monitoring system. Prometheus do storage backend, Grafana do analysis and visualization interface.

Prometheus is an open source system monitoring / alarm tool library, which is very wide, and has an active developer and user community . Prometheus via HTTP periodically active pull (the Pull) obtained manner indicators (direct access gateway or by pushing), all grab samples stored locally, operating rules and these data, the existing data from the polymerization and recording new time series or generate an alert.
Prometheus native visual interface done relatively primitive (mainly for debugging), the community (the official recommended) using Grafana do the data show.
Grafana focus on data display, it is rich with a mature display mode and plug-in data source support Elasticsearch, Prometheus, Graphite, InfluxDB and so on. It lets you click through the interface (no need to write front-end code) to quickly build a very nice professional presentation interface. Even for zero-based front-end developers are very friendly!

Installation Prometheus

In the official website version download required ( uname -rvsee linux kernel version and release number). For example, on x86 download linux-386 series.
Prometheus will take the initiative to collect objective metrics monitored by an HTTP request. For example, it is also exported their health status data via HTTP Rest API, so it can be used to monitor themselves. Extracting a source file containing the download basic prometheus.ymlconfiguration can be referred to. Configuration is very simple.
```
global: # 全局配置
  scrape_interval:     15s #主动拉取指标的间隔
  evaluation_interval: 15s #计算间隔

scrape_configs: #监控的目标配置
  - job_name: prometheus #名称 
    static_configs: # 静态配置
      - targets: ['127.0.0.1:9090'] #监控目标暴露的HTTP API端口，是个列表
```
- The inside of the localhost into your machine corresponding to the IP.
- Other detailed configuration visible configuration documentation .

Reception starts Prometheus, if in a production environment, you need to start the background, it is best to configure their own Systemd .

# Start Prometheus.
# By default, Prometheus stores its database in ./data (flag --storage.tsdb.path).
./prometheus --config.file=prometheus.yml

Open the browser http: // IP: 9090 / metrics query list of all metrics.
Open your browser to http: // IP: 9090 / graph, native simple display interface (too simple, basically no one will use).
PS: Because Promethues own export index and display interface 9090 are the same port. But in practice the interface metrics are indicators pointing to a list of target machines for Promethues initiative to pull.

Rich Exporter can be downloaded to use out of the box. The following can be used NodeExporter to be a model.

Installation NodeExporter

NodeExporter exposure and a lot of hardware / software-related indicators (metrics).

Direct download corresponding versions .
Decompression NodeExporter.

$ tar xvfz node_exporter-*
$ cd node_exporter-*

Start NodeExporter.

$ ./node_exporter
INFO[0000] Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)  source="node_exporter.go:156"
INFO[0000] Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:43)  source="node_exporter.go:157"
INFO[0000] Enabled collectors:   source="node_exporter.go:97"
INFO[0000]  - arp                source="node_exporter.go:104"
...
INFO[0000] Listening on :9100    source="node_exporter.go:170"

It can be used ./node_exporter -hto view specific startup parameters. You can see the port it uses from above is 9100, a list of all the indicators and the above example can prometheus the same interface:

$ curl http://localhost:9100/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.8138e-05
go_gc_duration_seconds{quantile="0.25"} 4.1588e-05
go_gc_duration_seconds{quantile="0.5"} 0.000102923
go_gc_duration_seconds{quantile="0.75"} 0.000162106
go_gc_duration_seconds{quantile="1"} 0.000495923
go_gc_duration_seconds_sum 0.060153937
go_gc_duration_seconds_count 537
# HELP go_goroutines Number of goroutines that currently exist.
...

You can see a list of all the indicators of node_exporter. The next step is to make prometheus to take these indicators.

Pull node configuration prometheus exporter of indicators. Namely the 9100 targets to increase port.
```
scrape_configs:
- job_name: 'node'
  static_configs:
  - targets: ['127.0.0.1:9100']
```
Restart prometheus.
```
./prometheus --config.file=./prometheus.yml
```
Check your browser again open graph view: http: //127.0.0.1: 9090 / graph . Check Enable query historydirectly to the input node, you can see a large number of indicators on the node prefix. For example: node_filesystem_avail_bytesCheck the file system available space conditions.

The interface is still too raw, but can be used to experience PromQL . Then demonstrate what access grafana to display the data.

Installation Grafana

According to official guidelines to download and install: such as Centos is installed

$ wget https://dl.grafana.com/oss/release/grafana-6.3.3-1.x86_64.rpm 
$ sudo yum localinstall grafana-6.3.3-1.x86_64.rpm

Configuration grafana

You can /etc/grafana/grafana.iniin a port and others, all of the specific configuration items: https://grafana.com/docs/installation/configuration/, we are here to keep the default value, the default port is 3000. username / password by default ADMIN .
You can find the corresponding start-up mode Here , for example, is on CentOS

sudo service grafana-server start

After a successful start, you can use the browser to open http: // IP: 3000 using admin / admin login.

Creation interface

Prometheus data source (data source)

Click the icon in the sidebar Grafana -> DataSources -> Add New
Select the type of Prometheus.
Set Prometheus external URL (such as http: // IP: 9090).
Click Add to add

Prometheus chart show

Click the graph title -> Edits.
You just select the step increase of Prometheus database under Metrics tab.
Enter Prometheus expression in the Query field will auto-complete.
Name of the custom chart indicators abscissa: Legend format.

Import Dashboards

There are a Grafana.com lot of optimization of dashboards to share with others , we can find the corresponding node exporter dashboard directly from above to use. Download corresponding json file , then import.

other

How to choose the right charts show Prometheus corresponding data types in Grafana (monotonically increasing Counter, can be reduced liter Gauge, Histogram histogram for display), providing Summary sliding window summing .