Prometheus + Grafana monitoring system deployment and use

About Prometheus

Prometheus is a combination open & alarm monitoring time-series & databases, by starting SoundCloud developed. With the development, more and more companies and organizations to accept the use of Prometheus, the community is also very active, they put it into a separate open source projects, and companies have to operate. Google SRE also mentioned in the book with them BorgMon monitoring system similar implementation is Prometheus. Now the most common Kubernetes container management system, usually with Prometheus monitor.

1) Prometheus features

Powerful data model, Prometheus monitoring data are collected in the form of the metric stored in the built-in timing manageable database TSDB them, because the core of the Prometheus of only a single binary file, so there is no third party reliance powerful query language , Prometheus built a powerful data query language PromQL, which can be achieved for monitoring data query and aggregation high availability and scalability, Prometheus not only data can be stored locally, you can also use remote storage, but also can use third-party tool plus its federal cluster to achieve high availability `` visualization platform, Prometheus comes with a UI, but more primitive, but Grafana natural support for Prometheus, the perfect combination, you can create a more attractive data monitor icon;

2) Prometheus data collection

pull mode : This mode is a form of pull, first need to install the Client exporter, exporters daemon running in a mode, and begin collecting data exporters of each program as a separate http-server, since it may be made to the http request data response and return metric (K / V data) Prometheus-server to access the exporter on each node and collected in the form of pull required, for the default mode;

push mode : This mode is in the form of a push for the use of push mode, you need to install the plug-pushgateway pushgateway on the server or the client we can be understood as a proxy, which is located pushgateway between the client and the server receives the first indicator and then pushed to the pushgateway prometheus-server, pushgateway here do not have to install the client, can also be installed on the server or even other nodes;

Specifications Type 3) Prometheus's

Index English called metrics, a set of metrics is K / V data, metrics can be understood as a generic term for data collection over rather than a particular value or indicator, since the type of indicators are diverse, but in actual Prometheus -server which does not distinguish between types of indicators `` but simply put these indicators are regarded as the type of non-time series, at this point, we look at indicators of what type of metrics;

1.Counter (counter) : from 0 cumulative basis, the data represents a sample of monotonically increasing index, which is only to rise, unless the monitoring system reset occurs we can use this type of indicators to represent the server the number of requests, the total number of the total number of errors occurred, has been completed at the time of use Counter type, do not apply it to non-monotonically increasing sample data indicators, such as the number of processes currently running system, this time should use other types;

2.Gauge (dashboard) : The data it collects is instantaneous, and the time does not matter, this type of data represents a sample of indicators which can be changed, that is, it will not necessarily continue to grow, but will not necessarily continue to decrease it does not change the rules, we can use this type of metrics to collect hard disk capacity usage and memory usage, but can also represent the total increase or decrease at any time, such as the number of concurrent requests;

3.Histogram (histogram) : is used to represent the histogram data acquisition over time results and statistics can be specified interval and the total number, which is usually used to calculate the histogram quantiles, for example, the duration of the request, or size of the response;

4.Summary (Abstract) : Histogram type and it is similar to the results for data acquisition is a period of time, but it is directly stored quantile not be calculated by the interval

Prometheus advantage

In addition there is also the advantage Prometheus:
1. Efficient : single Prometheus can handle millions of monitoring index; data processing hundreds of thousands of points per second.

2. Easy stretch : Federal clusters (federation) of Prometheus can be extended through the use of functional areas (sharing) +, to form a logical cluster; Prometheus offers multilingual client SDK, the SDK can be quickly integrated into the application to the Prometheus monitoring them.

3. Good visualization : Prometheus addition comes with Prometheus UI, Prometheus also offers an independent solution based Dashboard Ruby On Rails program Promdash. In addition the latest Grafana visualization tool also provides a complete Proetheus support, API-based Prometheus provided can also implement their own monitoring visualization UI.

4. Monitoring more diversity **: prometheus client that supports multiple languages, by our clients convenient Buried core business, such as orders, add shopping cart and other processes.
And prometheus has a lot of system integration;

For example: the application level monitoring: nginx, haproxy, mysql, docker and so on;
the system level, such as: SNMP protocol monitoring, host monitoring, process monitoring;
container Cloud Monitoring: docker, k8s openstack private cloud;

The very efficient memory : a sampled data accounted for an average of ~ 3.5 bytes, 320 million for time series sampled every 30 seconds, for 60 days, the consumption of the disk about 228G.

Prometheus architecture

This figure illustrates the architecture of ecosystem Prometheus and some of its components:

Control of each part of FIG introduced:

Server 1.prometheus : for collecting and storing time-series data.
2.Retrieval : sampling module, pulling Prometheus server where the data is retrieved pulled TSDb of distributed data store;
TSDb of: a storage module to store the default local TSDb of;
the HTTP Server: provides http and Interface Query Panel, the default port 9090;
Discovery 3.Service :
service discovery, prometheus support multiple service discovery mechanisms: file, DNS, k8s, openstack, etc., based on service discovery process, through third-party interfaces, prometheus query to the list of target need to be monitored, then these poll target acquisition monitoring data;
4. Client SDK
client libraries official there go, java, python, ruby;
Jobs-lived 5.short :
there is not enough time for a short or deleted volume business, not by pulling the pull mode, you need to use push way, used in conjunction with pushgeteway;
6.proDash
use dashboard rails developed for visualizing index data;
7.Exporters : for exposing existing third-party service metrics to Prometheus.
8.Alertmanager : Prometheus server receives from Alerts, removal will duplicate data packet, and route the received acceptance embodiment, alarm. Common reception methods are: e-mail, pagerduty, OpsGenie, webhook and so on.
Gateway 9.PUSH : mainly used for short-term jobs. Because of the presence of such jobs a short time, possibly before Prometheus to pull it disappeared. To this end, the jobs can push their metrics directly to Prometheus server side.
10.prometheus_cli : command-line tool;

Prometheus workflow

prometheus server configured periodically from jobs or exporters pull metrics. metrics or accept from pushgateway sent, or pull metrics from the other prometheus server
prometheus server in the local storage to collect metrics, and run already defined arlt.rules, recording new time series or to push an alarm alertmanager
Alertmanager based on the profile, the acceptance of an alarm processing, an alarm
In the graphical interface, visualization, data collection, you can use someone else written grafana template.

Prometheus installation and deployment

Download: https: //github.com/prometheus/prometheus/releases/

[root@ chenc01 ~]# tar xf prometheus-2.16.0.linux-amd64.tar.gz 
[root@ chenc01 ~]# mv prometheus-2.16.0.linux-amd64 /usr/local/prometheus
[root@ chenc01 ~]# cd /usr/local/prometheus/
[root@ chenc01 prometheus]# pwd
/usr/local/prometheus
[root@ chenc01 prometheus]# ls
console_libraries  consoles  LICENSE  NOTICE  prometheus  prometheus.yml  promtool  tsdb
[root@ chenc01 prometheus]# ./prometheus --version
prometheus, version 2.16.0 (branch: HEAD, revision: b90be6f32a33c03163d700e1452b54454ddce0ec)
  build user:       root@7ea0ae865f12
  build date:       20200213-23:50:02
  go version:       go1.13.8

# 后台启动普罗米修斯
[root@ chenc01 prometheus]# ./prometheus &
[1] 1920
。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

Prometheus use

1, access Prometheus

Visit http through the background: // ip: 9090

Here Insert Picture Description

2, view the status

Here Insert Picture Description

3, check the configuration

Here Insert Picture Description

4, view the running information

Here Insert Picture Description

Grafana Profile

Grafana is an open source data visualization tool developed by Go language, you can do data monitoring and statistics, with alarm function. At present there are many companies use grafana, such as paypal, ebay, intel and so on.

Grafana also supports a variety of data sources, including Elasticsearch, InfluxDB, MySQL, OpenTSDB, before you deploy and use Grafana, let's look at a few basic terms of Grafana;

1) of the basic terms Grafana

1.DataSource (data source) : In the above, we talk, Grafana supports a variety of data sources, including but not limited to Prometheus, and the data source is popular in terms of providing data objects, such as Zabbix, such as Prometheus

2.Dashboard (dashboard) : dashboard is a page is rendered, when we add a good corresponding data source, this time on the need to visualize data in Grafana, it is possible to organize and manage data visualization charts by Dashboard;

3.Row (OK) : the row is a constituent unit Dashboard, it may comprise a plurality of rows Dashboard, and a line which in turn may exhibit a combination of one or more information such as load state, memory usage, disk usage;

4.Panel (panel) : panel could be interpreted as a way to present information Row, it supports tables (tables), lists (alert list), heat (heatmap) chart, which in Grafana, all panels are in the form of plug-ins be using;

5.Query Editor (Query Editor) : the role of the interrogator is to specify which part of the data acquisition, this is somewhat similar to the SQL statement, but if we add the data source for Prometheus, that is actually used is PromQL;

6.User (user) : users and user login system concepts we usually like, Grafana contains three roles, respectively, admin, editor, viewer, read only editor in which, admin highest authority, it can perform how , editor can only create DashBoard, viewer can only view DashBoard, read only editor you can modify the DashBoard, but not save;

7.Organization (Organization) : In Grafana them, DataSource with Dashboard belong to an organization, each user can have several different organizations, in different tissues which can be given different permissions for different users need to know is, different between organizational data sources and dashboards are different, whenever Grafana create an organization in which the equivalent of opening a new view, this time all the data sources and dashboards and other content to be re going to add and configure

Grafana installation and deployment

1) Download and install software packages Grafana

First landing grafana official website, select the version to download the rpm package, download link is: Grafana official website to download path

`[root@prometheus ~]# wget -c https:``//dl.grafana.com/oss/release/grafana-6.6.2-1.x86_64.rpm``[root@prometheus ~]# yum -y install /root/grafana-6.6.2-1.x86_64.rpm`

2) Familiar Grafana profile

Grafana default configuration file: /etc/grafana/grafana.ini, its configuration items have more than 700 lines, most of the default, let's look at a few common configuration items

[root@ chenc01 ~]# vim /etc/grafana/grafana.ini
[paths]
# 定义数据存储路径，用来存储sqlite3，临时文件
data = /var/lib/grafana
 
# 在data目录中的临时数据要存储多久，默认为24h，也可以
temp_data_lifetime = 24h
 
# 定义日志存储路径
logs = /var/log/grafana

# 定义插件存储目录
plugins = /var/lib/grafana/plugins
 
[server]
# 定义访问的协议，默认为http
protocol = http

# 定义监听的IP地址，默认为所有
http_addr = 10.2.3.11

# 定义监听的端口，默认为3000
http_port = 3000

# 定义在浏览器中访问Grafana的全路径，默认即可，也可以写成http://ip:port
root_url = %(protocol)s://%(domain)s:%(http_port)s/

# 是否记录web请求日志，默认为关闭
router_logging = false

# 前端静态文件的存储路径，默认为/usr/share/grafana/public目录
static_root_path = public

# 选择数据库类型，默认为sqlite3，也可以选择MySQL
[database]
type = sqlite3

# 定义连接地址与端口
host = 127.0.0.1:3306

# 定义数据库名称
name = grafana

# 定义连接用户
user = root

# 定义连接密码
password = 123

# 关闭SSL
ssl_mode = disable

# 这里只是sqlite3需要，定义sqlite3的数据存储路径，默认为/var/lib/grafana目录
path = grafana.db
 
[users]
# 是否允许普通用户登录，默认为允许
allow_sign_up = true

# 是否允许普通用户创建组织，默认为允许
allow_org_create = true

# 如果为true，则自动把新增的用户增加到id为1的组织中，如果为false，则新建用户的时候会新增一个组织 ，默认为true
auto_assign_org = true

# 默认的背景页面，也可以选择light
default_theme = dark

3) the definition of Grafana profile

[root@ chenc01 ~]# cp /etc/grafana/grafana.ini /etc/grafana/grafana.ini.bak
[root@ chenc01 ~]# vim /etc/grafana/grafana.ini
[paths]
data = /var/lib/grafana
temp_data_lifetime = 24h
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = conf/provisioning
 
[server]
protocol = http
http_addr = 10.2.3.11
http_port = 3000
root_url = %(protocol)s://%(domain)s:%(http_port)s/
serve_from_sub_path = false
router_logging = false
static_root_path = public
enable_gzip = true
 
[users]
allow_sign_up = true
allow_org_create = true
auto_assign_org = true
default_theme = dark

4) Start Grafana and set to boot from Kai

[root@ chenc01 ~]# systemctl enable grafana-server
Created symlink from /etc/systemd/system/multi-user.target.wants/grafana-server.service to /usr/lib/systemd/system/grafana-server.service.
[root@ chenc01 ~]# systemctl start grafana-server
level=info ts=2020-03-28T06:28:07.583Z caller=compact.go:496 component=tsdb msg="write block" mint=1585366069255 maxt=1585368000000 ulid=01E4FX6YNHTQPYGTEKNJ9TCH2E duration=3.110826386s

level=info ts=2020-03-28T06:28:07.682Z caller=head.go:661 component=tsdb msg="head GC completed" duration=72.63582ms
[root@ chenc01 ~]# systemctl status grafana-server
● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-03-28 14:28:09 CST; 4s ago
     Docs: http://docs.grafana.org
 Main PID: 17041 (grafana-server)
   CGroup: /system.slice/grafana-server.service
           └─17041 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/...

Mar 28 14:28:08 localhost.localdomain grafana-server[17041]: t=2020-03-28T14:28:08+0800 lvl=..."
Mar 28 14:28:08 localhost.localdomain grafana-server[17041]: t=2020-03-28T14:28:08+0800 lvl=..."
Mar 28 14:28:09 localhost.localdomain grafana-server[17041]: t=2020-03-28T14:28:08+0800 lvl=..."
Mar 28 14:28:09 localhost.localdomain grafana-server[17041]: t=2020-03-28T14:28:08+0800 lvl=...s
Mar 28 14:28:09 localhost.localdomain grafana-server[17041]: t=2020-03-28T14:28:09+0800 lvl=...s
Mar 28 14:28:09 localhost.localdomain grafana-server[17041]: t=2020-03-28T14:28:09+0800 lvl=...e
Mar 28 14:28:09 localhost.localdomain grafana-server[17041]: t=2020-03-28T14:28:09+0800 lvl=...p
Mar 28 14:28:09 localhost.localdomain grafana-server[17041]: t=2020-03-28T14:28:09+0800 lvl=..."
Mar 28 14:28:09 localhost.localdomain grafana-server[17041]: t=2020-03-28T14:28:09+0800 lvl=...=
Mar 28 14:28:09 localhost.localdomain systemd[1]: Started Grafana instance.
Hint: Some lines were ellipsized, use -l to show in full.