foreword
This article mainly introduces how to use Grafana, Prometheus and node_exporter to monitor the performance of Linux servers. The following two pictures show the two servers:
server A
server B
Overview
Prometheus is an open source service monitoring system that collects data from remote machines through HTTP protocol and stores it in a local time series database.
- Multidimensional data model (time series data consists of metric name and a set of key/value)
- Flexible query language (PromQl) in multiple dimensions
- Does not rely on distributed storage, single master node works.
- Collect time series data through HTTP-based pull
- Time series data push (pushing) can be done through push gateway
- The target server to be collected can be obtained through service discovery or static configuration
- A variety of visual charts and dashboards support
Prometheus collects monitoring data through the exporter installed on the remote machine, and we will use node_exporter to collect system data later.
Architecture
Grafana is an out-of-the-box visualization tool with a full-featured metric dashboard and graphical editor, flexible and rich graphical options, a mix of styles, and support for multiple data source features.
Architecture
Install
Exporter
Download and unzip:
#下载
wget https://github.com/prometheus/node_exporter/releases/download/v0.14.0/node_exporter-0.15.0.linux-amd64.tar.gz -O node_exporter-0.15.0.linux-amd64.tar.gz
# 可自定义解压目录
tar -xvf node_exporter-0.15.0.linux-amd64.tar.gz
Run node_exporter:
# 后台运行
./node_exporter &
Prometheus
Download address: https://prometheus.io/download
Execute the following commands:
## 下载
wget https://github.com/prometheus/prometheus/releases/download/v2.0.0-rc.3/prometheus-2.0.0-rc.3.linux-amd64.tar.gz
## 可自定义解压目录
tar -xvf prometheus-2.0.0-rc.3.linux-amd64.tar.gz
配置 prometheus ， you prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
labels:
instance: prometheus
- job_name: linux1
static_configs:
- targets: ['192.168.1.120:9100']
labels:
instance: sys1
- job_name: linux2
static_configs:
- targets: ['192.168.1.130:9100']
labels:
instance: sys2
The IP corresponds to the server on our intranet, and the port corresponds to the listening port of the corresponding exporter.
Run Prometheus
./prometheus
level=info ts=2017-11-07T02:39:50.220187934Z caller=main.go:215 msg="Starting Prometheus" version="(version=2.0.0-rc.2, branch=HEAD, revision=ce63a5a8557bb33e2030a7756c58fd773736b592)"
level=info ts=2017-11-07T02:39:50.22025258Z caller=main.go:216 build_context="(go=go1.9.1, user=root@a6d2e4a7b8da, date=20171025-18:42:54)"
level=info ts=2017-11-07T02:39:50.220270139Z caller=main.go:217 host_details="(Linux 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 iZ2ze74fkxrls31tr2ia2fZ (none))"
level=info ts=2017-11-07T02:39:50.223171565Z caller=web.go:380 component=web msg="Start listening for connections" address=0.0.0.0:9090
......
After the startup is successful, we can access it through the built-in web interface of Prometheus, http://ip:9090 . If the following interface appears, the configuration is successful
Grafana
Execute the following installation commands:
## 安装依赖grafana运行需要go环境
yum install go -y
## 安装 grafana
yum install https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-4.6.1-1.x86_64.rpm -y
Installation package information:
二进制文件： /usr/sbin/grafana-server
init.d 脚本： /etc/init.d/grafana-server
环境变量文件： /etc/sysconfig/grafana-server
配置文件： /etc/grafana/grafana.ini
启动项： grafana-server.service
日志文件：/var/log/grafana/grafana.log
默认配置的sqlite3数据库：/var/lib/grafana/grafana.db
你可以执行以下启动命令：
service grafana-server start
启动grafana，并设置开机启动：
systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server
systemctl enable grafana-server.service
服务器端图像（PNG）渲染是可选的功能，但在共享可视化时非常有用，例如在警报通知中。
如果图像缺少文本，请确保已安装字体包。
yum install fontconfig
yum install freetype*
yum install urw-fonts
访问Grafana通过Nginx代理，默认登录用户名密码：admin/admin，需及时修改。
server {
listen 80;
server_name grafana.52itstyle.com;
charset utf-8;
location / {
default_type text/html;
proxy_pass http://127.0.0.1:3000;
}
}
编辑配置文件/etc/grafana/grafana.ini ，修改dashboards.json段落下两个参数的值：
[dashboards.json]
enabled = true
path = /var/lib/grafana/dashboards
安装仪表盘JSON模版:
git clone https://github.com/percona/grafana-dashboards.git
cp -r grafana-dashboards/dashboards /var/lib/grafana/
最后，通过service grafana-server start命令启动服务，访问地址：http://grafana.52itstyle.com
然后在Data Sources选项中添加数据源：
添加成功以后，我们就可以查看到文章开头的效果图了。
总结
讲道理，这一套东西还是很强大的，各种开源组间一整合完美搭建出一套监控系统。当然了以上仅仅是系统的一个监控，Grafana以及exporter组间还可以实现对Nginx、MySql、Redis以及MongDB的监控。
监控不是目的，目的是出现问题能够及时发现并解决问题。
Dingding alarm function of Grafana system monitoring
References
https://github.com/prometheus/node_exporter
https://github.com/percona/grafana-dashboards
https://www.percona.com/blog/2016/02/29/graphing-mysql-performance-with-prometheus-and-grafana/
Author: Xiaoqi
Source: https://blog.52itstyle.com
Sharing is a joy, and it also witnesses the personal growth process. Most of the articles are summaries of work experience and daily learning accumulation. Based on the lack of self-knowledge, it is inevitable. Please correct me and make progress together.