Prometheus-Grafana monitoring MysqlD && Linux server detailed version

readme

In fact, the previous article has basically written how to operate it (because it was refined from this article). This article may be more about supplementing some theories and relationships between Prometheus and Grafana.

Operation label : It is a must-see and must be operated.
It is not necessary to read the label : right or wrong, it is a variety of theories about software.
PS : The first thing to run below should be node_exporter, then Prometheus, and finally Grafana.
Overall architecture diagram:

As shown above, the monitoring data of MySQL is obtained through mysql_exporter, and the monitoring data of Linux server is obtained through node_exporter. Pass the obtained monitoring data to Prometheus, and finally display it through Grafana , the effect is as follows:

0. Introduction to Prometheus (not necessary)

Multidimensional data model. Flexible query language. Without relying on distributed storage, a single server node is autonomous. Time-series data is collected through HTTP-based pull. Time series data can be pushed through the intermediate gateway. Discover target service objects through service discovery or static configuration. Supports a variety of charts and interface displays, such as Grafana, etc.

Prometheus architecture diagram (not necessary to see)

Description: prometheus grabs data directly or through pushgateway. Store data locally and run rules on it to aggregate and record new time series from existing data, or to generate alerts. Grafana etc. can be used to visualize data.

Prometheus component description (not necessary)

prometheus Server: Complete data collection, service discovery and data storage according to the configuration.

Push Gateway: A plug-in provided to deal with some push scenarios. The monitoring data is first pushed to the Push Gateway, and then collected and pulled by the Prometheus Server. It is used for jobs that exist for a short time and may disappear before Prometheus comes to pull (if the data on the Push Gateway does not change during the Prometheus Server collection interval, Prometheus Server will collect the same data twice, only the timestamp is different)

Exporters (probe): is the general term for a class of data collection components of Prometheus. It is responsible for collecting data from the target and converting it into a format supported by Prometheus. Different from traditional data acquisition components, it does not send data to the central server, but waits for the central server to come to grab it actively.

Alertmanager: Prometheus server is mainly responsible for analyzing data based on PromQL-based alert rules. If the rules defined by PromQL are satisfied, an alert will be generated and the alert information will be sent to Alertmanager. Alertmanager will process and send the alert information according to the configuration.

**Common receiving methods are:** email, webhook, etc. Alertmanager has three ways of processing alarm information: grouping, suppression, and silence.

1. Download and install Prometheus (operation)

First, you need to download and install Prometheus. You can download the latest version of binaries from Prometheus' official website (https://prometheus.io/). After downloading, you need to extract Prometheus to your computer.

About monitoring (not necessary)

Monitoring system (optional)

What to monitor (optional)

Configure Prometheus (Operation)

After unpacking Prometheus, you need to configure Prometheus to monitor your application. In the root directory of Prometheus, you'll find a prometheus.ymlfile called . This file contains configuration information for Prometheus.

You need to edit this file and add the targets you want to monitor. In this file, you need to define one or more job. Each jobrepresents a target you want to monitor. For each job, you need to specify a target, which represents the address and port number of the application you want to monitor.

For example, assuming you want to monitor an myappapplication called , running on a port localhoston 8080, your prometheus.ymlfile should look like this:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'myapp'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:8080']

In this configuration, we define a target myappcalled job, which will monitor localhost:8080this target every 5 seconds.

Set the sampling time and evaluation time through configure scrape_intervaland to control the accuracy and latency of the indicator.evaluation_interval

Example of connecting to my own cloud server (operation)

This port 9100 is the default port running on the cloud server after downloading below.

-----------------------------------

Add remote host (connect to linux cloud server) (monitor cloud server) (operation)

1. Download the node_exporter installation package from the official website. The official website download address is as follows: https://prometheus.io/download/

Pass node_exporter to the MySQL machine to be monitored and decompress it:

sdo utar -zxvf node_exporter-1.5.0.linux-amd64.tar.gz -C /opt/

Create a soft link:

sudo ln -s /opt/node_exporter-1.5.0.linux-amd64/ /opt/node_exporter

start up

sudo nohup /opt/node_exporter/node_exporter & (you can run it after downloading, before running Prometheus)

Running Prometheus (operations)

After completing the configuration, you can start Prometheus. In the root directory of Prometheus, run the following command:

./prometheus (.exe)

This will start Prometheus and start monitoring the targets you specified in the configuration file.

View Monitoring Data (Operation)

Once Prometheus starts to monitor your application, it will collect various indicator data, and these data (after startup, you can enter in the browser, http://localhost:9090/visit the UI interface of Prometheus, where you can see the data that can already be collected index.)

At this time, if an error is reported here: get context deadline exceeded, it means that this port is not open. I just opened it on Alibaba Cloud.

Enter in the browser: http://192.168.150.123:9100/metrics, if there is data similar to the following, it means that the deployment of node_exporter is completed.

-----------------------------------

Add remote host (connect to mysql server) (monitor mysql server) (operation)

Deploy the mysqld_exporter component (operation)

First find the corresponding mysqld_exporter version in https://prometheus.io/download/.

Transfer mysqld_exporter to the MySQL machine to be monitored and decompress it:

sudo tar zxvf mysqld_exporter-0.14.0.linux-amd64.tar.gz -C /opt

create soft link

sudo ln -s /opt/mysqld_exporter-0.14.0.linux-amd64/ /opt/mysqld_exporter

Create a monitoring user on MySQL

create user ‘exporter’@‘localhost’ IDENTIFIED BY ‘112233lml!’;

GRANT SELECT, PROCESS, SUPER, REPLICATION CLIENT, RELOAD ON . TO ‘exporter’@‘localhost’;

Create a new configuration file:

sudo vim /opt/mysqld_exporter/mysqld_exporter.cnf

Configure MySQL to monitor user information:

[client]
user=exporter
password=112233lml!

start mysqld_exporter

sudo nohup /opt/mysqld_exporter/mysqld_exporter --config.my-cnf=/opt/mysqld_exporter/mysqld_exporter.cnf &

Browser input: http://192.168.150.123:9104/metrics, you can get MySQL monitoring data, as shown below (partial data):

Configure Prometheus to obtain monitoring data (operation)

restart prometheus

-----------------------------------

Configure the alarm rules of Prometheus (there are detailed settings at the end of the article)

Prometheus supports indicator-based alarm rules to send alarm notifications when monitoring data reaches a certain threshold. You can configure alert rules through Prometheus prometheus.ymlfiles and send alert notifications through Alertmanager. In the alarm rule, you can set the threshold, alarm level, and alarm notification method, etc.

Separator || Integrate Prometheus and Grafana (not necessary)

Prometheus and Grafana are two commonly used monitoring and analysis tools, and they can work well together to achieve more efficient monitoring and analysis. You can pass the monitoring data to Grafana through the API provided by Prometheus, and use Grafana's visualization tools to display and analyze the data. Through the integration of Prometheus and Grafana, you can better understand the operation of your system and take timely measures to solve problems and risks.

In short, Prometheus is a powerful monitoring and analysis tool that can help you monitor and analyze the performance and operation of the system in real time. With Prometheus, you can gain a better understanding of your systems and take timely action to address potential issues and risks.

The monitoring display diagram of Prometheus is usually displayed based on Grafana. Grafana is an open source data visualization tool that can easily display the monitoring data collected by Prometheus as various forms of graphs.

2. Introduction to Grafana (not necessary)

Grafana is an open source data visualization tool developed in Go language. It can do data monitoring and statistics. You only need to provide the data you need to monitor, and it can help you generate various visual instruments with alarm functions.

Grafana does not make assumptions about the data source, it supports the following types of data, which means that if your data source is any of the following, it can help generate gauges

(no matter what language you use).

Data sources supported by Grafana (not necessary)

• Prometheus The example in this article, it doesn’t matter if you haven’t heard of it, it doesn’t affect the reading, just think of it as MySQL with timestamp
• Graphite
• OpenTSDB
• InfluxDB
• MySQL/PostgreSQL
• Microsoft SQL Server
• etc.

Features (optional)

①Visualization: Fast and flexible client graphics with various options. Dashboard plugins to visualize metrics and logs in many different ways.

②Alarm: Visually define alert rules for the most important indicators. Grafana will continuously evaluate them and send notifications.

③ Notification: When the alarm changes status, it will send a notification. Receive email notifications.

④Dynamic Dashboards: Create dynamic and reusable dashboards using template variables that appear at the top of the dashboard as drop-down menus.

⑤Mix data sources: Mix different data sources in the same graph! Data sources can be specified per query. This even works with custom data sources.

⑥ Notes: Notes from different data source charts. Hovering over an event reveals full event metadata and tags.

⑦Filter: Filter allows you to dynamically create new key/value filters that will be automatically applied to all queries using that data source.

3. Install and use Grafana (operation)

Download and install Grafana (Operation)

First, you need to download the latest version of the binary from Grafana's official website (https://grafana.com/grafana/download). After downloading, you need to extract Grafana to your computer. (I downloaded windows)

Start Grafana (Operation)

After unpacking Grafana, you need to start Grafana. From the root directory of Grafana, run the following command:

./bin/grafana-server web

This will start Grafana running on the default port 3000. http://localhost:3000You can access Grafana's web interface by navigating in your browser .

(

Default port: 3000

Address: http://ip:3000

Initial administrator account: admin, password: admin

)

Configure data sources (operations)

After accessing Grafana's web interface, you need to configure Grafana's data sources. The data source is the source used by Grafana to obtain monitoring data. In Grafana's web interface, select "Configuration" -> "Data Sources" in the left menu bar, and then click the "Add data source" button to add your data source.

Grafana supports a variety of data sources, including Prometheus, InfluxDB, Graphite, etc. Select the data source you want to use and fill in the corresponding configuration information. For example, if you want to use Prometheus as a data source, then you need to fill in the address and port number of Prometheus.

Create a dashboard (for details, please skip to the back - build your first dashboard) (operation)

After configuring the data source, you can start creating your first dashboard. In Grafana's web interface, select "Create" -> "Dashboard" in the left menu bar, and then click the "Add panel" button to add your chart.

When adding a chart, you can choose different display forms, such as line chart, bar chart, pie chart, etc. You can also choose which metrics and timeframe you want to display.

View monitoring data (not necessary)

Once you have created your dashboard, after you have created the dashboard, you can view the monitoring data and analyze it. In Grafana's web interface, you can select "Dashboard" on the left menu bar to view your dashboard.

In the dashboard, you can see the visualization charts and corresponding indicators of the monitoring data you choose. You can adjust and modify the charts as needed to better display the metrics and data you care about.

In addition, Grafana also provides various visualization tools and plug-ins to allow users to customize and expand the display of monitoring data. You can find and download these plugins on Grafana's official website to better meet your needs.

In short, Grafana is a powerful monitoring and analysis tool that can help you monitor and analyze system performance and operation in real time. With Grafana, you can better understand your systems and take timely action to address potential issues and risks.

-----------------------------------

Build your first dashboard (operations)

Frequently Asked Questions (not necessary)

1. How to configure service monitoring? How to ensure data security?
2. What exactly should a healthy service monitor?
3. Why did the configured dashboard disappear?
4. How should query query be written?

Note that your docker should be running in all subsequent steps.

Here we have started three services (not necessary to see)

• Prometheus Prometheus time series database, used to store and query your monitoring data
• Promethues-exporter is a simulated data source used to monitor the status of your machine, such as how many CPUs, CPU load, etc.
• Grafana deity

At this point, your Grafana has been set up. Notice that we have created three services in the Docker configuration file. What is the relationship between these three services?

In other words, what is the relationship between Grafana and time series databases and data sources? Please see below how Grafana works

How Grafana works (not necessary)

As mentioned above, Grafana is a dashboard, and the dashboard must be used to display data.

Grafana itself is not responsible for the data layer, it only provides a common interface so that the underlying database can send data to it. And another service we started, called Prometheus (Chinese name Prometheus database), is responsible for storing and querying data.

That is to say, every time Grafana wants to display a dashboard, it will send a query request to Prometheus.

So what Prometheus-exporteris ?

This is the source of your real monitoring data. Prometheus-exporterThis service will query the information of your local computer, such as how much memory is left, CPU load, etc., and then export the data to the Prometheus database.

In the real world, your purpose is to monitor your own services, such as your web server, your database and so on.

Then you need to send the data to the Prometheus database in your own server. Of course, you can send data to MySQL (Grafana also supports it), but Prometheus is almost a standard time-series database, and it is strongly recommended that you use it.

Use a picture to illustrate the relationship between them:

Here, the leftmost Docker service will send service data to the middle Prometheus (corresponding to the above Prometheus-exporter), and the rightmost Grafana will query the middle Prometheus to display the dashboard.

Build your first dashboard (operations)

Step 1 - Set Data Source (Operation)

After that set the data source URL. Please note that the working principle of Promethues is to obtain data by polling an HTTP request, and Grafana also uses an HTTP request when obtaining the data source, so here you need to tell Grafana what the data endpoint of your Prometheus is.

We http://prometheus:9090can .

// Maybe something is wrong here? (I'm using localhost) (currently no problem)

(Why localhost:9090not docker-compose? The reason is that the three services we used can be imagined as three independent servers, so we need to use a domain name to communicate with each other. The Prometheus we set in docker-compose.yml The name of the Si server is called prometheus, so the former needs to be used here.)

When you click OK, be sure to confirm that Data source is workingthis , which means that your Grafana has talked to Prometheus. (save directly)

Step 2 - Import Dashboard (Operation)

In Grafana, the configuration of the dashboard can be performed through a graphical interface, but the configured dashboard is stored in JSON. That is to say, if you share your JSON data, others can directly import the same dashboard (provided that your monitoring data is the same).

For our example, recall that because we use prometheus-exporterthe local system information monitoring, then we can first find a dashboard that also uses this data source. On the Grafana website, you can actually find a lot of instruments that others have already done, which can be used to monitor many standardized services.

Grafana's Dashboard Marketplace: https://grafana.com/grafana/dashboards

For example, standard dashboards for some of the following services can be found here

• JVM
• Spring Boot
• MySQL monitoring
• Laravel Monitoring

So, here we use a standard dashboard: https://grafana.com/grafana/dashboards/1860

In the plus sign on the left, click Importto import, and fill in the interface that appears 1860(there is also a 11074 below that may be more detailed (there is a picture below)), which is the number of the dashboard we want to import.

Then fill in the information you need, such as the name of the dashboard, etc.

After confirmation, Grafana will generate information such as CPU load, memory and I/O based on your local information. My disk status is shown in the figure (monitored by my own cloud server):

Step 2 - Import Dashboard (Operation)

Change the above import to 11074

----------------------------------- (It is not necessary to look at the starting position)

Step 3 - Generate and create a new dashboard (currently using the default above) (Operation)

Finally, based on the information imported above, you can start creating dashboards related to your service and business.

But before this step, you need to start logging some data in your service.

How to use Prometheus to query data (currently use the above default) (operation)

Prometheus, including all other time series databases, usually defines a query language, such as PromQL.

cpu使用率:
100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) * 100)
 
内存使用率:
100-(node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes)/node_memory_MemTotal_bytes*100
 
磁盘使用率:
100 - (node_filesystem_free_bytes{mountpoint="/data01",fstype=~"ext4|xfs"} / node_filesystem_size_bytes{mountpoint="/data01",fstype=~"ext4|xfs"} * 100)
 
下载带宽:（指定某一个网卡）
irate(node_network_receive_bytes_total{device='eth0'}[5m])
 
上传带宽:（指定某一个网卡）
irate(node_network_transmit_bytes_total{device='eth0'}[5m])
 
服务器负载:
node_load1
node_load5
node_load15

How to manually generate a dashboard (operation)

Assuming that you have generated a basic dashboard according to the above steps, you can now start adding dashboards manually. Also click the plus sign on the left, and Dashboardclick to enter the interface for adding a dashboard.

Here we choose a data called scrape_duration_seconds, don’t care what its meaning is, just treat it as the sales of Double 11:

----------------------------------- (It is not necessary to see the end position)

Configure Mysql data source (operation)

Grafana supports multiple data sources, Mysql, InfluxDB, OpenTSDB, Prometheus, Elasticsearch, CloudWatch and KairosDB, etc.

1 Configure data source (operation)

The same is him, we all use promethus here

2 Grafana displays MySQL monitoring data

Enter 7362 under "Import via grafana.com":

Or download the JSON template on the https://grafana.com/grafana/dashboards/7362 page, then click "Upload JSON file" to import, and the following information will be displayed (currently there are some other templates, such as https://github.com /percona/grafana-dashboards, if you are interested, you can try it):

Select the Prometheus data source created before in the prometheus option, click "Import", and it will automatically jump to the following interface:

So far, Grafana has completed displaying the monitoring data of MySQL in Prometheus.

Alarm configuration

Usually in a formal environment, it is necessary to monitor the usage of server resources, and when the usage rate exceeds a certain amount, the operation and maintenance personnel will be alerted.

At this time, the Alerting function of grafana is used to perform regular inspection/monitoring indicators. And send warning messages according to the rules, and integrate into office software such as WeChat/DingTalk

warning form

1. Add group robots

Create a DingTalk /corporate WeChat group –> add a robot –> after adding a robot, there is a webhook address

2. Configure the alarm channel in grafana

2.1. Add alarm channel

2.2. Configure the alarm mode as the default mode

Specific alarm rule configuration

1. Select the target to be alerted

In the monitoring panel, click the drop-down button of the monitoring indicator –> select Edit

2. Add warning rules

After clicking Edit, the Add Alarm Rule button appears below, click Add

3. Alarm rule configuration

Finally click Save, for example the configured rule is to check every minute. When the CPU exceeds 3% (configured according to specific needs), it will enter the pengding state, and if it exceeds 3% for 2 minutes, an alarm notification will be issued

After adding an alarm, you can view the monitoring status in the list or monitoring panel

Enterprise WeChat correspondingly received an alarm notification

reference article

Set port:
firewall-cmd --list-ports ##List open ports
firewall-cmd --add-port=9090/tcp --permanent #Permanently add 9090 port
firewall-cmd --add-port=9100/tcp --permanent #Permanently add port 9100
firewall-cmd --add-port=9104/tcp --permanent #Permanently add port 9104
firewall-cmd --list-ports #List open ports
systemctl stop firewalld #Close firewall
systemctl start firewalld #Open firewall
systemctl status firewalld #View firewall status
systemctl restart firewalld #Restart firewall

Reference article: https://blog.csdn.net/liyanbin1214/article/details/125373510

Reference article: https://blog.csdn.net/easylife206/article/details/114312284

Reference article: https://blog.csdn.net/ok_shuai/article/details/128950355

Reference article: https://blog.csdn.net/weixin_43997319/article/details/124983451

Reference article: https://blog.csdn.net/qq_38571773