Article directory

Huawei Cloud Yaoyun Server L instance evaluation | Trial monitoring service Prometheus on Huawei Cloud

Huawei Cloud Yaoyun Server L instance evaluation | Trial monitoring service Prometheus on Huawei Cloud

1. Introduction to the three-piece monitoring service Prometheus

A common combination of Prometheus monitoring systems is Prometheus + Alertmanager + Grafana, which is the so-called Prometheus three-piece set.

Prometheus: Responsible for the collection, storage, query, alarm triggering and other functions of time series data, and is the core of the monitoring system.
Alertmanager: manages alarm rules, removes duplication, grouping, routing and other processing of alarms issued by Prometheus, and sends alarm notifications, supporting multiple notification channels.
Grafana: A data visualization tool that can query Prometheus data sources, display monitoring indicator data through rich charts, and build dashboards. Grafana provides powerful visualization capabilities.

The combination of Prometheus + Alertmanager + Grafana is very common in monitoring systems.

The division of labor is clear, and each component focuses on doing its own thing. Prometheus focuses on data collection and query, Alertmanager focuses on alert sending, and Grafana focuses on visualization.
High integration between components. Prometheus, Alertmanager and Grafana are connected through the HTTP interface, seamlessly integrated, and easy to expand.

The Prometheus three-piece suite is a de facto standard, with good component collaboration and full coverage of functions. This is also an important reason why it has become the first choice for monitoring technology.

2. Huawei cloud host preparation

Purchase Huawei cloud host. The evaluation system for this evaluation is as follows:
2. Create a new security group and develop all ports to facilitate testing
and change the security group. As follows, select our security group to develop all ports:
After developing all the ports, we can log in to the Huawei Cloud host via ssh~

3. Prometheus installation

Official download: https://prometheus.io/download/

wget https://github.com/prometheus/prometheus/releases/download/v2.37.9/prometheus-2.37.9.linux-amd64.tar.gz

Execute the following command to decompress:

tar xvfz prometheus-*.tar.gz
cd prometheus-*

Before starting Prometheus, let's configure it.

Configure Prometheus to monitor yourself

Prometheus monitors the target node (hereinafter referred to as the "sampling target") by collecting metrics (telemetry special words, metric indicators) on the HTTP port of the target node. Because Prometheus also exposes its own data in the same way, it can also collect and check its own health.

While it's not useful in production practice for a Prometheus server to just collect its own data, this is a good starting example. Save the following basic configuration to the file prometheus.yml:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']

For a complete description of configuration options, please view the configuration documentation ( https://prometheus.io/docs/prometheus/latest/configuration/configuration/ )

Start Prometheus

# Start Prometheus.
# By default, Prometheus stores its database in ./data (flag --storage.tsdb.path).
./prometheus --config.file=prometheus.yml

Prometheus should be started at this point. You should also be able to open the status page via your browser at localhost:9090 .

You can also verify that Prometheus is serving metrics about itself by navigating to its metrics endpoint: localhost:9090/metrics

Here is another common parameter for you -storage.tsdb.pathto specify the storage location
. Example demo:

/elkeid/prometheus/prometheus-2.31.1.linux-amd64/prometheus --config.file /elkeid/prometheus/prometheus.yml --storage.tsdb.path /elkeid/prometheus/

4. Grafana installation

Refer to the official website instructions for installation.
Grafana official website offline

Grafana provides powerful monitoring data visualization capabilities.

wget https://dl.grafana.com/enterprise/release/grafana-enterprise-10.1.1.linux-amd64.tar.gz
tar -zxvf grafana-enterprise-10.1.1.linux-amd64.tar.gz

Note: You need to change the password when logging in for the first time
http://127.0.0.1:3000/login

By default, Grafana will listen on http://localhost:3000**. The default login is "admin" / "admin"

5. Alertmanager installation

Official download address: https://prometheus.io/download/

Alertmanager manages alerts generated by Prometheus and supports multiple alert notification methods such as voice, email, and WeChat.

Alertmanager is a very important component in the Prometheus monitoring system. It is responsible for processing alarm information when abnormal monitoring data occurs, and supports rich alarm strategies and alarm notification methods. The configuration of Alertmanager can be very flexible to meet the alarm needs of different scenarios.

You can use wget to download the installation package from github

wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz

Alertmanager can run as a separate process or be deployed with Prometheus. Alertmanager supports containerized deployment and can be deployed using tools such as Docker. Deploying Alertmanager requires placing the configuration file in a specific path and starting the Alertmanager process.

Alertmanager listens to port 9093 by default. You need to ensure that this port is open in the firewall. The command to start the Alertmanager process is as follows:

./alertmanager --config.file=alertmanager.yml

6. Start and stop management of three services

1. Prometheus, Alertmanager and Grafana startup sequence

There are certain startup sequence dependencies between the three services of Prometheus, Alertmanager and Grafana:

Prometheus needs to be started first, because both Alertmanager and Grafana rely on the data sources provided by Prometheus.
Alertmanager needs to be started after Prometheus is started, because it needs to receive and process alerts sent by Prometheus in real time.
It is best to start Grafana after Prometheus and Alertmanager are started, because it needs to use the indicator data provided by Prometheus and the alert status provided by Alertmanager to display monitoring information.

The recommended startup sequence is:

Prometheus
Alert manager
Grafana

2. Use supervisord to manage Prometheus ecological components

Using supervisord to manage Prometheus ecological components is a common solution.

Using supervisord to manage Prometheus, Alertmanager and Grafana is a good choice and is also a solution adopted by many people. supervisord is a process control system that can easily manage and monitor processes. The main benefits of using it to manage these three components are:

1. Can be managed uniformly and control the startup and stop of components through supervisord.

2. Supports automatic process restart. The KEEPALIVE mechanism allows the process to automatically restart after exiting abnormally.

3. You can view the running status, logs and other information of the management process through the web UI interface.

4. Simple configuration, supervisord uses INI format configuration files to facilitate setting the running parameters of the process.

5. Supports log management, and can output process logs to supervisord for unified viewing.

6. Low resource usage, supervisord itself is lightweight and has little impact on the system. Therefore, using supervisord to manage Prometheus ecological components is a stable and efficient solution, which is also adopted by many users. It can greatly simplify the starting and stopping, monitoring and log management of components. Of course, you can also consider using other process management tools such as systemd.

3. Supervisord configuration example demo:

The three services Prometheus, Alertmanager and Grafana

This sequence can be configured in Supervisor in the following ways:

Put Prometheus at the front of the configuration file
Set a higher priority value for Prometheus
to ensure that Prometheus starts first, Alertmanager starts after Prometheus is available, and Grafana starts after Alertmanager is ready.

[program:prometheus]
command=/opt/lighthouse/server/env/prometheus/prometheus --config.file=/opt/lighthouse/server/env/prometheus/prometheus.yml
autorestart=true
autostart=true
user=www_lighthouse
priority=10
stdout_logfile=/opt/lighthouse/server/logs/supervisor/prometheus.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10

[program:alertmanager]
command=/opt/lighthouse/server/env/alertmanager/alertmanager --config.file=/opt/lighthouse/server/env/alertmanager/alertmanager.yml
autorestart=true
autostart=true
user=www_lighthouse
priority=5
stdout_logfile=/opt/lighthouse/server/logs/supervisor/alertmanager.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10

[program:grafana]
command=/opt/lighthouse/server/env/grafana/bin/grafana-server -config=/opt/lighthouse/server/env/grafana/conf/grafana.ini
autorestart=true
autostart=true
user=www_lighthouse
priority=1
# environment=HOME="/usr/share/grafana", USER="grafana"
stdout_logfile=/opt/lighthouse/server/logs/supervisor/grafana.log 
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10

Huawei Cloud Yaoyun Server L instance evaluation | Installation of monitoring service Prometheus on Huawei Cloud three-piece installation