Prometheus is an open source monitoring, querying and alerting tool. Originally built by Soundcloud in 2012, the feature-rich tool has been adopted by several companies to monitor their IT infrastructure and ensure all systems are running smoothly. Prometheus allows you to query and extract time-series metrics such as CPU and memory utilization over the HTTP protocol and visualize them on real-time charts. You can also configure Prometheus to push alerts when a node or service is down, and integrate it with other third-party monitoring tools such as Grafana for enhanced data visualization. In this guide, we will look at Prometheus installation on CentOS 8 / RHEL 8 systems.
Step 1) Create Prometheus user and group
First, we will create a system user for Prometheus. Execute the following command to achieve this.
[root@prometheus ~]# useradd -m -s /bin/false prometheus
[root@prometheus ~]# id prometheus
uid=1002(prometheus) gid=1002(prometheus) groups=1002(prometheus)
[root@prometheus ~]#
As you may have noticed, the system user does not have login privileges as specified in the / bin/false option
Step 2) Create configuration directory for Prometheus
After creating a user for Prometheus, we will create configuration directories inside the /etc and /var directories which will store Prometheus configuration files and data. So run the following command:
[root@prometheus ~]# mkdir /etc/prometheus
[root@prometheus ~]# mkdir /var/lib/prometheus
Set ownership on /var/lib/prometheus
[root@prometheus ~]# chown prometheus /var/lib/prometheus/
Step 3) Download the Prometheus tar file
With the directories in place, we can now download Prometheus. To get the latest version, go to the download page to get the latest version for your environment. At the time of writing, the latest version is v 2.14.0 . Alternatively, simply run the following command
[root@prometheus ~]# dnf install wget -y
[root@prometheus ~]# wget https://github.com/prometheus/prometheus/releases/download/v2.14.0/prometheus-2.14.0.linux-amd64.tar.gz -P /tmp
After the download is complete, unzip the compressed package file as shown below
[root@prometheus tmp]# tar -zxpvf prometheus-2.14.0.linux-amd64.tar.gz
This will leave you with a directory called prometheus-2.14.0.linux-amd64.
Use the tree command to view the directory structure,
The extracted directory contains 2 binaries prometheus & promtool which we need to copy to / usr/local/bin path.
So, navigate to the extracted directory and copy them with:
[root@prometheus ~]# cd /tmp/prometheus-2.14.0.linux-amd64
[root@prometheus prometheus-2.14.0.linux-amd64]# cp prometheus /usr/local/bin
Do something similar to another binary
[root@prometheus prometheus-2.14.0.linux-amd64]# cp promtool /usr/local/bin
Step 4) Create configuration file for Prometheus
To start with configuration, create a file /etc/prometheus/prometheus.ym l and paste the configuration into the file
[root@prometheus ~]# vi /etc/prometheus/prometheus.yml
# Global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
scrape_timeout: 15s # scrape_timeout is set to the global default (10s).
# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
This will only monitor your local system (Prometheus server).
Next, adjust the firewall as follows to allow external connections to the server on port 9090
[root@prometheus ~]# firewall-cmd --add-port=9090/tcp --permanent
success
[root@prometheus ~]# firewall-cmd --reload
success
[root@prometheus ~]#
Step 5) Create Systemd service file for Prometheus Server
In order to manage Prometheus as a service using systemd, we need to create a system file for it. So create the file as shown and paste the content,
[root@prometheus ~]# vi /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Time Series Collection and Processing Server
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
For the changes to take effect, reload systemctl,
[root@prometheus ~]# systemctl daemon-reload
Now start and enable Prometheus to run on boot
[root@prometheus ~]# systemctl start prometheus
[root@prometheus ~]# systemctl enable prometheus
To make sure Prometheus is running, run the following command:
[root@prometheus ~]# systemctl status prometheus
From the displayed output, we can clearly see that Prometheus is running as expected with no errors. Also, you can use the netstat utility to check if the service is listening on port 9090.
[root@prometheus ~]# netstat -tunlp
awesome! Prometheus is running on port 9090 as expected. Now go to your browser to browse the IP of the server as shown
http://server-ip:9090
Click the Status tab, then click Target
Your system will display as follows
Step 6) Install and configure node_exporter
Node Exporter is a utility for collecting and delivering numerous Linux system metrics such as CPU, memory usage, file system and network statistics. In this section, we will install node_exporter on a Prometheus server and a remote CentOS 8 Linux host and monitor system metrics on both hosts.
On the Prometheus node, we will create a system user for node_exporter.
[root@prometheus ~]# useradd -m -s /bin/false node_exporter
Next, head over to the Prometheus download page and download the node_exporter tarball or download it from the command line using the following wget command,
[root@prometheus ~]# wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
After downloading the node_exporter file, go ahead and extract it as shown
[root@prometheus ~]# tar -zxpvf node_exporter-0.18.1.linux-amd64.tar.gz
You can check the content of the extracted folder using the tree command as follows
[root@prometheus ~]# tree node_exporter-0.18.1.linux-amd64
Next, copy the binary named node_exporter to the /usr/local/bin path
[root@prometheus ~]# cp node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin
Next, set the file permissions of the copied node_exporter file as shown
[root@prometheus ~]# chown node_exporter:node_exporter /usr/local/bin/node_exporter
Next, we need to configure node_exporter to run as a service. So go ahead and create a systemd service file as follows
[root@prometheus ~]# vi /etc/systemd/system/node_exporter.service
Then paste the configuration shown below and save the file
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
For the changes to take effect, reload the systemd manager with the following command:
[root@prometheus ~]# systemctl daemon-reload
Next, start and enable the node_exporter service
[root@prometheus ~]# systemctl start node_exporter
[root@prometheus ~]# systemctl enable node_exporter
To make sure the service is running, execute:
[root@prometheus ~]# systemctl status node_exporter
To make sure the service is running, use the netstat utility to check that it is listening on port 9100 by default.
[root@prometheus ~]# netstat -pnltu | grab 9100
tcp6 0 0 :::9100 :::* LISTEN 3472/node_exporter
[root@prometheus ~]#
Complete! The Node_exporter service works as expected.
Next, open port 9100 in the firewall as shown below
[root@prometheus ~]# firewall-cmd --add-port=9100/tcp --permanent
success
[root@prometheus ~]# firewall-cmd --reload
success
[root@prometheus ~]#
Also repeat the steps outlined for the remote CentOS 8 Linux system .
Finally, you need to add the node_exporter target to your prometheus.yml file. Append the following line to define node_exporter for prometheus server
[root@prometheus ~]# vi /etc/prometheus/prometheus.yml
---------
- job_name: 'node_exporter'
static_configs:
- targets: ['localhost:9100']
Restart the prometheus service
[root@prometheus ~]# systemctl restart prometheus
Go to your browser again, click on the " Status " tab, then click on " Targets "
Be sure to watch a new endpoint called node_exporter for the Prometheus server on your browser
To add an endpoint for a remote Linux system , go back to the prometheus.yml file and append the following line
- target: ['192.168.10.90:9100']
The node_exporter section should now look like this
- job_name: 'node_exporter'
static_configs:
- targets: ['localhost:9100']
- targets: ['192.168.10.90:9100']
Save the changes and restart the Prometheus service
[root@prometheus ~]# systemctl restart prometheus
Refresh your browser and notice the second endpoint added for the remote CentOS Linux system
Be sure you are receiving metrics from configured nodes. Just use the curl command as follows:
# curl http://node-ip:9100/metrics
For example, to display metrics from a Prometheus server run:
[root@prometheus ~]# curl http://localhost:9100/metrics
For a remote CentOS 8 host, I executed the following commands:
[root@prometheus ~]# curl http://192.168.10.90:9100/metrics
This can also be done by opening a browser and browsing the URL
http://192.168.10.90:9100/metrics
You can also choose to graph the metrics you want. Just go to the home page of your Prometheus server and click on the dropdown labeled " Insert Metric at Cursor ".
Select the metric to graph,
Click on the ‘Execute’ button and click on the ‘graph’ tab just below to reveal the graph
And that brings us to the end of this topic. You have successfully installed and configured Prometheus to monitor system metrics on your server and remote hosts. In the next guide, we'll integrate Prometheus with Grafana for better visualization and analysis of metrics. Please feel free to share your feedback with us and share the article with your friends.