Prometheus+Grafana visual monitoring [ElasticSearch status]

Article directory

1. Install Docker

Note: I use the script I wrote before to install Docker. If you already have Docker, please omit this step. Docker is installed to facilitate the deployment of the ElasticSearch service. If you already have a database, you can omit the first two steps.

Click to get the Docker offline installation script

tar zxf docker20.10.14Install.tar.gz
cd docker20.10.14Install
bash install.sh

Check the Docker status as shown below, which means there is no problem:

systemctl status docker

Insert image description here

2. Install ElasticSearch (Docker container method)

1. Prerequisite preparation:

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.17.6
mkdir /home/software/elasticsearch/{
    
    data,plugins,config,logs} -p
chmod -R 777 /home/software/elasticsearch
echo "vm.max_map_count=262144" >> /etc/sysctl.conf

systemctl stop firewalld
systemctl disable firewalld
iptables -F

2. Create configuration file

cat > /home/software/elasticsearch/config/elasticsearch.yml << EOF
cluster.name: "docker-cluster"
network.host: 0.0.0.0
EOF

3. Run the container:

docker run -itd --name elasticsearch  -p 9200:9200 -p 9300:9300 \
    -v /home/software/elasticsearch/data:/usr/share/elasticsearch/data \
    -v /home/software/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
    -v /home/software/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
    -v /home/software/elasticsearch/logs:/usr/share/elasticsearch/logs \
    -v /etc/localtime:/etc/localtime  -e ES_JAVA_OPTS="-Xms512m -Xmx512m" \
    -v /etc/sysctl.conf:/etc/sysctl.conf  -e "node.name=es1" \
    -e "discovery.seed_hosts=es1" -e "cluster.initial_master_nodes=es1" \
    -e "http.host=0.0.0.0" --privileged --restart=always \
    docker.elastic.co/elasticsearch/elasticsearch:7.17.6

Settings vm.max_map_countto prevent startup failure

docker exec -it elasticsearch sysctl -w vm.max_map_count=262144

4. Verify whether the ElasticSearch service is normal.
This step requires waiting patiently for ES to start before requesting it. It takes about 1 minute, depending on the server performance.

curl  http://127.0.0.1:9200

As shown below: ElasticSearch status information is returned, indicating that it is correct.
Insert image description here

3. Install Prometheus

1. Time and time zone synchronization

timedatectl set-timezone Asia/Shanghai
yum -y install ntpdate
/usr/sbin/ntpdate -u ntp1.aliyun.com

Configure scheduled task synchronization time

echo "0 5 * * * /usr/sbin/ntpdate -u ntp1.aliyun.com >/dev/null &" >> /var/spool/cron/root
crontab -l

2. Install Prometheus

wget https://github.com/prometheus/prometheus/releases/download/v2.42.0/prometheus-2.42.0.linux-amd64.tar.gz

tar zxf prometheus-2.42.0.linux-amd64.tar.gz 
mv prometheus-2.42.0.linux-amd64 /usr/local/prometheus

3. Configure systemd management

cat > /usr/lib/systemd/system/prometheus.service << EOF
[Unit]

[Service]
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
ExecReload=/bin/kill -HUP \$MAINPID

[Install]
WantedBy=multi-user.target
Alias=dbus-org.fedoraproject.FirewallD1.service
EOF

4. Start and set up auto-start at boot

systemctl enable prometheus --now
systemctl status prometheus

Insert image description here
The display is as shown above, which means that there is no problem with Prometheus. The default port is 9090. We can access it with a browser.
Click Status> Targetsto view Prometheus' own metrics as shown below:

4. Install Grafana

1. Install Grafana

wget https://dl.grafana.com/enterprise/release/grafana-enterprise-9.4.1-1.x86_64.rpm
sudo yum install grafana-enterprise-9.4.1-1.x86_64.rpm -y

systemctl enable grafana-server.service --now

2. WEB page verification.
The default port is 3000. When you visit for the first time, you will be prompted to reset the password, as shown below:
Insert image description here

5. Pronetheus and Grafana are related

Insert image description here

6. Install elasticsearch_exporter

1. Install elasticsearch_exporter

wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.3.0/elasticsearch_exporter-1.3.0.linux-amd64.tar.gz
tar zxf elasticsearch_exporter-1.3.0.linux-amd64.tar.gz
mv elasticsearch_exporter-1.3.0.linux-amd64 /usr/local/elasticsearch_exporter

2. Configure systemd management
Note: The --es.uri parameter sets your own es access address. This is the local machine. What I wrote ishttp://127.0.0.1:9200

cat > /usr/lib/systemd/system/elasticsearch_exporter.service << EOF
[Unit]
Description=elasticsearch_exporter Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/elasticsearch_exporter/elasticsearch_exporter  --es.all --es.indices --es.cluster_settings --es.indices_settings --es.shards --es.snapshots --es.uri http://127.0.0.1:9200
ExecReload=/bin/kill -HUP \$MAINPID
Restart=on-failure
RestartSec=30s

[Install]
WantedBy=multi-user.target
EOF

Start and join auto-start at boot

systemctl enable elasticsearch_exporter --now
systemctl status elasticsearch_exporter

Insert image description here
The default port is 9114

netstat -anput |grep 9114

4. Specify elasticsearch_exporter information in the Prometheus configuration file

vim /usr/local/prometheus/prometheus.yml

  - job_name: "ElasticSearch_115"
    static_configs:
      - targets: ["16.32.15.115:9114"]

After adding the configuration file, use the command to test whether there are any problems with the format.

cd  /usr/local/prometheus/
./promtool check config prometheus.yml

Restart Prometheus

systemctl restart prometheus.service

7. Grafana adds ElasticSearch monitoring template

The template ID of ElasticSearch is used here: 7259
click 下方红圈地方>>import输入模板ID
Insert image description here

Insert image description here

The final rendering is as follows:

Insert image description here