Promethus cluster deployment notes: (1) Install and configure Grafana high-availability cluster

The system environment and tengine and mariadb components still use the deployed environment and basic components in the "Jumpserver High Availability Cluster Deployment". Tengine recompiles and changes the prefix to /sas/tengine as a basic component shared by multiple services. Related deployment process can refer to:

Jumpserver high availability cluster deployment: (1) System basic environment configuration
Jumpserver high availability cluster deployment: (2) Tengine proxy service deployment
Jumpserver high availability cluster deployment: (3) MariaDB Galera cluster deployment

Official document: https://grafana.com/docs/grafana/latest/administration/set-up-for-high-availability/

1. Add the Yum source corresponding to Grafana Community Edition

vi etc/yum.repos.d/grafana.repo

[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

2, cheap Grafana

yum install grafana

3. Start Grafana

systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server

# 设置自启动
systemctl enable grafana-server

4. Create a mysql database to save the cluster configuration

mysql -uroot -p

# 创建grafana数据库
create database grafana;

# xxxxxxxx 表示数据库密码，授权相应的IP地址及hostname访问数据库
grant all on grafana.* to 'grafana'@'10.255.200.%' identified by 'xxxxxxxx';
grant all on grafana.* to 'grafana'@'localhost' identified by 'xxxxxxxx';
grant all on grafana.* to 'grafana'@'devops01' identified by 'xxxxxxxx';
grant all on grafana.* to 'grafana'@'devops02' identified by 'xxxxxxxx';
grant all on grafana.* to 'grafana'@'devops03' identified by 'xxxxxxxx';

# 保存权限
flush privileges;

5. Modify Grafana configuration to use mysql database

vi /etc/grafana/grafana.ini

#################################### Database ####################################
[database]
# You can configure the database connection by specifying type, host, name, user and password
# as separate properties or as on string using the url properties.

# Either "mysql", "postgres" or "sqlite3", it's your choice
type = mysql
host = 10.255.200.1:3306
name = grafana
user = grafana
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password = xxxxxxxx

6. Modify the default port to 3001, configure the default port 3000 as the tengine listening port, and disable usage feedback, because an error may be reported during feedback due to networking problems

vi /etc/grafana/grafana.ini

#################################### Server ####################################
[server]
# Protocol (http, https, h2, socket)
;protocol = http

# The ip address to bind to, empty will bind to all interfaces
;http_addr =

# The http port  to use
http_port = 3001

#################################### Analytics ####################################
[analytics]
# Server reporting, sends usage counters to stats.grafana.org every 24 hours.
# No ip addresses are being tracked, only simple counters to track
# running instances, dashboard and error counts. It is very helpful to us.
# Change this option to false to disable reporting.
reporting_enabled = false

7. Restart the Grafana service

systemctl restart grafana-server
systemctl status grafana-server

8. Configure tengine

vi /sas/tengine/conf/conf.d/grafana.conf

upstream grafana {
    server 10.255.200.1:3001;
    server 10.255.200.2:3001;
    server 10.255.200.3:3001;

    session_sticky;
}

server {
    listen 3000 ssl backlog=32768;
    server_name grafana.hbrtv.org;
    ssl_certificate   /sas/tengine/sslkey/devops.crt;
    ssl_certificate_key  /sas/tengine/sslkey/devops.key;
    ssl_session_timeout 5m;
    ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://grafana;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

9. Configure the firewall

firewall-cmd --zone=public --add-port=3000/tcp --permanent
# 如果配置错误，可使用以下格式删相应端口
# firewall-cmd --zone=public --remove-port=3000/tcp --permanent

firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.255.200.1/30" port protocol="tcp" port="3001" accept"
# 如果配置错误，可使用以下格式删相应端口
# firewall-cmd --permanent --remove-rich-rule="rule family="ipv4" source address="10.255.200.1/30" port protocol="tcp" port="3001" accept"

# 重新载入防火墙配置，使配置生效
firewall-cmd --reload

10. Restart the nginx service

systemctl restart tengine
systemctl status tengine