HDFS & Yarn is the monitoring of Hadoop by Jmx on Promethues + Grafana (2)-Promethues data source configuration

Note : If Promethues has not been installed, please refer to HDFS & Yarn on Promethues + Grafana monitoring (1)-Promethues installation

One, configure the data source of Promethues

1. Download the jmx_prometheus_javaagent-0.13.0.jar package:

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.13.0/jmx_prometheus_javaagent-0.13.0.jar

2. Configure the connection between jmx and promethues

(1) Create a folder and create different jmx configuration yaml under this folder (this step must have this configuration file on each node)

a, create a folder

cd /data
mkdir hdfs

b. Create the JMX configuration file of NameNode in the /data/hdfs/ directory (this step must have this configuration file on each node)

vim /data/hdfs/nn.yaml

startDelaySeconds: 0
hostPort: 10.21.0.86:6444
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

c. Create the JMX configuration file of DateNode in the /data/hdfs/ directory (this step must have this configuration file on each node)

vim /data/hdfs/dn.yaml

startDelaySeconds: 0
hostPort: 10.21.0.86:6454
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

d. Create the JMX configuration file of ResourceManager in the /data/hdfs/ directory (this step must have this configuration file on each node)

vim /data/hdfs/rm.yaml

tartDelaySeconds: 0
#master为本机IP（一般可设置为localhost）；6464为想设置的jmx端口
hostPort: 10.21.0.86:6464
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

e. Create the JMX configuration file of NodeManager in the /data/hdfs/ directory (this step must have this configuration file on each node)

vim /data/hdfs/nm.yaml

startDelaySeconds: 0
#master为本机IP（一般可设置为localhost）；6474为想设置的jmx端口（可设置为未被占>用的端口）
hostPort: 10.21.0.86:6474
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

(2) CDH version & open source community version cluster are recommended to do this:

a. Modify the startup command file of hadoop's bin/hdfs, just add the following environment variable at the top

The cdh directory is here: /opt/cloudera/parcels/CDH/lib/hadoop-hdfs/bin/hdfs

##hdfs jmx monitor##
#######################
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=6444 -javaagent:/data/hdfs/jmx_prometheus_javaagent-0.13.0.jar=17106:/data/hdfs/nn.yaml $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=6454 -javaagent:/data/hdfs/jmx_prometheus_javaagent-0.13.0.jar=17107:/data/hdfs/dn.yaml $HADOOP_DATANODE_OPTS"
#######################

b、在/opt/cloudera/parcels/CDH/lib/hadoop-yarn/bin/yarn

##yarn jmx monitor##
#######################
export YARN_RESOURCEMANAGER_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=6464 -javaagent:/data/hdfs/jmx_prometheus_javaagent-0.13.0.jar=17104:/data/hdfs/rm.yaml $YARN_RESOURCEMANAGER_OPTS"
export YARN_NODEMANAGER_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=6474 -javaagent:/data/hdfs/jmx_prometheus_javaagent-0.13.0.jar=17105:/data/hdfs/nm.yaml $YARN_NODEMANAGER_OPTS"
#######################

Note : Of course, you can also refer to the configuration of other big guys to add these configurations in /etc/hadoop/conf/ hadoop-env.sh . I don’t like to change the command. It is recommended to add environment variables.

(3) Configure in promethues (my promethues directory is under /usr/local)

Note : The advantage like a and b below is that you don’t need to restart Promethues to modify the target monitored node, but you need to restart it if you configure it directly in prometheus.yml

a. Create a configs directory under the promethues directory, and then write the target json configuration file

mkdir configs
cd configs

vim nn.json
--------------------------------------------------
[
 {
  "targets": ["nn的host:17106"]
 }
]
--------------------------------------------------

vim dn.json
--------------------------------------------------
[
 {
  "targets": ["dn1的host:17107","dn2的host:17107","dn3的host:17107"]
 }
]
--------------------------------------------------

vim rm.json
--------------------------------------------------
[
 {
  "targets": ["rm1host:17104","rm2host:17104"]
 }
]
--------------------------------------------------

vim nm.json
--------------------------------------------------
[
 {
  "targets": ["nm1host:17105","nm2host:17105","nm3host:17105"]
 }
]
--------------------------------------------------

b. Modify the prometheus.yml file in the promethues directory

vim prometheus.yml  
  
  - job_name: 'yarn-rm'
    file_sd_configs:
    - files:
      - configs/rm.json

  - job_name: 'yarn-nm'
    file_sd_configs:
    - files:
      - configs/nm.json

(4) Restart prometheus

$ systemctl restart prometheus

Second, verify that the data source is successfully configured

1. There are two ways to verify whether the configuration is successful:

(1) According to the target path access test configured just now, if a bunch of data is returned, the configuration is successful

curl "[http://rm1host:17104/metrics](http://localhost:17107/metrics)"

返回：
# HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.13.0",name="jmx_prometheus_javaagent",} 1.0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 266.8
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.611046723427E9
# HELP process_open_fds Number of open file descriptors.

。。。。**等等 **

(2) Or visit Port 9090 of Promethues to check whether the target is normal.

Click Status and then click Target to see if it is green, red is configuration failure

[External link image transfer failed. The origin site may have an anti-leech link mechanism. It is recommended to save the image and upload it directly (img-oKewO9cW-1611311044018)(https://secure-static.wolai.com/static/82j7NqcK7KoNakU8AXuouS/image.png )]

Three, configure Grafana

Refer to my other article HDFS & Yarn on Promethues + Grafana Monitoring (3)-Grafana Dashboard configuration (take Yarn as an example)