Promenade, a monitoring tool, is based on the process-exporter monitoring service process

Background introduction

Since our commonly used node_exporter does not cover all monitoring items, we often need to monitor the process of a certain program in the production environment to determine whether the business is normal, so Process-exporter came into being.
Process exporter is used to monitor processes in prometheus. Through process_exporter, the running status of the application can be monitored from a macro perspective (such as monitoring the process resources of redis, mysql, etc.).

deploy

1. Download the monitoring plug-in

wget https://github.com/ncabatoff/process-exporter/releases/download/v0.7.5/process-exporter-0.7.5.linux-amd64.tar.gz

2. Unzip and configure

Unzip

cd /usr/local/src
tar -xf process-exporter-0.7.5.linux-amd64.tar.gz
mv process-exporter-0.7.5.linux-amd64 process-exporter
chown -R root:root /usr/local/src/process-exporter

Edit the configuration file (refer to line 80 of the README.md file). The method of
Insert picture description here
process_exporter is to configure the name of the process to be monitored. It will search for the process to obtain the monitoring information it needs. In fact, it is what we often do "ps -efl" | grep xxx" command to view the corresponding process. The configuration file does not exist at the beginning, we need to create it, and the name can be customized:

Note: If a process matches multiple matches, it will only belong to the first matched groupname group.
Among them, there are four name options (official translation https://github.com/ncabatoff/process-exporter):
{ {. Comm}} contains the base name of the original executable file, that is, the second field /proc//stat
{ {.ExeBase}} contains the basename of the executable file
{ {.ExeFull}} contains the fully qualified path of the executable file
{ { .Username}} contains the username of a valid user
{ {.Matches}} map contains all matches generated by cmdline regexps
Additional explanation: The name option actually affects the key: groupname in metrics. Take monitoring zookeeper and kafka as an example

touch process-name.yaml
vim process-name.yaml
process_names:
 - name: "{
    
    {.Matches}}"
    cmdline: 
    - '/u01/isi/application/component/zookeeper-3.4.10/bin/../zookeeper-3.4.10.jar'

 - name: "{
    
    {.Matches}}"
    cmdline: 
    - '/u01/isi/application/component/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1.jar'

3. Start-stop service

Note: startup parameters

-config.path    #process-name.yaml的配置文件路径
-web.listen-address   #blackbox_exporter监听的服务端口，默认为9256

cd /usr/local/src/process-exporter      #进入blackbox_exporter的目录
mkdir logs   							#创建日志目录
nohup ./process-exporter  -config.path="process-name.yaml"  -web.listen-address=":9256">> logs/process-exporter.log 2>&1 &   #启动服务
netstat -tanlp | grep 9256   									 #查看监听端口
ps -ef |grep process-exporter |grep -v grep |awk '{print $2}' | xargs kill -9  #停止服务

4. Add configuration in Prometheus

1) Write fd_config file

touch zk_kafka.json
vim zk_kafka.json
[
        {
    
    
                "labels": {
    
    
                        "desc": "zk node",
                        "group": "zk_process",
                        "host_ip": "192.168.16.113",
                        "hostname": "isi-16-113"
                },
                "targets": [
                        "192.168.16.113:9256"
                ]
},
        {
    
    
                "labels": {
    
    
                        "desc": "kafka node",
                        "group": "kafka_process",
                        "host_ip": "192.168.16.113",
                        "hostname": "isi-16-113"
                },
                "targets": [
                        "192.168.16.113:9256"
                ]
}
]

2) Add configuration in Prometheus.yml

...
...
  - job_name: 'zk_kafka-process'
    scrape_interval: 2m
    scrape_timeout: 120s
    static_configs:
    file_sd_configs:
      - files:
        - /home/monitor/prometheus/conf.d/zk_kafka.json

5. Verify configuration and reload service

cd /usr/local/src/prometheus/
./promtool check config prometheus.yml                  #检查配置文件是否正确
 curl -XPOST http://192.168.16.115:9090/-/reload        #重载服务

Insert picture description here