4. Prometheus: Use Exporter to monitor Windows and Linux and configure encryption authentication

In the architecture design of Prometheus , Prometheus Server does not directly serve and monitor specific targets, but its main task is responsible for data collection, storage and external data query support. So in order to be able to monitor something, such as the CPU usage of the host, you need to use Exporter . Prometheus periodically pulls monitoring sample data from the HTTP service address (usually /metrics ) exposed by Exporter .

In order to collect the monitoring sample data of the host, a Node Exporter program is installed on the host , which exposes an HTTP access address for obtaining the current monitoring sample data. Such a program is called Exporter , and an instance of Exporter is called a Target . Prometheus regularly obtains monitoring data samples from these Targets through polling and stores them in the database.

Exporter can be a program that runs independently from the monitoring target, or it can be directly built into the monitoring target. As long as the monitoring sample data in a standard format can be provided to Prometheus .

You can also implement a custom Exporter program, as long as it meets the standard. Broadly speaking, all programs that can provide monitoring sample data to Prometheus can be called an Exporter . An instance of Exporter is called target , as shown below, Prometheus periodically obtains sample data from these targets by polling :

Exporter source

Exporter source: provided by the community and user-defined

node_exporter monitors Linux, mysqld_exporter monitors MySQL, windows_exporter monitors windows, and other exporters can access EXPORTERS AND INTEGRATIONS: Exporters and integrations | Prometheus

• provided by the community

The Prometheus community provides a wealth of Exporter implementations, covering monitoring functions in all aspects of infrastructure, middleware, and network. These Exporters can fulfill most common monitoring requirements. The following table lists some commonly used Exporters in the community :

scope	Common Exporter
database	MySQL Exporter, Redis Exporter, MongoDB Exporter, MSSQL Exporter等
hardware	Apcupsd Exporter，IoT Edison Exporter， IPMI Exporter, Node Exporter等
message queue	Beanstalkd Exporter, Kafka Exporter, NSQ Exporter, RabbitMQ Exporter等
storage	Ceph Exporter, Gluster Exporter, HDFS Exporter, ScaleIO Exporter等
HTTP service	Apache Exporter, HAProxy Exporter, Nginx Exporter等
API service	AWS ECS Exporter， Docker Cloud Exporter, Docker Hub Exporter, GitHub Exporter等
log	Fluentd Exporter, Grok Exporter等
surveillance system	Collectd Exporter, Graphite Exporter, InfluxDB Exporter, Nagios Exporter, SNMP Exporter等
other	Blockbox Exporter, JIRA Exporter, Jenkins Exporter ，Confluence Exporter等

• Custom

In addition to directly using the Exporter program provided by the community , you can also create your own Exporter program based on the Client Library provided by Prometheus . Currently, the Promthues community officially provides support for the following programming languages: Go , Java/Scala , Python , Ruby . At the same time, there are third-party implementations such as: Bash , C++ , Common Lisp , Erlang, Haskeel , Lua , Node.js , PHP , Rust , etc.

How the Exporter works

In terms of the way the Exporter operates, it can be divided into:

• independent use

Take the Node Exporter that has been used as an example, because the operating system itself does not directly support Prometheus , and users cannot provide support for Prometheus directly from the operating system level. Therefore, the user can only convert the operating status data of the system into monitoring data that can be read by Prometheus through the relevant interface provided by the operating system by running a program independently . In addition to Node Exporter , such as MySQL Exporter , Redis Exporter, etc. are all implemented in this way. These Exporter programs act as an intermediary agent.

• integrated into the application

In order to better monitor the internal operating status of the system, some open source projects such as Kubernetes and ETCD directly use the Prometheus Client Library in the code , providing direct support for Prometheus . This method breaks the boundaries of monitoring and allows applications to directly expose the internal running status to Prometheus , which is suitable for some projects that require more custom monitoring indicators.

Windows Exporter

Windows Exporter is maintained by Prometheus Community, the official address is: https://github.com/prometheus-community/windows_exporter. is a collector that collects Windows machine metrics. It supports Windows Server 2008R2 and above or Windows 7 and above. Windows Exporter provides files in two formats when it is released, namely *.exe and *.msi.

Each edition of Windows Exporter provides a .msi installer. The installer sets up windows_exporter as a Windows service and creates an exception in Windows Firewall. You can choose any format to install when you use it.

download

Download address: Releases prometheus-community/windows_exporter GitHub

or

https://download.csdn.net/download/zhouruifu2015/87579035

Install

view service

 

Access Windows Exporter Monitoring Metrics

The default port of windows_exporter is 9182, and you can access it by entering: localhost:9182 in the browser.

This list is the indicators that Windows Exporter supports to collect. Some indicators are opened by default after startup, and some need to be opened manually. For details, please refer to: https://github.com/prometheus-community/windows_exporter

Use a configuration file to start Exporter. Windows Exporter supports using the --config.file parameter to specify a YAML format file as a configuration file to start, for example: .\windows_exporter.exe --config.file=config.yml . This configuration file supports Configure the content to be collected, logs, etc.

scrape_configs placement windows exporter

Restart prometheus to view the configured exporter

after reboot

Node Exporter

Download address https://github.com/prometheus/node_exporter/releases

or

https://download.csdn.net/download/zhouruifu2015/87579035

In order to be able to collect the operating indicators of the host such as CPU, memory, disk and other information. You can use Node Exporter . Node Exporter is also written in Golang , and does not have any third-party dependencies. It only needs to be downloaded and decompressed to run. Binary packages of the latest node exporter version can be obtained from Download | Prometheus .

Unzip to directory: /usr/local/bin/

service configuration script

cat >> /etc/rc.d/init.d/node_exporter <<EOF
#!/bin/bash
#
# /etc/rc.d/init.d/node_exporter
#
#  Prometheus node exporter
#
#  description: Prometheus node exporter
#  processname: node_exporter

# Source function library.
. /etc/rc.d/init.d/functions

PROGNAME=node_exporter
PROG=/opt/prometheus/$PROGNAME
USER=root
LOGFILE=/var/log/prometheus.log
LOCKFILE=/var/run/$PROGNAME.pid

start() {
    echo -n "Starting $PROGNAME: "
    cd /opt/prometheus/
    daemon --user $USER --pidfile="$LOCKFILE" "$PROG &>$LOGFILE &"
    echo $(pidofproc $PROGNAME) >$LOCKFILE
    echo
}

stop() {
    echo -n "Shutting down $PROGNAME: "
    killproc $PROGNAME
    rm -f $LOCKFILE
    echo
}


case "$1" in
    start)
    start
    ;;
    stop)
    stop
    ;;
    status)
    status $PROGNAME
    ;;
    restart)
    stop
    start
    ;;
    reload)
    echo "Sending SIGHUP to $PROGNAME"
    kill -SIGHUP $(pidofproc $PROGNAME)#!/bin/bash
    ;;
    *)
        echo "Usage: service node_exporter {start|stop|status|reload|restart}"
        exit 1
    ;;
esac
EOF

Add executable permissions: chmod +x node_exporter

run node exporter

or

service node_exporter start

Start by binding ports, multiple node_exporters can run on one machine

./node_exporter --web.listen-address 127.0.0.1:9100

./node_exporter --web.listen-address 127.0.0.1:9200

./node_exporter --web.listen-address 127.0.0.1:10000

After the startup is successful, check the port

netstat -anplt|grep 9100

Visit Node Exporter to see the following page

Status of multiple node_exporters

If the page cannot be accessed, check whether the host firewall is closed

Initial Node Exporter monitoring metrics

 

Visit http://localhost:9100/metrics , you can see all the monitoring data of the current host obtained by the current node exporter , as follows:

Before each monitoring indicator, there will be a piece of information similar to the following:

# HELP node_cpu Seconds the cpus spent in each mode.
# TYPE node_cpu counter
node_cpu{cpu="cpu0",mode="idle"} 362812.7890625
# HELP node_load1 1m load average.
# TYPE node_load1 gauge
node_load1 3.0703125

Among them, HELP is used to explain the meaning of the current indicator, and TYPE indicates the data type of the current indicator. In the above example, the comment of node_cpu indicates that the current indicator is the total CPU time occupied by the idle process on cpu0 . The CPU usage time is a metric that only increases but not decreases. It can also be seen from the type that the data type of node_cpu is a counter (counter ) , consistent with the actual meaning of the indicator. Another example is node_load1 , which reflects the load of the current host within the last minute. The system load will change with the use of system resources. Therefore, node_load1 reflects the current state, and the data may increase or decrease. From the comments, you can see It can be seen that the current indicator type is gauge , which is consistent with the actual meaning reflected by the indicator.

Some monitoring indicators:

• node_boot_time : system boot time
• node_cpu : System CPU usage
• node disk * : disk IO
• node filesystem * : file system usage
• node_load1 : system load
• node memeory * : memory usage
• node network * : network bandwidth
• node_time : current system time
• go_* : go related indicators in node exporter
• process_* : node exporter's own process-related running indicators

scrape_configs placement node exporter

In order to enable Prometheus Server to obtain monitoring data from the current node exporter , the Prometheus configuration file needs to be modified here. Edit prometheus.yml and add the following under the scrape_configs node :

Access Prometheus web service after binding

 

Among them, "1" means normal, otherwise "0" means abnormal.

Prometheus authenticates Node_Exporter TLS encryption

In Promethues' monitoring system, there has always been a view in the community that Metrics do not contain too private information. So you can see that most of the /metrics interfaces are directly exposed, and there are no special security measures. But as Prometheus is widely used in production, security issues become more important.

Enable TLS for connections between Prometheus and monitoring targets. However, since various exporters do not natively support TLS connections, they usually choose to cooperate with a reverse proxy to complete. This method can meet the needs, but it is a bit more complicated. Recently, Prometheus has modified its security model. From Node Exporter to other subsequent components, it will support TLS and basic auth, and also lists the latest security benchmarks (by default, TLS v1.2 and above are all supported)

• Modify the prometheus.yml configuration file

- job_name: 'node_exporter'
    basic_auth:
      username: admin
      password: ******************
    scheme: https
    tls_config:
      ca_file: node_exporter.crt
      insecure_skip_verify: true  # 跳过不安全认证
	file_sd_configs:
- files
- ‘targets.json’
    static_configs:
    - targets: ['localhost:9090']