textfile custom metrics collector
node_exporter In addition to collecting system indicators by itself, we can also textfile collect our custom monitoring indicators through modules, which provides more flexible usage space for system monitoring. For example, the monitoring data we collect through scripts can be exposed through this module. Used in Prometheus for monitoring and alarming. node_exporter The component is enabled by default textfile , but you need to --collector.textfile.directory set a path for collection with parameters. All generated monitoring indicators will be placed in this directory and .prom end with a file name suffix.
All custom generated monitoring metrics need to be stored as follows, for example, we use shell or python scripts to write files:
# HELP example_metric Metric read from /some/path/textfile/example.prom
# TYPE example_metric untyped
example_metric 1
This is actually a standard metrics interface content format. If no HELP information is added here, the system will help generate a simple description information, but if the same indicator name appears in multiple files, it is necessary to ensure that the HELP sum of these indicators is TYPE To be consistent, otherwise the acquisition will be wrong.
.prom Generally speaking , the script task of outputting indicators to a file will be put into crontab it for execution, and the time to collect indicators is set according to the requirements. However, if node_exporter the file is being written during the collection, it may cause problems with the file. We can put the task Move to a temporary file first, and then reduce the risk by renaming the temporary file, as follows:
*/5 * * * * $TEXTFILE/printMetrics.sh > /path/to/directory/metrics.prom.$$ && mv /path/to/directory/metrics.prom.$$ /path/to/directory/metrics.prom
For .prom file collection, the system will automatically add the modification time of the collected files. Through this indicator, we can set an alarm to determine whether the file has changed. For example, if the collection indicator time is every 10 minutes, the modification time should be <15 minutes. It should alarm that the last collection was unsuccessful, the indicator name is node_textfile_mtime_seconds, and the indicator collection time is the unixtime format time.
At the same time, in addition to loading some probe information, this method can also be used to collect static information, such as defined system role information, or special server configuration information, etc., which can also be transmitted through metrics.
echo 'role{role="application_server"} 1' > /path/to/directory/role.prom.$$
mv /path/to/directory/role.prom.$$ /path/to/directory/role.prom
Here we take a shell script provided by the official script for collecting the size of the folder directory as an example. The script address is: https://github.com/prometheus-community/node-exporter-textfile-collector-scripts/blob /master/directory-size.sh with the following contents:
#!/bin/sh
#
# Expose directory usage metrics, passed as an argument.
#
# Usage: add this to crontab:
#
# */5 * * * * prometheus directory-size.sh /var/lib/prometheus | sponge /var/lib/node_exporter/directory_size.prom
#
# sed pattern taken from https://www.robustperception.io/monitoring-directory-sizes-with-the-textfile-collector/
#
# Author: Antoine Beaupré <[email protected]>
echo "# HELP node_directory_size_bytes Disk space used by some directories"
echo "# TYPE node_directory_size_bytes gauge"
du --block-size=1 --summarize "[email protected]" \
| sed -ne 's/\\/\\\\/;s/"/\\"/g;s/^\([0-9]\+\)\t\(.*\)$/node_directory_size_bytes{directory="\2"} \1/p'
First, you need to specify the collector directory in the node_exporter startup program. The directory we specify here is :textfile/root/p8strain/textfile
Then reboot node_exporter:
☸ ➜ systemctl daemon-reload
☸ ➜ systemctl restart node_exporter
This node_exporter will start to collect textfile the custom indicator data in the directory we specified. In order to use the above test script, we can put the generated file into a temporary file, and then re-command. Another way is to use the sponge command to ensure that the content is written in an atomic manner <collector_script> | sponge <output_file>. We are here on a CentOS system, and this tool needs to be additionally installed. :
☸ ➜ yum -y install epel-release
☸ ➜ yum -y install moreutils
Save the above example script as directory-size.sh, put it /root/p8strain under the directory, and then crontab add the following command to count the size of the directory:
☸ ➜ crontab -e
# 加入如下所示定时任务
☸ ➜ crontab -l
*/5 * * * * /root/p8strain/directory-size.sh /root/p8strain | sponge /root/p8strain/textfile/directory_size.prom
Normally /root/p8strain/textfile , the indicator file specified above will be generated under the directory directory_size.prom , and the content is as follows:
# HELP node_directory_size_bytes Disk space used by some directories
# TYPE node_directory_size_bytes gauge
node_directory_size_bytes{directory="/root/p8strain"} 459378688
Normally, the indicator has been collected by Prometheus, and we can directly query the indicator to obtain relevant information:
