Prometheus dynamically configures targets

Prometheus dynamically configures targets

(Jin Qing's column 2018.4)

The simplest configuration is a static target:

scrape_configs:
  - job_name: 'prometheus'

    static_configs:
      - targets: ['localhost:9090', 'localhost:9100']
        labels:
          group: 'prometheus'

After changing this file, a SIGHUP can be sent to trigger a configuration reload.

Prometheus provides a service discovery function, which can discover new targets from various sources such as consul, dns, kubernetes, file, etc.
The easiest of these is to discover the service from a file.

For example /root/prometheus/prometheus.yml is configured as follows:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['127.0.0.1:9090', '127.0.0.1:9100']
        labels:
          group: 'prometheus'
  - job_name: 'test'
    file_sd_configs:
      - files: ['/etc/prometheus/test_sd_config/*.yml']
        refresh_interval: 5s

Start prometheus with docker:

docker run -d --net=host \
  -v /root/prometheus:/etc/prometheus \
  --name prometheus-server \
  prom/prometheus

Create /root/prometheus/test_sd_config/test.yml as follows

- targets: [ "192.168.93.192:8080" ]
  labels:
    group: "my_test_group"

Within 5s, test.yml will be automatically read and new targets will be added.
You can open the Prometheus 9090 port with a browser,
check the Configuration, Targets, Service Discovery in Status, and
you can see the newly added target.