How Prometheus Prometheus monitoring study notes the heat load configuration updates

0x00 Overview

When Prometheus have to modify the configuration file, hot update method provided by Prometheus we can use to achieve the realization reload the configuration file in the case of non-stop services.

0x01 hot update

Hot update is loaded in two ways:

#1.  kill -HUP pid
#2.  curl -X POST http://192.168.1.102:9090/-/reload

When you take any more than a way to perform reload successful, you will see the following information in promtheus log in:

 If because of the configuration information is not properly filled out the update fails, you will see similar information:

ERRO[0161] Error reloading config: couldn't load configuration (-config.file=prometheus.yml): unknown fields in scrape_config: job_nae source=main.go:146

prompt:

1. Personally, I prefer using curl -X POST way, because after each reload, pid change, need to find a way to use kill the current process.
2. Since 2.0, hot reload feature is off by default, To turn, required when booting Prometheus, add the  --web.enable-lifecycle parameter.

0x02 heat source code comparison update

Here we come to explore the internal way to achieve both of these principles.

The first: HUP (hang up) parameters to achieve the kill command:

First Prometheus in  cmd/promethteus/main.go implementing the system call to monitor the process, if you receive syscall.SIGHUP signal, will perform reloadConfig function.

Code like this:

hup := make(chan os.Signal) signal.Notify(hup, syscall.SIGHUP) go func() { for { select { case <-hup: if err := reloadConfig(cfg.configFile, reloadables...); err != nil { log.Errorf("Error reloading config: %s", err) } } } }()

The second: web module through  /-/reload fulfill the request:

1. Prometheus is first registered in the web (web / web.go) a module of a POST http request  /-/reload, it is handler  web.reload function, which is mainly to  web.reloadCh chan inside a transmission  error.
2. In the Prometheus  cmd/promethteus/main.go has a separate goroutine in to listen  web.reloadChwhen a new value is received it will execute reloadConfig function.

Code like this:

hupReady := make(chan bool)

go func() { <-hupReady for { select { case rc := <-webHandler.Reload(): if err := reloadConfig(cfg.configFile, reloadables...); err != nil { log.Errorf("Error reloading config: %s", err) rc <- err } else Rc { < - nil } } } } ( )

Prometheus provides a sophisticated internal hot reload program, which greatly facilitates modify the configuration file and reload in Prometheus ecology, many Exporter also be implemented in a similar manner agreed.