Prometheus has several configuration files, each of which may have multiple specific files.
These configuration files may include:
- Configuration files such as prometheus.yml
- Alarm rule file such as rule.yml
- Monitoring target files for service discovery such as targets.json
- node-exporter 的 textfile
Some of these files may be manually modified and then manually triggered to reload, some may be monitored and reloaded by the sidecar program, such as thanos-sidecar, and some may be modified by a custom program, such as the go program written by yourself that regularly updates the target file.
Taking the monitoring target file as an example, Prometheus will read this file in three scenarios:
- First boot loading
- Trigger loading when the file changes
- Load regularly
The first two scenarios are not problematic, and the third scenario may cause problems. Prometheus may read the file while you are updating the file, which may cause an empty file, part of the file, or a reading error. For example, in my targets.json file 7M+, the writing time takes several milliseconds.
I thought about locking and writing, but this may cause unnecessary troubles, such as read blocking. The official correct update posture is to write a temporary file first, and then rename the file to be the file to be modified. Note that it must be on the same file system, preferably in the same directory, otherwise it may not be an atomic renaming operation. At the same time, it should be noted that only file systems that comply with the standard POSIX specification can guarantee that this operation is atomic, which is one of the main reasons why NFS is not officially supported by Prometheus.