Adan's hand-drawn flow chart: the picture may be a little small, please zoom in to see it!
prometheus core components
prometheus server
Prometheus Server is the core part of Prometheus components, responsible for the acquisition, storage and query of monitoring data. Prometheus Server can manage monitoring targets through static configuration, and can also use Service Discoverv to dynamically manage monitoring targets and obtain data from these monitoring targets. Secondly, Prometheus Server needs to store the collected monitoring data. Prometheus Server itself is a time-series database, which stores the collected monitoring data in the local disk in a time-series manner. Finally, Prometheus Server provides a custom PromQL language to realize query and analysis of data.
1. take
Through the service discovery component, Prometheus Server periodically pulls the monitoring indicator data from the above-mentioned ]ob, Exporter, and Pushgateway in the form of HTTP polling.
2. Storage
The captured monitoring data is cleared and organized according to certain rules (use the relabel configs method provided by service discovery before capturing, and use the metrics relabel configs method in the job after capturing), and the obtained results will be stored in a new time Persistence in sequence. The storage module has undergone several redesigns over the years, and the storage system in version 20 of Prometheus is the third iteration. The storage system can handle the ingestion of millions of samples per second, making it possible to monitor thousands of machines with one Prometheus server. The compression algorithm used can achieve 1.3B per sample on real data. It is recommended to use SSD, but it is not strictly required
that the storage of Prometheus is divided into local storage and remote storage
. Remember, NFS is not supported by any version of Prometheus. Some actual production cases tell us that if Prometheus stores files using NFS, there is a possibility of losing historical data.
Remote Storage: Suitable for storing large amounts of monitoring data. The remote storage supported by Prometheus includes OpenTSDB, InfluxDB.Elasticsearch, Graphite, CrateDB, Kakfa, PostgresQL, TimescaleDB, TiKV, etc. Remote storage needs to be converted with the adapter of the middle layer, mainly involving the remote_write and remote_read interfaces in Prometheus. In actual production, various problems will occur in remote storage, which requires continuous optimization, stress testing, architecture transformation, and even rewriting of modules for uploading data logic.
3. Query
After Prometheus persists the data, the client can query the data through the PromQL statement and use the
Express Browser Ul built into the Prometheus Server. Through this U, the data query and visualization can be realized directly through PromQL.
The federated cluster capability of Prometheus Server enables it to obtain data from other Prometheus Server instances, so in the case of large-scale monitoring, Prometheus Server can be expanded through federated clusters and functional partitions.
Exporters
Use tools to collect data, similar to the concepts of buried points and probes in skywalking.
Exporter exposes the endpoint of monitoring data collection to Prometheus Server through HTTP service, and Prometheus Server can obtain the monitoring data that needs to be collected by accessing the Endpoint endpoint provided by the Exporter. Generally speaking, Exporter can be divided into two categories:
direct
collection : This type of Exporter directly has built-in support for Prometheus monitoring, such as cAdvisor, Kubernetes, EtcdGokit, etc., all of which directly have built-in endpoints for exposing monitoring data to Prometheus.
Indirect acquisition: Indirect acquisition, the original monitoring target does not directly support Prometheus, so we need to write the monitoring acquisition program of the monitoring target through the ClientLibrary provided by Prometheus. For example: MysalExporter, ]MX Exporter, ConsulExporter, etc.
In layman's terms, the well must be drilled before water can be used.
AlxrtManager
Prometheus Server supports the creation of alert rules based on PromQL. If the rules defined by PromQL are met, an alert will be generated and the subsequent processing of the alert will be managed by AlertManager. In AlertManager, we can integrate with built-in notification methods such as email, Slack, etc., or customize the alarm processing method through Webhook. AlertManager is the alarm processing center in the Prometheus system.
Summary: In fact, it is the alarm manager.
PushGateway
Since Prometheus data collection is designed based on the Pull model, it is necessary to allow Prometheus Server to communicate directly with Exporter in the configuration of the network environment. When this network requirement cannot be directly met, PushGateway can be used for transit. You can actively push the monitoring data of the internal network to the Gateway through PushGatewav. And Prometheus Server can use the same Pull method to obtain monitoring data from PushGateway.
Service Discovery
Prometheus provides comprehensive support for monitoring scenarios in cloud and container environments through the service discovery mechanism. There are two ways of service discovery: configuration files and service dynamic perception 1. Configuration files
Prometheus
will periodically read the latest from the file target information.
For service discovery that supports files, it can be derived to be used in combination with automated configuration management tools (Ansible, CronJob, Puppet. SaltStack, etc.) in practical scenarios
.
2. Through service dynamic perception
Prometheus also supports a variety of common service discovery components, such as Kubernetes, DNS, Zookeeper, Azure, EC2 and GCE. For example, Prometheus can use the Kubernetes API to obtain container information changes (such as container creation and deletion) to dynamically update monitoring objects. Through
service discovery, administrators can dynamically discover target instances that need to be monitored without restarting the Prometheus service. information.