Prometheus tag management (6)

The previous five chapters have explained related knowledge points around prometheus . This chapter mainly supplements another knowledge point of prometheus.

Prometheus label management

In the prometheus monitoring system, the label label is an extremely important parameter. To use labels reasonably, you need to use standard labels to manage and control the entire service, especially in complex environments.

1. Some common label operation cases

1、重命名标签名
2、删除标签
3、过滤目标

Special attention is paid to the above operations, there are only two stages we can operate on the label:

The first stage is to re-mark the target from service discovery.
The metadata label information for service discovery refers to the label on the metric, which will be done in the relabel_configs module.

The second stage is after scrape, but before saving to the storage system.
This allows us to make sure that we have saved those metrics, deleted those metrics, and what these metrics will look like, which will be done in the metric_relabel_configs module

In fact: use relabel_configs before scrape, and use metric_relabel_configs after scape.

2, action: re-label action

replace：默认，通过regex匹配source_label的值，使用replacement来引用表达式匹配的分组
keep：删除regex与连接不匹配的目标 source_labels
drop：删除regex与连接匹配的目标 source_labels
labeldrop：删除regex匹配的标签
labelkeep：删除regex不匹配的标签
hashmod：设置target_label为modulus连接的哈希值source_labels
labelmap：匹配regex所有标签名称。然后复制匹配标签的值进行分组，replacement分组引用（${1},${2},…）替代

3. The format of action

relable_configs:
  # 源标签
  [ source_labels: '[' <labelname> [, ...] ']' ]
  
  # 多个源标签时连接的分隔符
  [ separator: <string> | default = ; ]
  
  # 重新标记的标签
  [ target_label: <labelname> ]
  
  # 整则表达式匹配源标签的值
  [ regex: <regex> | default = (.*) ]
  
  # 用的少,占时略
  [ modulus: <uint64> ]
  
  # 替换正则表达式匹配的分组，分组引用 $1,$2,$3,....
  [ replacement: <string> | default = $1 ]
  
  # 基于正则表达式匹配执行的操作
  [ action: <relabel_action> | default = replace ]

Configuration case

1. Add tags

1) Add tag key-value pairs

cat /home/monitor/prometheus/conf.d/es_cluster.json 
[
	{
    
    
		"labels": {
    
        #labels下的键值对都是标签，可以根据实际需要添加即可
			"desc": "es-21",
			"group": "es_cluster",
			"host_ip": "192.168.16.21",
			"hostname": "wg-16-21",
            "nodes": "es651"   #新增的标签键值对
		},
		"targets": [
			"192.168.16.21:4221"
		]
	}
]

2) Hot start service view

curl -XPOST http://192.168.16.115:9090/-/reload   #重载服务

Interface view
Insert picture description here

2、relabel_configs：action为replace

1) Redefine a task of prometheus.yml

cat  /home/monitor/prometheus/prometheus.yml
...
...
...
  - job_name: 'es_cluster'  
    relabel_configs:     #添加relabel_configs配置，添加在job_name是es_cluster的下面
    - source_labels:  ["job"]   #匹配原labels名称，此处匹配是job
      regex: "(.*)"  #正则匹配所有
      action: replace  #动作是替换，除了替换别的动作
      replacement: $1  #替换第一个值
      target_label: "es651"  #新的label的key值

    scrape_interval: 1m
    static_configs:
    file_sd_configs:
      - files:
        - /home/monitor/prometheus/conf.d/es_cluster.json
    honor_labels: true
...
...
...

2) Hot start service view

cd /home/monitor/prometheus && ./promtool check config prometheus.yml  #校验配置文件
curl -XPOST http://192.168.16.115:9090/-/reload   #重载服务

Interface view
Insert picture description here

3. relabel_configs: action is drop action

1) Delete a task in prometheus.yml

cat  /home/monitor/prometheus/prometheus.yml
...
...
...
  - job_name: 'es_cluster'  
    relabel_configs:     #添加relabel_configs配置，添加在job_name是es_cluster的下面
    - source_labels:  ["job"]   #匹配原labels名称，此处匹配是job
      regex: "(.*)"  #正则匹配所有
      action: replace  #动作是替换，除了替换别的动作
      replacement: $1  #替换第一个值
      target_label: "es651"  #重新定义的job的label值
      
    - source_labels: ["job"]  #匹配原labels名称，此处匹配是job
      action: drop   #将原标签为job实例均删除，即不采集

    scrape_interval: 1m
    static_configs:
    file_sd_configs:
      - files:
        - /home/monitor/prometheus/conf.d/es_cluster.json
    honor_labels: true
...
...
...

2) Hot start service view

After loading, as shown in the figure below, no data-nodata
Insert picture description here

4、relabel_configs：action为keep

1) Reserve a task in prometheus.yml

cat  /home/monitor/prometheus/prometheus.yml
...
...
...
  - job_name: 'es_cluster'  
    relabel_configs:     #添加relabel_configs配置，添加在job_name是es_cluster的下面
    - source_labels:  ["job"]   #匹配原labels名称，此处匹配是job
      regex: "(.*)"  #正则匹配所有
      action: replace  #动作是替换，除了替换别的动作
      replacement: $1  #替换第一个值
      target_label: "es651"  #重新定义的job的label值
      
    - source_labels: ["job"]  #匹配原labels名称，此处匹配是job
      action: keep   #将原标签为job实例保留

    scrape_interval: 1m
    static_configs:
    file_sd_configs:
      - files:
        - /home/monitor/prometheus/conf.d/es_cluster.json
    honor_labels: true
...
...
...

2) Hot start service view

After loading, as shown below
Insert picture description here

5、relabel_configs：action为labeldrop

1) Delete the original label of a task in prometheus.yml

cat  /home/monitor/prometheus/prometheus.yml
...
...
...
  - job_name: 'es_cluster'  
    relabel_configs:     #添加relabel_configs配置，添加在job_name是es_cluster的下面
    - source_labels:  ["job"]   #匹配原labels名称，此处匹配是job
      regex: "(.*)"  #正则匹配所有
      action: replace  #动作是替换，除了替换别的动作
      replacement: $1  #替换第一个值
      target_label: "es651"  #重新定义的job的label值
      
    - source_labels: ["job"]  #匹配原labels名称，此处匹配是job
      action: keep   #将原标签为job实例保留
      
    - action: labeldrop  #将原标签为job删除
      regex: "job"  #匹配名称为job的标签

    scrape_interval: 1m
    static_configs:
    file_sd_configs:
      - files:
        - /home/monitor/prometheus/conf.d/es_cluster.json
    honor_labels: true
...
...
...

2) Hot start service view

After loading, as shown below
Insert picture description here