Remark

Remark the target

Remarking the target is a powerful tool to dynamically rewrite the target label before data capture. In each data capture configuration, multiple relabel steps can be defined, and they will be executed in sequence according to the defined book order.
For each target discovered, Prometheus will perform the following operations by default:

The label of the job is set to the value of the job_name to which it belongs
The value of the __address__ tag is the socket address of the target "<host>:<port>"
The value of the instance label is the value of __address__
The value of the __scheme__ tag is the protocol (http or https) used when crawling the indicators on the target
The value of the __metrics_path__ tag is the URL path used when crawling the metrics on the target, the default is "/metrics"
The value of the __param_<name> tag is the value of the first parameter named <name> in the passed URL parameters

During the re-marking period, meta tags starting with "__meta__" on the target can also be used. The meta tags added by each service discovery mechanism to its target will be different.
After the relabeling is complete, all labels on the target starting with "__" will be removed. If the value of the label needs to be stored temporarily during the relabeling process, it must be saved with the label name "__tmp" to avoid the same Prometheus's built-in label conflicts.

Re-marking basic configuration

Insert picture description here

parameter	Detailed
source_labels	Quote those existing tags, you can quote multiple
target_label	Assign the referenced label to the new label
separator	What kind of connector to use when referencing multiple tags, the default is ";"
regex	Perform regular matching on the value of the reference label, and the matched value can be assigned to target_label
action	What operation should be performed on the matched result, the default is "replace"
replacement	Equal to the value matched by regex

Action field can perform actions

Replace tag value:

replace first concatenates the value of each label specified in source_labels, then matches the regular expression in the regex field to source_labels, if it matches, replaces the label value defined in the target_label field with the value saved in the replacement field
Hashmod sets the value of target_labels to a hash value, and the hash is generated by the hash module specified by the modules field to calculate the serial value of each label on source_labels

Delete indicator: each indicator name here corresponds to a target

Keep, when the regex cannot match the serial value of each label on the source_labels on the target, delete the target
drop, when the regex can match the serial value of each label on the source_labels on the target, delete the target

Create or delete labels:

labelmap, which matches regex to all label names, and then assigns the value of the matched label to the label name specified by the replacement field; it is usually used to take out a part of the matched label to generate a new label
labeldrop, match the regex to all label names, and the matched labels will be deleted from the target label set
labelkeep, match the regex to all label names, and the labels that cannot be matched will be deleted from the label set of the target

Note: Make sure that after labeldrop or labelkeep operation, the remaining label set can still uniquely label the indicator

Example of remarking

rplace

Modify Prometheus configuration file

  - job_name: 'consul-node'
    consul_sd_configs:
    - server: "192.168.0.181:8500"
      tags:
      - "nodes"
      refresh_interval: 2m
    relabel_configs:
    - source_labels:
      - __scheme__
      - __address__
      - __metrics_path__
      regex: "(http|https)(.*)"
      separator: ""
      target_label: "endpoint"
      replacement: "${1}://${2}"
      action: replace

Explanation: Matching "__scheme__, __address__, __metrics_path__" these three tags contain http or https and all the values in the tag, use replacement to reorganize the matched values and assign them to the endpoint in target_label

Restart Prometheus to view the newly created tags
Insert picture description here

labelmap

  - job_name: 'consul-node'
    consul_sd_configs:
    - server: "192.168.0.181:8500"
      tags:
      - "nodes"
      refresh_interval: 2m
    relabel_configs:
    - regex: "(job|app)"
      replacement: ${1}_name
      action: labelmap

Before
Insert picture description here
re-marking : After re-marking:

Explanation: Match all labels on the target. For the matched labels, a new label name will be generated with the name of the label as the prefix and end with "_name". The value is the same as the value of the original label

Index name relabel operation

drop

  - job_name: 'consul-node'
    consul_sd_configs:
    - server: "192.168.0.181:8500"
      tags:
      - "nodes"
      refresh_interval: 2m
    metric_relabel_configs:
    - source_labels:
      - __name__
      regex: "go_memstats.*"
      action: drop

Check the metric before generating:
Insert picture description here

View metric after generation:
Insert picture description here

Prometheus monitoring and learning road (3)