[Prometheus & Pushgateway] Push data to pit

Only numeric types are allowed for metric values

报错text format parsing error in line 1: expected float as value, got “1.1.1.1”

Reason:
In order to draw better, only numeric indicator values ​​are allowed to be returned

$ echo ipaddr 1.1.1.1 curl --data-binary @- -g http://ip:9090/metrics/job/pushgateway/instance/test

ipaddr 值为 1.1.1.1 是会报错
text format parsing error in line 1: expected float as value, got "1.1.1.1"

解决方法：
将1.1.1.1 转为数字

function checkIP()
{
    ip=$1
    if [ $ip != "${1#*[0-9].[0-9]}" ]; then
        # IPv4
        a=`echo $ip | awk -F'.' '{print $1}'`
        b=`echo $ip | awk -F'.' '{print $2}'`
        c=`echo $ip | awk -F'.' '{print $3}'`
        d=`echo $ip | awk -F'.' '{print $4}'`
    
        echo "$(((a<<24)+(b<<16)+(c<<8)+d))"
    elif [ "$ip" != "${1#*:[0-9a-fA-F]}" ]; then
        # IPv6
        echo $ip
    else
        echo 0
    fi
}

Reference link: https://github.com/prometheus/prometheus/issues/2227

The metric value can only accept up to 16 digits. After 16 digits, the number becomes 0

“FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF”：340282366920938463463374607431768211455

$ echo ipaddr 340282366920938463463374607431768211455 \ 
curl --data-binary @- -g http://ip:9090/metrics/job/pushgateway/instance/test

实际结果：
ipaddr{instance="test"}  340282366920938500000000000000000000000

Pushgateway data persistence

In order to prevent pushgateway reboot or hang accident, loss of data, we can -persistence.fileand -persistence.intervalparameter data persistence down.

---

Prometheus official website explanation

Metric labels and label naming

Metric name and label data model

exceeded maximum resolution of 11,000 points per timeseries. Try decreasing the query resolution

When performing this operation:
GET http://xxx/prometheus/api/v1/query_range?query=bps{mac=~'xx:xx:xx:xx:xx:xx'}&start=2019-09-19T09: 29:26Z&end=2019-09-20T09:29:26Z&step=15s&timeout=60s

Reason: Prometheus sets a hard limit of 11k data points for each query.
Reference link:
https://github.com/prometheus/prometheus/issues/1968
https://github.com/prometheus/prometheus/issues/2253

docker-compose restart will not take effect the newly changed docker-compose.yml

必须 docker-compose down

然后 docker-compose up 

Open hot update

Starting from 2.0, the hot reload function is disabled by default.
If you need to enable it, you need to add the --web.enable-lifecycle parameter when starting Prometheus

热更新加载方法有两种：
kill -HUP pid
curl -X POST http://IP/-/reload  【推荐】

Blackbox_exporter prompts an error: Timeout reading from socket

解决方法：
重启 blackbox 容器

Pushgateway Delete Group 报错：Deleting metric group failed: Bad Request

如果 key="", 会报错Deleting metric group failed: Bad Request
解决：
对每个KEY 设置默认值，保证每一个 key 都有值

PushGateway push and Prometheus pull time setting

Each time Prometheus pulls data from PushGateway, it is not all the data pushed by the user during the pull cycle, but the data pushed to PushGateway the last time,
so it is recommended to set the push time less than or equal to the time pulled by Prometheus to ensure The data pulled each time is the latest Push.