XV. API resource indicators and custom indicators API

Resource indicators: Metrics-Server

Resource indicators: Metric-Server Introduction

From then k8s v1.8, introduced Metric-APIpreviously in the use of heapster access to resources index, heapster have their own acquisition path, not by apiServer, so the data before the resource index and can not apiServer direct access, users, and other components must Kubernetes through the master proxy way to access to. later k8s introduces resource index API (metrics API), with metrics Server components, but also to collect some of the data, but also exposed api, api but because to unify, how to request api-server's /apis/metricsrequest to forward it to the metrics Server, the solution is: kube-aggregator, so index data resources on directly from the api in k8s, no longer through other means.

• Metrics API can only query the current measurement data is not stored historical data
• Metrics API URI is /apis/metrics.k8s.io/, maintenance k8s.io/metrics
• Metrics-server must be deployed in order to use the API, metrics-server data acquired by calling Kubelet Summary API

Metrics-Server mode indicator data collection is to collect data from the Summary API provided by each node kubelet, collect Node and Pod core indicator data resources, mainly memory and cpu usage aspects of the information collected and stored in memory, so when can not view resource data through kubectl top historical circumstances, other resources index data is collected by prometheus.

k8s are many components depending upon the functionality of the API resource metrics, such kubectl top, hpa, if not a resource index API interface, these components are not running;

A new generation of monitoring system consists of core indicators pipelines and pipeline monitoring indicators synergistic composition

• Pipeline core indicators: the kubelet, metrics-server and api provided by the API server composed; the CPU cumulative utilization, memory utilization in real-time, disk usage and resource utilization of container pod
• Monitoring the pipeline: a variety of indicators used to collect data from the system and provided to end users, storage systems and HPA, they contain many non-core indicators and core indicators. Non-core indicators can not be resolved k8s

Metric-Server deployment

Download file yaml

for i in auth-delegator.yaml auth-reader.yaml metrics-apiservice.yaml metrics-server-deployment.yaml metrics-server-service.yaml resource-reader.yaml;do wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/metrics-server/$i; done

Because of the wall, the mirror image to download in advance, of course, can also modify the related files manually yaml

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.5
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.5 k8s.gcr.io/metrics-server-amd64:v0.3.5

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/addon-resizer:1.8.5
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/addon-resizer:1.8.5 k8s.gcr.io/addon-resizer:1.8.5

Modify the file, or an error

modifyresource-reader.yaml

# 在resources下添加一行nodes/stats, 下列代码为部分代码
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes/stats         #添加此行
  - nodes
  - namespaces

modifymetrics-server-deployment.yaml

Default index data acquisition HTTP port of 10255-based kubelet from, but for security communication purposes, kubeadm when initializing the cluster will turn off port 10255, making it impossible to obtain normal data

# 第一个container metrics-server的command只留下以下三行
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.5
        command:
        - /metrics-server
        - --kubelet-insecure-tls                        # 不验证客户端证书
        - --kubelet-preferred-address-types=InternalIP  # 直接使用节点IP地址获取数据

# 第二个container metrics-server-nanny的command中内存和CPU修改为自己需要的具体的数值
        command:
          - /pod_nanny
          - --config-dir=/etc/config
          - --cpu=20m
          - --extra-cpu=0.5m
          - --memory=200Mi #{{ base_metrics_server_memory }}
          - --extra-memory=50Mi
          - --threshold=5
          - --deployment=metrics-server-v0.3.5
          - --container=metrics-server
          - --poll-period=300000
          - --estimator=exponential
          - --minClusterSize=10

createMetric-Server

# 进入到yaml文件目录执行命令
kubectl apply -f ./

# 可以看到pod已经运行起来了
kubectl get pods -n kube-system |grep metrics-server

[root@master ~]# kubectl api-versions|grep metrics #已经可以看到metric的api了
metrics.k8s.io/v1beta1

[root@master ~]# kubectl proxy --port=8080
[root@master ~]# curl http://localhost:8080/apis/metrics.k8s.io/v1beta1
[root@master ~]# curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes

# kubectl可以使用了
[root@master ~]# kubectl top node
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
master   513m         25%    1348Mi          78%
node01   183m         18%    1143Mi          66%

Custom resource indicators: Prometheus

Prometheus can capture a variety of other indicators, prometheus collected metrics and not directly to K8S, because both the data format is not compatible, it is also necessary another component (kube-state-metrics), converted into a data format prometheus the metrics k8s API interfaces recognizable after format conversion, as is the custom API, so we must also use Kubernetes aggregatorregistered in the main server API in order directly through the / apis / access.

k8s-prometheus-adapter project

Prometheus

As shown above, each can be monitored by a dedicated host exporterprovides program interface output monitoring data, and waits for Prometheusthe server to fetch data periodically. It will be calculated according to the rules, if after an alarm rule exists, then the fetch data, alarm satisfying an alarm condition is generated and sent to Alertmanagercomplete alarm aggregation and distribution. When there is a demand to monitor active push target data, you can Pushgatewayreceive and temporarily store data components, and then waits for Prometheusthe server to complete the data acquisition.

• Monitoring Agents: If node_exporterindexes data from multiple dimensions of indicator data collection host, such as the average load, CPU, memory, disk, network, and so on:
• kubelet (cAdvisor): index data collecting container, is collected K8S the core indicators, each container associated index data include: CPU usage, limits, read and write the file system quota limits, and memory usage, the network packet transmission, reception discard rate and so on.
• API Server: API Server collect performance data, including the control performance of the queues, the request rate and delay time length and the like
• etcd: metrics data collection etcd Storage Cluster
• kube-state-metrics: This component can be derived k8s plurality of index data relating to the main counter and metadata information is associated with the resource type, comprising a total number of objects of the type of formulation, resource quotas, and the container status Pod resource label series.

Prometheus find the API Server as a service discovery system for all objects that can be monitored and monitoring cluster
need of special note here, Pod resources need to add the following annotation information can be automatically discovered Prometheus system and its built-in grab index data.

• prometheus.io/scrape: whether to acquire index data, true / false
• prometheus.io/path: URL path used when the index data fetch, often / metrics
• prometheus.io/port: socket port port number to use when crawling index data

When only the desired Prometheus generate custom indicators for the back-end, you can only deploy the service Prometheus, even without lasting function

Prometheus deployed in a cluster k8s

github address

https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus

A list of services to be deployed

• node-exporter: Prometheus the export, the level of monitoring data collected Node
• prometheus: Monitoring server, pull-up data from the node-exporter and stored as time series data.
• kube-state-metrics: Prometheus may be used to convert the query to the index PromQL data into corresponding data k8s
• k8s-prometheus-adpater: Polymerization feed apiserver, that is, a custom-metrics-apiserver implemented
• 开启Kubernetes aggregator功能(See above metric-server)

All services and plug-in installation and deployment

Deployment kube-state-metrics

# 下载相关yaml文件
for i in kube-state-metrics-deployment.yaml kube-state-metrics-rbac.yaml kube-state-metrics-service.yaml; do wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/prometheus/$i; done

# 所有节点都要执行
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/addon-resizer:1.8.6
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/addon-resizer:1.8.6 k8s.gcr.io/addon-resizer:1.8.6
docker pull quay-mirror.qiniu.com/coreos/kube-state-metrics:v1.3.0
docker tag quay-mirror.qiniu.com/coreos/kube-state-metrics:v1.3.0 quay.io/coreos/kube-state-metrics:v1.3.0

# 查看提供的指标数据
curl 10.105.51.200:8080/metrics  # 10.105.51.200 是Service的IP

Deployment and Node Exporter Exporter

Listening 9100 port
Indeed, each node itself can provide index data node by kubelet or cAdvisor, there is no need to install node_exporter program

for i in node-exporter-ds.yml node-exporter-service.yaml; do wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/prometheus/$i; done

kubectl apply -f ./

curl 10.0.0.51:9100/metrics  # 10.0.0.51是node01节点的IP

Alarm system Alertmanager

prometheusThe transmitting alarm information to the alarm rule alertmanager, and then alertmanagerof the received alarm information processing, including de-duplication, and routes the packet to the reception side alarm
alertmanager using persistent storage volumes, PVC, only test here, so this part of the modified ; port 9093 will be Web UI

for i in alertmanager-configmap.yaml alertmanager-deployment.yaml alertmanager-pvc.yaml alertmanager-service.yaml; do wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/prometheus/$i; done

# 修改alertmanager-deployment.yaml的pvc设置
      volumes:
        - name: config-volume
          configMap:
            name: alertmanager-config
        - name: storage-volume
          emptyDir: {}
#          persistentVolumeClaim:
#            claimName: alertmanager

# 修改alertmanager-service.yaml
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 9093
      nodePort: 30093
  selector:
    k8s-app: alertmanager
  type: "NodePort"

kubectl apply -f ./
kubectl get deployments -n kube-system

# 浏览器可以直接访问到Web UI
http://10.0.0.50:30093/#/alerts

Service deployment prometheus

Prometheus provides Web UI, port 9090, the storage volume needed, is provided by volumeClaimTemplates, this test only, so this part of the modification , so the use of mounting deployment Marco

# 官方安装yaml文件
for i in prometheus-configmap.yaml prometheus-rbac.yaml prometheus-service.yaml prometheus-statefulset.yaml; do wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/prometheus/$i; done

# 马哥安装yaml文件
git clone https://github.com/iKubernetes/k8s-prom.git && cd k8s-prom #我只使用了这里边的prometheus文件, 并且把namespace统一修改成了kube-system

[root@master prometheus]# ls
prometheus-cfg.yaml  prometheus-deploy.yaml  prometheus-rbac.yaml  prometheus-svc.yaml

kubectl apply -f ./

# 查看Web UI: 
http://10.0.0.50:30090

Custom metric adapter k8s-prometheus-adapter

PromQL not directly interface to the data source as a custom indicator, it is not a polymerization API server
requiresk8s-prometheus-adapter

# 配置ssl证书
cd /etc/kubernetes/pki/
(umask 077;openssl genrsa -out serving.key 2048)
openssl req -new -key serving.key -out serving.csr -subj "/CN=serving"
openssl x509 -req -in serving.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out serving.crt -days 3650

k8s-prometheus-adapter deployed in custom-metrics default namespace, create secret objects in this namespace

Certificates and private keys, called serving.crt and serving.key

cd /etc/kubernetes/pki/
kubectl create namespace custom-metrics
kubectl create secret generic cm-adapter-serving-certs -n custom-metrics --from-file=serving.crt=./serving.crt --from-file=serving.key=./serving.key

git clone https://github.com/DirectXMan12/k8s-prometheus-adapter
cd k8s-prometheus-adapter/deploy/manifests/

# 编辑：custom-metrics-apiserver-deployment.yaml 第28行. 因为我的promethus部署在了kube-system名称空间中
--prometheus-url=http://prometheus.prom.svc:9090/  -> --prometheus-url=http://prometheus.kube-system.svc:9090/

# 查看API
kubectl api-versions | grep custom
# 列出指标名称
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq '.resources[].name'
# 查看pod内存占用率
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/kube-system/pods/*/memory_usage_bytes" | jq 

HPA automatic elastic scaling

Automatic retractable elastic tool Auto Scaling:

• HPA, Horizontal Pod Autoscaler, two versions, HPA index CPU support only; HPAv2 support resources index API and custom metrics API
• CA, Cluster Autoscaler, resilient and elastic cluster size automatically, can automatically increase or decrease the number of nodes for a cloud environment
• VPA, Vertical Pod Autoscaler, Pod vertical scaling application tool, adjusting objects Pod CPU and memory resource requirements to complete the expanded or contracted
• AR, Addon Resizer, a simplified version of the application Pod vertical scaling tool, based on the number of nodes in the cluster to adjust demand for resources additional components

Horizontal Pod Autoscaling can according to CPU utilization(内存为不可压缩资源) autostretch a Replication Controller, Deployment or the number of Pod Replica Set;

HPA itself is a control cycle, the cycle defined by --horizontal-pod-autoscaler-sync- period of the option controller-manager, default 30s
Pod for an undefined target resource demand, the controller can not define a container HPA CPU utilization rate, and will not take any action for the indicators
for each of the Pod custom metrics, the HPA, not only the original value utilization process

The default length of the delay volume reduction 5min, the time delay expansion length 3min, in order to prevent jitter

HPA currently supports only two versions, one version only supports v1 defined core indicators;

[root@master ~]# kubectl api-versions |grep autoscaling
autoscaling/v1           # 仅支持CPU一种资源指标的扩容
autoscaling/v2beta1      # 支持更多自定义资源指标的扩容
autoscaling/v2beta2      # 支持更多自定义资源指标的扩容

Experiment one: HPA

Creating a pod with limited resources with the command line

kubectl run myapp --image=ikubernetes/myapp:v1 --replicas=1 --requests='cpu=50m,memory=100Mi' --limits='cpu=50m,memory=100Mi' --labels='app=myapp' --expose --port=80

[root@master ~]# kubectl get pod
NAME                   READY   STATUS    RESTARTS   AGE
myapp-cf57cd7b-2r6q2   1/1     Running   0          2m3s

Here we let myapp the pod can automatically horizontal expansion

With kubectl autoscale, in fact, it is to create HPA controller

kubectl autoscale deployment myapp --min=1 --max=8 --cpu-percent=60
# --min：表示最小扩展pod的个数  
# --max：表示最多扩展pod的个数 
# --cpu-percent：cpu利用率

[root@master ~]# kubectl get hpa
NAME    REFERENCE          TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
myapp   Deployment/myapp   0%/60%    1         8         1          4m14s

kubectl patch svc myapp -p '{"spec":{"type": "NodePort"}}'
kubectl get svc |grep myapp

# [root@master ~]# kubectl get svc |grep myapp
# myapp        NodePort    10.99.246.253   <none>        80:31835/TCP   11m

#压测命令
ab -c 100 -n 500000000 http://10.0.0.51:30304/index.html 

[root@master manifests]# kubectl describe hpa |grep -A 3 "resource cpu"
  resource cpu on pods  (as a percentage of request):  81% (40m) / 60%
Min replicas:                                          1
Max replicas:                                          8
Deployment pods:                                       3 current / 3 desired

[root@master manifests]# kubectl get pod
NAME                   READY   STATUS    RESTARTS   AGE
myapp-cf57cd7b-2lqdx   1/1     Running   0          14m
myapp-cf57cd7b-bwm4h   1/1     Running   0          3m19s
myapp-cf57cd7b-fc5ns   1/1     Running   0          91s

# 压测结束五分钟后, 资源恢复到初始值
[root@master manifests]# kubectl describe hpa |grep -A 3 "resource cpu"
  resource cpu on pods  (as a percentage of request):  0% (0) / 60%
Min replicas:                                          1
Max replicas:                                          8
Deployment pods:                                       1 current / 1 desired
[root@master manifests]# kubectl get pod
NAME                   READY   STATUS    RESTARTS   AGE
myapp-cf57cd7b-2lqdx   1/1     Running   0          22m

Experiment two: HPA v2

HPA (v2) support requests from the core indicators metrics-server; and obtaining data from the custom metrics k8s-prometheus-adapter API, a custom class, when a plurality of index calculation, the numerical results of a large win

Rule number one

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp
spec:
  scaleTargetRef:  # 要缩放的目标资源
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 50
  - type: Resource
    resource:
      name: memory
      targetAverageValue: 50Mi

metrics, calculates the required number of copies Pod index list, as the maximum value calculated separately for each index, taking the results of all of the final number of copies of

• external: any object attached to the non-reference global index, the index data may be a component other than the cluster, such as a message queue length
• object: reference a particular cluster index description of a single object, such as a hits-per-second on the other objects Ingress
• pods: specific reference index currently being elastically stretchable object Pod
• resource: resource reference index, i.e. requests currently being elastically stretchable and container objects Pod limits defined metrics
• type: Type Indicator source, for Objects, Pods, Resource

Rule number two

ikubernetes / metrics-app runtime by / metrics and path output http_requests_total http_requests_per_second two indicators
Note prometheus.io/scrape:"true "Pod enable objects to be collected metrics Promethues

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: metrics-app
  name: metrics-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: metrics-app
  template:
    metadata:
      labels:
        app: metrics-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "80"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - image: ikubernetes/metrics-app
        name: metrics-app
        ports:
        - name: web
          containerPort: 80
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-app
  labels:
    app: metrics-app
spec:
  ports:
  - name: web
    port: 80
    targetPort: 80
  selector:
    app: metrics-app

curl 10.98.175.207/metrics  # IP为上一个文件创建的service IP

Creating HPA

PrometheusPod newly created object found by the service discovery mechanism, according to configuration information included in the Acquisition Object discrimination index and annotation provided by the then k8s-prometheus-adapterthese metrics register to custom API, it is provided to HPA(v2)the controller and scheduler scheduling the like is used as evaluation parameters

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: metrics-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: metrics-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metricName: http_requests_per_second
      targetAverageValue: 800m # 800m 即0.8个/秒

# 压测命令
while true; do curl 10.98.175.207/metrics &>/dev/null; sleep 0.1; done # IP为service IP

Pressure test results

[root@master ~]# kubectl describe hpa metrics-app-hpa |grep -A 4 Metrics
Metrics:                               ( current / target )
  "http_requests_per_second" on pods:  4350m / 800m
Min replicas:                          2
Max replicas:                          10
Deployment pods:                       10 current / 10 desired

Add custom metrics http_requests_per_second

Edit k8s-prometheus-adapter/deploy/manifests/custom-metrics-config-map.yamladd a rule:

    rules:
    - seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
      resources:
        overrides:
          kubernetes_namespace: {resource: "namespace"}
          kubernetes_pod_name: {resource: "pod"}
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

Custom Rules Reference document:

The prometheus indicators upgraded to k8s custom metrics, we need to define rules

• https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/config-walkthrough.md
• https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/docs/walkthrough.md

The http_requests_total command http_requests_per_second custom indicators
allow configuration to take effect:
you need to use custom-metrics-config-map.yaml then custom-metrics-apiserver-xxxx Pod manual delete custom-metrics space
Note: After modifying config-map, do not delete Pod, do not take effect

Test:
kubectl GET PODS -w
curl 10.104.226.230/metrics
kubectl Expediting IT Client --image RUN = cirros --rm - / bin / SH
the while to true; do curl HTTP: // App-metrics; the let I ++; SLEEP 0 . $ RANDOM; done # analog pressure

After reaching the test requires a few minutes to see the automatic expansion because: the default length of the delay volume reduction when 5min, 3min when the length of the delay expansion

Reference links

https://www.cnblogs.com/fawaikuangtu123/p/11296510.html
https://www.qingtingip.com/h_252011.html
https://www.servicemesher.com/blog/prometheus-operator-manual/
https://www.cnblogs.com/centos-python/articles/10921991.html
https://pdf.us/tag/docker