[Cloud native--Kubernetes] Pod resource management and probe detection

1. Resource constraints

1.1 Concept

• When defining a Pod, you can optionally set the required amount of resources for each container. The most common configurable resources are CPU and memory size, and other types of resources
• When a request resource is specified for a container in a Pod, the scheduler uses this information to decide which node to schedule the Pod on. When the limit resource is also specified for the container, the kubelet will ensure that the running container will not use more than the set limit resource. Kubelet will also reserve the set amount of request resources for the container for use by the container
• If the node the Pod is running on has enough resources available, the container can use more than the set request resource amount. However, the container cannot use more than the set limit resources
• If the memory limit value is set for the container, but the memory request value is not set, Kubernetes will automatically set the request value that matches the memory limit. Similarly, if the CPU limit value is set for the container but the CPU request value is not set, Kubernetes will automatically set the CPU request value for it and match it with the CPU limit value Official
  example
  https://kubernetes.io/docs /concepts/configuration/manage-compute-resources-container/

1.2 Resource Requests and Limits for Pods and Containers

spec.containers[].resources.requests.cpu #Define the pre-allocated CPU resources when creating containers
spec.containers[].resources.requests.memory #Define the pre-allocated memory resources when creating containers
spec.containers[].resources. limits.cpu #Define the resource upper limit of cpu
spec.containers[].resources.limits.memory #Define the resource upper limit of memory

1.3 CPU resource unit

• The request and limit of CPU resources are in units of cpu. A cpu in Kubernetes is equivalent to 1 vCPU (1 hyperthread)
• Kubernetes also supports requests with fractional CPUs. A container whose spec.containers[].resources.requests.cpu is 0.5 can obtain half of the CPU resources of a CPU (similar to the time slice of CPU resources by Cgroup). The expression 0.1 is equivalent to the expression 100m (millicore), which means that the total amount of CPU time that the container can use per 1000 milliseconds is 0.1*1000 milliseconds

1.4 Memory resource unit

• The memory request and limit are in bytes. Can be expressed as an integer, or in units of exponents with a base 10 (E, P, T, G, M, K), or in units of an exponent with a base 2 (Ei, Pi, Ti, Gi, Mi, Ki) to represent. Such as 1KB=103=1000, 1MB=106=1000000=1000KB, 1GB=10^9=1000000000=1000MB 1KiB=2^10=1024, 1MiB=2^20=1048576=1024KiB
• PS: When buying a hard disk, the quantity reported by the operating system is smaller than the product marked or claimed by the merchant. The main reason is that the marked is in MB and GB. The binary system is the processing unit, so when checking the hard disk capacity, the unit of MiB and GiB is used, 1GB=2^30=1,073,741,824. In comparison, 1GiB is 1,073,741,824-1,000,000,000=73,741,824Byte more than 1GB, so the detection Actual results were less than indicated—some

1.5 Characteristics of Requests and Limits of CPU and memory

• Both Requests and Limits are optional. When Pod is created and updated, if Requests and Limits are not set, the default value provided by the system is used, which depends on the cluster configuration.
• If Requests is not configured, it is set equal to Limits by default.
• requests is the amount of resources that need to be reserved when creating the container. If it cannot be satisfied, the pod cannot be scheduled. However, this is not the resource actually used by the container to run. The actual resource used by the container may be larger or smaller than this.
• Limit is the upper limit that limits the resources that pod containers can use. The resources used by the container cannot exceed this limit. In any case, Limits should be set to be greater than or equal to Requests.

1.6 Examples

vim demo1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: web
    image: nginx
    env:
    - name: WEB_ROOT_PASSWORD
      value: "password"
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  - name: db
    image: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: "abc123"
    resources:
      requests:
        memory: "512Mi"
        cpu: "0.5"
      limits:
        memory: "1Gi"
        cpu: "1"

#发布
kubectl apply -f demo1.yaml 
#查看pod信息
kubectl get pod
kubectl describe pod frontend 

#查看pod详细信息，查询pod所在节点
kubectl  get pod -o wide 

#查看节点信息
kubectl  describe nodes k8s-node1

2. Probe

Health check, also known as probe (Probe): The probe is performed by kubelet to periodically diagnose the container.

2.1 Probe rules

• StartupProbe (newly added in version 1.17 of the startup probe) : Determine whether the application in the container has been started, mainly for applications that cannot determine the specific startup time. If startupProbe detection is configured, all other probes are in an invalid state until the startupProbe status is success, and other probes will not work until it succeeds. If the startupProbe fails, the kubelet will kill the container and the container will be restarted according to the restartPolicy. If the container does not have a startupProbe configured, the default status is Success.
• Survivability detection, to determine whether the pod needs to be restarted.
  LivenessProbe (survival probe) : Determine whether the container is running. If the probe fails, kubectl will kill the container, and the container will set the pod state according to the restartPolicy. If the container does not provide a liveness probe, the default status is Success.
• Readiness detection, to determine whether the pod can provide normal services
  ReadinessProbe (readiness probe) : generally used to detect whether the program in the container is healthy, if its return value is success, it means that the container has been started, and the program has been completed is a state where traffic can be accepted.

Note: The above rules can be defined at the same time. Before the readinessProbe detects successfully, the running state of the Pod will not change to the ready state.

The difference between startupProbe, livenessProbe, and ReadinessProbe:
startupProbe: Pod is only detected once, and the remaining two types will always be detected as long as your pod exists.
LivenessProbe, ReadinessProbe: It is recommended to use interface-level health checks

2.2 Three inspection methods supported by Probe

• Exec: Execute a command in the container. If the return value is 0, the container is considered healthy.

• TCPSocket: Check whether the port in the container is connected through the TCP connection. If it is connected, the container is considered healthy.

• HTTPGet: Check whether the program is normal through the API address exposed by the application. If the status code is between 200 and 400, the container is considered healthy.

2.3 Three types of results obtained by detection

For each probe, one of the following three results will be obtained:

• SUCCESS: The container passed the diagnostics
• Failed: Container failed diagnostics
• Unknown: The diagnostic failed, so no action will be taken

2.4 exec

vim exec.yaml
 
apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness				#为了健康检查定义的标签
  name: liveness-exec
spec:						#定义了Pod中containers的属性
  containers:
  - name: liveness
    image: busybox
    args:						#传入的命令
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy;sleep 600
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5			#表示pod中容器启动成功后，多少秒后进行健康检查 
      periodSeconds: 5				#在首次健康检查后，下一次健康检查的间隔时间 5s

Probe optional parameters:

• initialDelaySeconds: Specifies that the kubelet should wait 5 seconds before performing the first detection, that is, the first detection is executed 6 seconds after the container is started. The default is 0 seconds, and the minimum value is 0.
• periodSeconds: specifies that the kubelet should perform a liveness probe every 5 seconds. The default is 10 seconds. The minimum value is 1.
  -== failureThreshold==: When a probe fails, the number of times Kubernetes will retry before giving up. Giving up in case of a liveness probe means restarting the container. Abandoned pods with readiness probes are marked as not ready. The default value is 3. The minimum value is 1.
• timeoutSeconds: How many seconds to wait after the probe times out. The default is 1 second. The minimum value is 1. (Before Kubernetes version 1.20, the exec probe ignored the timeoutSeconds probe and continued to run indefinitely, possibly beyond the configured time limit, until it returned a result.) In
  this configuration file, you can see that there is only one container in the Pod .
• periodSecondsfield specifies that the kubelet should perform a liveness probe every 5 seconds.
• initialDelaySecondsfield tells the kubelet that it should wait 5 seconds before performing the first probe.

Kubelet executes the command cat /tmp/healthy in the container to detect. If the command executes successfully and the return value is 0, the kubelet considers the container to be healthy and alive. If this command returns a non-zero value, the kubelet will kill the container and restart it.

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy;sleep 60
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5

#创建pod
kubectl create  -f exec.yaml 
#跟踪查看pod 信息
kubectl get pod  -o wide -w
#查看pod 的消息信息
kubectl describe pod liveness-exec 

2.5 httpget

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/liveness
    args:
    - /server
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
        httpHeaders:
        - name: Custom-Header
          value: Awesome
      initialDelaySeconds: 3
      periodSeconds: 3

In the configuration file, you can see that the Pod has a single Container. The periodSeconds field specifies that kubectl should perform liveness probes every 3 seconds. The initiaDelaySeconds field tells the kubelet that it should wait 3 seconds before performing the first probe. To perform a probe, kubectl sends an HTTP GET request to a server running in the Container and listening on port 8080. If the handler for the server/healthz path returns a success code, kubectl considers any code greater than or equal to 400 to indicate success, and any other code to indicate failure.

#加载yaml文件
kubectl create -f http.yaml                                      
#查看pod的详细参数
kubectl get pod -o wide -w
kubectl describe pod liveness-httpget       

2.6 tcpsocket

Define TCP liveness probes

The third type of liveness probe uses TCP sockets, with this configuration the kubelet will try to open the container's socket on the specified port. If a connection can be established, the container is considered healthy, and if not, the container is considered to be a failed container.

apiVersion: v1
kind: Pod
metadata:
  name: probe-tcp
spec:
  containers:
  - name: nginx
    image: soscscs/myapp:v1
    livenessProbe:
      initialDelaySeconds: 5
      timeoutSeconds: 1
      tcpSocket:
        port: 8080
      periodSeconds: 3

As you can see, the configuration for TCP inspection is very similar to HTTP inspection. The following example uses both readinessProbe and livenessProbe probes.

The kubelet sends the first readiness probe 5 seconds after the container starts. This will try to connect to port 8080 of the goproxy container. If the probe is successful, the Pod will be marked as ready and the kubelet will continue to run the check every 10 seconds.

In addition to the readiness probe, this configuration includes a liveness probe. The kubelet will perform the first liveness probe 15 seconds after the container starts. Similar to readiness detection, it will try to connect to port 8080 of the goproxy container. If the liveness probe fails, the container will be restarted

Write yaml file

kubectl  create  -f tcp.yaml
#查看pod的状态和详细信息
kubectl  get pods -o wide -w
kubectl describe pod liveness-httpget

As shown, the configuration of the TCP check is very similar to the HTTP check. This example uses both readiness and liveness probes. The kubelet will send the first readiness probe 5 seconds after the container starts. These try to connect to the goproxy container on port 8080. If the probe is successful, the container will be marked as ready and the kubelet will continue to run checks every 10 seconds.

In addition to readiness probes, this configuration also includes liveness probes. 15 seconds after the container is started, the kubelet will run the first activity talk, like a readiness probe, which tries to connect goproxy to the container on port 8080. If the liveness probe fails, the container will be restarted.

3. Summary

Probes (3 types)
1.livenessProbe (survival probe): judge whether the container is running normally, if it fails, kill the container (not pod), and then restart the container according to the restart strategy
2.readinessProbe (ready probe): judge the container Whether it can enter the ready state, if the probe fails, enter the noready state, and remove the container from the endpoints of the service
3.startupProbe: Determine whether the application in the container has started successfully. Before the success state, other probes are in an invalid state

Inspection methods (3 types)
1.exec: use the command field to set the command, execute the command in the container, if the command returns a status code of 0, the detection is considered successful
2.httpget: execute http get access by accessing the specified port and url path . If the returned http status code is greater than or equal to 200 and less than 400, it is considered successful
3. tcpsocket: connect the pod (IP) and the specified port through tcp, if the port is correct and the tcp connection is successful, the detection is considered successful

Optional parameters of the probe
1.initialDelaySeconds: How many seconds after the container starts to perform the detection
2.periodSeconds: The cycle frequency of the detection, every number of seconds to perform a detection
3.failureThreshold: After the detection fails, it is allowed to try a few more times
4.timeoutSeconds: Probe wait timeout