Probe of k8s Pod container

---

1. Probe of Pod container

Probes are regular diagnostics performed on containers by kubelet (probes in pods are divided into three categories):

• The liveness probe ( livenessProbe ) detects whether the container is running normally. If the detection fails, kubelet will kill the container (not the Pod), and the container will decide whether to restart based on the restart strategy.

• The readiness probe ( readinessProbe ) detects whether the Pod can enter the READY state and be ready to receive requests. If the detection fails, the Pod will enter the NOTREADY state (READY is 0/1) and be kicked out from the endpoints of the associated service resources. The service will no longer forward access requests to this Pod.

• The startup probe ( startupProbe ) detects whether the application in the container is started successfully. Before the startup probe detects successfully, other types of probes will be temporarily disabled.

Note: The startup probe only meets the configuration once after the container is started, and then no further detection is performed. Survival probes and readiness probes will continue to detect until the end of the Pod life cycle

kubectl explain pod.spec.containers

2. Detection method of probe

• exec : Set the Linux command executed in the container through the command field for detection. If the command return code is 0, the detection is considered successful. If the return code is non-0, the detection fails.
• httpGet : Initiate an HTTP GET request to the specified port and uri path of the container. If the HTTP return status code is >=200 and <400 (2XX, 3XX), the detection is considered successful. If the return status code is 4XX, 5XX, the detection fails.
• tcpSocket : By sending a tcp three-way handshake connection to the specified port of the container. If the port is correct but the tcp connection is successful, the detection is considered successful. If the tcp connection fails, the detection fails.

The probe detection results have the following values:
Success: Indicates that the test passed.
Failure: Indicates that the test failed.
Unknown: Indicates that the detection did not proceed normally.

Official website example:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

2.1 Use of survival probes

2.1.1 exec mode

//示例1：exec方式
apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
 - name: liveness
    image: k8s.gcr.io/busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 60
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      failureThreshold: 1
      initialDelaySeconds: 5
      periodSeconds: 5

• initialDelaySeconds : Specifies that kubelet should wait for 5 seconds before executing the first detection. That is, the first detection is executed 6 seconds after the container is started. The default is 0 seconds, the minimum value is 0.
• periodSeconds : specifies that the kubelet should perform a survival probe every 5 seconds. The default is 10 seconds. The minimum value is 1.
• failureThreshold : When a probe fails, the number of times Kubernetes will retry before giving up. Aborting in the case of liveness detection means restarting the container. Pods that are abandoned during readiness detection will be labeled as not ready. The default value is 3. The minimum value is 1.
• timeoutSeconds : How many seconds to wait after the probe's timeout. The default value is 1 second. The minimum value is 1. (Prior to Kubernetes 1.20, the exec probe ignored timeoutSeconds and the probe continued running indefinitely, possibly even beyond the configured deadline, until results were returned.)

You can see that there is only one container in the Pod. The kubelet needs to wait 5 seconds before performing the first probe. The kubelet will perform a survival probe every 5 seconds. kubelet executes the command cat /tmp/healthy in the container to detect. If the command is executed successfully and the return value is 0, kubelet will consider the container to be healthy and alive. When the 31st second is reached, this command returns a non-zero value and the kubelet will kill the container and restart it.

//示例2：exec方式
apiVersion: v1
kind: Pod
metadata:
  name: myapp-exec
spec:
  containers:
  - image: soscscs/myapp:v1
    name: myapp-static
    ports:
    - containerPort: 80
    command:
    - /bin/sh
    - -c
    - 'touch /opt/test.txt; sleep 30; rm -f /opt/test.txt; sleep 3600'
    livenessProbe:
      exec:
        command: ["test", "-e", "/opt/test.txt"]
      initialDelaySeconds: 1
      periodSeconds: 3
      failureThreshold: 2
  dnsPolicy: ClusterFirst
  restartPolicy: Always

kubectl describe pod demo-exec

2.1.2 httpGet method

//示例1
apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/liveness
    args:
    - /server
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
        httpHeaders:
        - name: Custom-Header
          value: Awesome
      initialDelaySeconds: 3
      periodSeconds: 3

In this configuration file, you can see that Pod has only one container. The initialDelaySeconds field tells kubelet that it should wait 3 seconds before performing the first probe. The periodSeconds field specifies that kubelet executes a survival probe every 3 seconds. The kubelet will send an HTTP GET request to the service running in the container (the service will listen on port 8080) to perform the detection. If the handler under the /healthz path on the server returns a success code, the kubelet considers the container to be healthy. If the handler returns a failure code, the kubelet kills the container and restarts it.

Any return code greater than or equal to 200 and less than 400 indicates success, other return codes indicate failure.

vim httpget.yaml
apiVersion: v1
kind: Pod
metadata:
  name: liveness-httpget
  namespace: default
spec:
  containers:
  - name: liveness-httpget-container
    image: soscscs/myapp:v1
    imagePullPolicy: IfNotPresent
    ports:
    - name: http
      containerPort: 80
    livenessProbe:
      httpGet:
        port: http
        path: /index.html
      initialDelaySeconds: 1
      periodSeconds: 3
      timeoutSeconds: 10
	  
kubectl create -f httpget.yaml

kubectl exec -it liveness-httpget -- rm -rf /usr/share/nginx/html/index.html

kubectl get pods

//示例2
vim demo-httpget.yaml
apiVersion: v1
kind: Pod
metadata:
  name: myapp-httpget
spec:
  containers:
  - image: soscscs/myapp:v1
    name: myapp-httpget
    ports:
    - name: http
      containerPort: 80
    livenessProbe:
      httpGet:
        port: http
        path: /index.html
      initialDelaySeconds: 5
      periodSeconds: 5
      failureThreshold: 3
      timeoutSeconds: 10
  dnsPolicy: ClusterFirst
  restartPolicy: Always

kubectl describe pod myapp-httpget

2.1.3 tcpSocket method

vim demo-tcp.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: myapp-tcp
spec:
  containers:
  - image: soscscs/myapp:v1
    name: myapp-tcp
    ports:
    - name: http
      containerPort: 80
    livenessProbe:
      httpGet:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
      failureThreshold: 2
      timeoutSeconds: 10
  dnsPolicy: ClusterFirst
  restartPolicy: Always

kubectl apply -f demo-tcp.yaml 

kubectl get pods -owide -w

2.2 Use of readiness probes

kubectl create svc clusterip demo-svc --tcp=8080:80 --dry-run=client -oyaml > demo-svc.yaml
vim demo-httpget.yaml   //添加标签
kubectl delete -f demo-httpget.yaml
kubectl apply -f demo-httpget.yaml

vim demo-svc.yaml

kubectl apply -f demo-svc.yaml 
kubectl get pod,svc

kubectl describe svc demo-svc

[root@master01 demo]# kubectl exec -it myapp-httpget sh
/ # cd /usr/share/nginx/html/
/usr/share/nginx/html # ls
50x.html    index.html
/usr/share/nginx/html # mv index.html index.html.bak
/usr/share/nginx/html # exit

kubectl get pods --show-labels -w

 kubectl describe pod myapp-httpget

kubectl describe svc demo-svc

2.3 Start the use of probes

Necessary to enable probe existence:

Sometimes, there are some existing applications that require a long initialization time when starting. In this case, it is tricky to set the survival detection parameters without affecting the detection of deadlocks that respond quickly. The trick is to use the same command to set up startup detection, for HTTP or TCP detection, by setting the failureThreshold * periodSeconds parameters to be long enough to handle the worst case startup time. (Because after the startup probe is started, the other two probes will enter a temporarily disabled state, so sufficient time is reserved for container startup to ensure the smooth start of the business in the container. And the startup probe will only Start once, and subsequent work is completely handed over to the other two probes until the end of the container's life cycle).

ports:
 1. name: liveness-port
  containerPort: 8080
  hostPort: 8080
 
livenessProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 1
  periodSeconds: 10
 
startupProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 30
  periodSeconds: 10

Here, thanks to the startup probe, the application will have up to 5 minutes (30 * 10 = 300s) to complete its startup process. Once the startup detection is successful, the survival detection task will take over the detection of the container and respond quickly to container deadlocks. If the startup probe never succeeds, the container is killed after 300 seconds and further processing is performed according to the restartPolicy.

2.4 Pod container startup and exit actions

Start action and exit action are two important attributes of Pod, used to define the actions to be performed when the container starts and stops.

1. Startup Probes:
   Startup action is a probe mechanism used to detect whether the container is successfully started . When a container in a Pod starts, Kubernetes will periodically perform startup actions to check whether the container is ready. If the startup action fails, Kubernetes can mark the Pod as unavailable and adopt an appropriate restart strategy (for example, restart the Pod or perform a rolling update).
   The main purpose of using a start action is to ensure that the container is ready to handle requests before running normally . For example, a container may take some time to initialize a database connection, load configuration files, or complete other startup tasks. The startup action ensures that these necessary initialization steps have been completed before the container is considered "ready".

2. Termination Probes:
   The termination action is a probe mechanism used to detect whether the container has been stopped correctly. When a container in a Pod receives a termination signal (for example, because the scheduler needs to stop the Pod or perform a rolling update), Kubernetes will periodically perform exit actions to check whether the container has terminated successfully. If the exit action fails, Kubernetes can take appropriate measures, such as waiting for a period of time and then forcibly terminating the container.
   The main purpose of using the exit action is to ensure that the container completes the necessary cleanup and resource release work before being terminated . For example, the container may need to write unsaved data to disk, close network connections, or clean up temporary files. The exit action ensures that these cleanup tasks have been completed before the container is terminated.

By using startup actions and exit actions, Kubernetes can more accurately control the life cycle of Pods and ensure correct behavior of containers during startup and stop. These capabilities are critical for building reliable, highly available containerized applications.

postStart configures the exec.command field to set the Linux command to implement additional operations that will be performed when the application container starts.

preStop configures the exec.command field to set the Linux command to implement the last operation that will be performed when the application container exits.

//示例
apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-demo
spec:
  containers:
  - name: lifecycle-demo-container
    image: soscscs/myapp:v1
    lifecycle:   #此为关键字段
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the postStart handler >> /var/log/nginx/message"]      
      preStop:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the prestop handler >> /var/log/nginx/message"]
    volumeMounts:
    - name: message-log
      mountPath: /var/log/nginx/
      readOnly: false
  initContainers:
  - name: init-myservice
    image: soscscs/myapp:v1
    command: ["/bin/sh", "-c", "echo 'Hello initContainers'   >> /var/log/nginx/message"]
    volumeMounts:
    - name: message-log
      mountPath: /var/log/nginx/
      readOnly: false
  volumes:
  - name: message-log
    hostPath:
      path: /data/volumes/nginx/log/
      type: DirectoryOrCreate

3. Summary

Pod container probes (health checks) 3 types

• The liveness probe (livenessProbe) detects whether the container is running normally. If the detection fails, Kubelet kills the container (the Pod will decide whether to restart based on the restart strategy)
• The readiness probe (readinessProbe) detects whether the Pod can enter the ready state (for example, the ready status bar changes to 1/1) and is ready to receive requests. If the detection fails, the Pod will change to the NotReady state (for example, the ready status bar changes to 0/1), the service will delete the associated endpoint (endpoint), and will no longer forward requests to the Pod that failed the readiness detection.
• The startup probe (startupProbe) detects whether the application in the container is started successfully. Both the liveness probe and the readiness probe are temporarily disabled until the startup probe is successful.

3 types of probe detection methods

• exec: Detection is performed by setting the Linux command executed in the container in the command field. If the command return code is 0, the detection is considered normal. If the command return code is non-0, the detection is considered failed.
• tcpSocket: By sending a TCP three-way handshake connection request to the specified port of the container. If the port is correct and the tcp connection is successful, the detection is considered normal. If the tcp connection fails, the detection fails.
• httpGet: Initiates an HTTP GET request to the specified port and URL path of the container. If the HTTP response status code returns 2XX 3XX, the detection is considered normal. If the response status code returns 4XX 3XX, the detection fails.

Probe parameters:
initialDelaySeconds: Set the delay in seconds after the container is started to start detection
periodSeconds: The interval between each detection (seconds)
failureThreshold: Determine detection failure after several consecutive detection failures
timeoutSeconds: Detection timeout waiting time (seconds)

Pod application container life cycle startup action and exit action
spec.containers.lifecycle.postStart Configure exec.command field settings Extra command operations when the application container in the Pod starts
spec.containers.lifecycle.preStop Configure exec.command field settings Delete Pod The command operation executed before the application container exits