The health detection container kubernetes

Brief introduction

This article describes how to configure a container liveness, readiness, startupprobe.

kubeletUse livenessprobe to know when to restart the container. For example, livenessthe probe can be captured to a deadlock (application is running, but not proceed with the subsequent steps). Under such circumstances restarting the container will help to make the application available in more problematic situation.

kubeletUse readinessdetector can know when the container is ready and can begin accepting requests traffic, when all containers in a Pod are ready to put this Pod seen all set. Use of such a control signal which is a rear end Pod Service. In Pod yet ready, it will be removed from the load balancer in the Service.

kubelet using the startupdetector can know when to start the application container. If this type of detector configuration, you can control the container after the successful launch livenessand readinesscheck to ensure that they survive, it will not affect the detector is ready to launch the application. This can be used to slow start containers viability testing, to avoid them being killed before starting the run.

NOTE: This document is to be understood with reference to their official documents, and. If misleading, please criticism.

Define a probe liveness

Many long-running applications will eventually transition to the disconnected state, unless restarted, otherwise can not be restored. Providing Kubernetes livenessprobe to identify and remedy the situation.

Create a Pod, which runs on a k8s.gcr.io/busyboxmirror container. Configuration file as follows. file name:exec-liveness.yaml

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/busybox
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5

In the configuration file, you can see only one container Pod. periodSecondsField specifies the kubelet detection should be performed once every 5 seconds survival. initialDelaySecondsFields told kubeletbefore performing the first probe should wait for 5 seconds. Run kubelet in a container cat /tmp/healthyto be detected. If the command is successful and returns a value of 0, kubelet would think this vessel is healthy survival. If this command returns a non-zero value, kubelet will kill this container and restart it. Run the following command

/bin/sh -c "touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600"

The first 30 seconds of container life, /tmp/healthythe file is there. Execute the command cat /tmp/healthywill return success code. After 30 seconds, run cat /tmp/healthyback to return a failure code.

Create a Pod

# kubectl apply -f /root/k8s-example/probe/exec-liveness.yaml

Within 30 seconds, Pod View events

kubectl describe pod liveness-exec

The output shows yet survive the failure detector

Events:
  Type    Reason     Age        From                 Message
  ----    ------     ----       ----                 -------
  Normal  Scheduled  <unknown>  default-scheduler    Successfully assigned default/liveness-exec to k8s-node04
  Normal  Pulled     22s        kubelet, k8s-node04  Container image "k8s.gcr.io/busybox" already present on machine
  Normal  Created    22s        kubelet, k8s-node04  Created container liveness
  Normal  Started    22s        kubelet, k8s-node04  Started container liveness

After 30 seconds, then look Pod events:

kubectl describe pod liveness-exec

In the lowermost output result, information displayed survived detector fails, the container is killed and rebuilt.

Events:
  Type     Reason     Age               From                 Message
  ----     ------     ----              ----                 -------
  Normal   Scheduled  <unknown>         default-scheduler    Successfully assigned default/liveness-exec to k8s-node04
  Normal   Pulled     47s               kubelet, k8s-node04  Container image "k8s.gcr.io/busybox" already present on machine
  Normal   Created    47s               kubelet, k8s-node04  Created container liveness
  Normal   Started    47s               kubelet, k8s-node04  Started container liveness
  Warning  Unhealthy  5s (x3 over 15s)  kubelet, k8s-node04  Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
  Normal   Killing    5s                kubelet, k8s-node04  Container liveness failed liveness probe, will be restarted

Wait an additional 30 seconds, and checks to see that the container is restarted:

kubectl get pod liveness-exec

NAME            READY   STATUS    RESTARTS   AGE
liveness-exec   1/1     Running   2          3m10s

View details Pod resources:

kubectl describe pod liveness-exec

The following output, a successful restart of the container.

Events:
  Type     Reason     Age                 From                 Message
  ----     ------     ----                ----                 -------
  Normal   Scheduled  <unknown>           default-scheduler    Successfully assigned default/liveness-exec to k8s-node04
  Warning  Unhealthy  35s (x6 over 2m)    kubelet, k8s-node04  Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
  Normal   Killing    35s (x2 over 110s)  kubelet, k8s-node04  Container liveness failed liveness probe, will be restarted
  Normal   Pulled     5s (x3 over 2m32s)  kubelet, k8s-node04  Container image "k8s.gcr.io/busybox" already present on machine
  Normal   Created    5s (x3 over 2m32s)  kubelet, k8s-node04  Created container liveness
  Normal   Started    5s (x3 over 2m32s)  kubelet, k8s-node04  Started container liveness

HTTP defines a request interface state survival

Another type of livenessdetection is to use the HTTP GET request. The following is a profile of the Pod, which runs on a k8s.gcr.io/livenessmirror container.

Create a Pod. File Name: ``

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/liveness
    args:
    - /server
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
        httpHeaders:
        - name: X-Custom-Header
          value: Awesome
      initialDelaySeconds: 3
      periodSeconds: 3

Profile, Pod only one container. periodSecondsField specifies kubelet a check is performed every 3 seconds. initialDelaySecondsTell kubelet field before performing the first detection should wait 3 seconds. kubelet Council (8080 listening services are port) to send the service running in the container a HTTP GETrequest to perform the detection. If the service /healthzhandler in the path returns a success code. Kubelet container is considered to be healthy survival. If the handler returns a failure code, the kubelet will kill this container and restart it.

Any value greater than or equal to 200 and less than 400 labeling success return code, return code other fail are marked.

You can see the source code of service here server.go .

Beginning the container survival for 10 seconds, /healthzthe processing procedure returns a status code of 200. After the processing routine is returned status code 500.

http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    duration := time.Now().Sub(started)
    if duration.Seconds() > 10 {
        w.WriteHeader(500)
        w.Write([]byte(fmt.Sprintf("error: %v", duration.Seconds())))
    } else {
        w.WriteHeader(200)
        w.Write([]byte("ok"))
    }
})

kubelet start 3 seconds after the container began to perform health monitoring. Therefore, previous health checks were successful. But after 10 seconds, the health check will fail, and kubelet will kill and restart the container vessel.

# kubectl apply -f /root/k8s-example/probe/http-liveness.yaml

After 10 seconds, an event is detected by looking Pod survival detector has failed and the container is restarted.

Events:
  Type     Reason     Age              From                 Message
  ----     ------     ----             ----                 -------
  Normal   Scheduled  <unknown>        default-scheduler    Successfully assigned default/liveness-http to k8s-node01
  Normal   Pulled     17s              kubelet, k8s-node01  Container image "k8s.gcr.io/liveness" already present on machine
  Normal   Created    17s              kubelet, k8s-node01  Created container liveness
  Normal   Started    16s              kubelet, k8s-node01  Started container liveness
  Warning  Unhealthy  1s (x2 over 4s)  kubelet, k8s-node01  Liveness probe failed: HTTP probe failed with statuscode: 500

Defined TCP survival probe

A third type of livenessprobe is to use the TCP socket. By configuring, kubelet will try to establish a link in the socket designated port and container. If you can establish a link, the container was seen as healthy, if not then the vessel was seen as problematic.

Create a Pod. file name:tcp-liveness-readiness.yaml

apiVersion: v1
kind: Pod
metadata:
  name: goproxy
  labels:
    app: goproxy
spec:
  containers:
  - name: goproxy
    image: k8s.gcr.io/goproxy:0.1
    ports:
    - containerPort: 8080
    readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
    livenessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20

TCP and HTTP detect the detection arrangement is very similar. The following example also ready for use and survival detector. kubelet sends a first detection after the container is ready to start 5 seconds. This will attempt to port 8080 goproxy container. If the probe is successful, the Pod is marked ready state, kubelet will continue to run once every 10 seconds is detected.

In addition to readinessdetection, the configuration includes a livenessprobe. kubelet will for the first time starting 15 seconds after the container livenessprobe. Like readinessprobe as attempts to port 8080 goproxy container. If the probe fails to survive, this container will be restarted.

# kubectl apply -f /root/k8s-example/probe/tcp-liveness-readiness.yaml

15 seconds is detected by the detector to see event-free survival Pod:

# kubectl describe pod goproxy

Use named port

For HTTP or TCP ports can survive the container named detection.

ports:
- name: liveness-port
  containerPort: 8080
  hostPort: 8080

livenessProbe:
  httpGet:
    path: /healthz
    port: liveness-port

Slow Start startup detector using protective containers

Sometimes, there will be some existing applications at startup requires more initialization time. Or the impact of rapid response induced deadlock detection, in this case, set the livenessdetection parameters to skills. The trick is to use a command to set startupdetection for HTTP or TCP detection, you can set failureThreshold * periodSecondsparameters to ensure there is enough time to respond to a long start-up time in the worst case.

So, the previous example becomes:

ports:
- name: liveness-port
  containerPort: 8080
  hostPort: 8080

livenessProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 1
  periodSeconds: 10

startupProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  failureThreshold: 30
  periodSeconds: 10

Thanks startupdetection, the application will have the most 5 分钟(30 * 10 = 300s)time to complete its startup. Upon startupsuccessful detection time, survival missions will take over the detection of containers, container deadlock can respond quickly. If startupthe probe has not been successful, the vessel will kill after 300 seconds, and according restartPolicyto set Pod status.

Defined readlinessprobe

Sometimes, the application will be temporarily unable to provide communications services. For example, an application may need to load at startup large data or configuration files, or wait for the start to rely on external services. In this case, you do not want to kill the application, it did not want to send the request. Kubernetes provides ready detector to detect and mitigate these situations. Pod where the container is not ready to report information, and does not accept Kubernetes Servicetraffic.

Note: the detector is ready to keep running throughout the life cycle of container.

Ready detector configured detector configuration and survival is similar. The only difference is that you want to use readinessProbefields instead of l ivenessProbefield.

readinessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

HTTP and TCP readlinessprobe configuration and also livenessthe configuration of the same probe.

readlinessAnd livenessdetecting may be used in parallel in the same vessel. Both can be used to ensure that traffic is not sent yet prepared container and the container will be re-started when they fail.

Detector configuration

There are many detector configuration fields, these fields can be used to accurately control the survival and behavior of the ready detection:

• initialDelaySeconds: after how many seconds to wait to start container and survival was only ready detector initialization, the default is 0 seconds, the minimum value is 0.
• periodSeconds: detecting execution time interval (in seconds). The default is 10 seconds. The minimum value is 1.
• timeoutSeconds: after how many seconds to wait for timed-out probe. The default value is one second. The minimum value is 1.
• successThreshold: probe after a failure, is considered successful minimum number of consecutive successful. The default value is 1. Survival detection of this value must be 1. The minimum value is 1.
• failureThreshold: Pod started and when a failure is detected, the number of retries Kubernetes. Giving up would mean restarting the container under the probe survival situation. Abandon Pod will be marked with a label not ready ready in case detection. The default value is 3. The minimum value is 1.

HTTP detector may be httpGetarranged on additional fields:

• host: connect the host name, the default is the Pod IP. You may be provided "Host" in place of the HTTP header.
• scheme: a host connection mode setting (HTTP or HTTPS). The default is HTTP.
• path: the path to access the HTTP service.
• httpHeaders: custom HTTP request header. HTTP header field allows repeated.
• port: Port number or port name to access the container. If the numbers must be between 1 to 65,535.

For HTTP probe, kubelet send an HTTP request to the specified path detection is performed and the port. Unless httpGetthe hostfield is set, otherwise kubelet default is to send a probe to the IP address of the Pod. If the schemefield is set to HTTPS, kubelet skips sending HTTPS requests authentication certificate. In most cases, you do not need to set the hostfield. There is a need to set hosta field of a scene, it is assumed listener vessel 127.0.0.1, and the Pod hostNetworkfield is set to true. So httpGetthe hostfield should be set 127.0.0.1. May be more common situation is that if Pod dependent on the virtual host, you should not set the hostfield, but should be httpHeadersset up in the Host.

For a probe, kubelet on the node (not in the Pod inside) establish probe connection, which means you can not hostconfigure the parameters service name, because kubelet not be resolved service name.