k8s container probe
In Kubernetes, there are three types of probes (Probes) used to check the health of the container and determine whether the request should be routed to the container. These probes can be configured in the Pod's specification.
The following are the three probe types in Kubernetes:
- Liveness Probe : The liveness probe is used to determine whether the container is running. If the liveness probe fails (returns a failure status code or times out), Kubernetes will consider the container unhealthy and restart the container. This can be used to detect errors or deadlock situations within the application.
- Readiness Probe : The readiness probe is used to determine whether the container is ready to accept traffic. If a readiness probe fails, Kubernetes will remove the container from the service's load balancing until the probe succeeds. This can be used to ensure that the application completes necessary initialization or data loading after startup.
- Startup Probe : The startup probe is used to determine whether the container has been started and is ready to accept traffic. Unlike liveness and readiness probes, startup probes are used during the initial startup phase of a container and are only run once. The readiness probe and survival probe will not be started until the probe is successfully started.
For each probe type, the following parameters can be configured:
- Probe type : Select a liveness probe, a readiness probe, or a startup probe.
- Probe actions : Check the health status of the container through HTTP GET, TCP socket or executing commands.
- Probe Path : The URL path used for the HTTP GET probe.
- Probe Port : The port number used for the TCP socket probe.
- Probe commands : A list of commands used to execute command probes.
Liveness Probe :
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 15
periodSeconds: 10The above example configures a liveness probe for HTTP GET requests. It will wait 15 seconds after the container starts and execute every 10 seconds. The probe will send an HTTP GET request to the container /healthpath and use port number 8080
Readiness Probe :
readinessProbe:
httpGet:
path: /ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 3The above example configures a readiness probe for an HTTP GET request. It will wait 5 seconds after the container starts and execute every 3 seconds. The probe will send an HTTP GET request to the container's /readypath, using port number 8080.
Startup Probe :
startupProbe:
exec:
command:
- cat
- /tmp/initialized
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 30The above example configures a startup probe that executes a command. It will wait 10 seconds after the container starts and execute every 5 seconds. The probe will execute the command cat /tmp/initialized. If the command is executed successfully, it means that the container has been initialized.
Container detection health check configuration parameters
主机存活性探测和就绪性探测配置参数一模一样，只是定义的命令不一样
[root@k8s-master ~]# kubectl explain pod.spec.containers.livenessProbe
KIND: Pod
VERSION: v1
RESOURCE: livenessProbe <Object>
FIELDS:
exec <Object>
command <[]string> //执行的命令
httpGet <Object>
host <string> //主机地址，默认为pod的ip
path <string> //请求的url路径
port <string> //端口号
scheme <string> //使用的协议，默认为http
tcpSocket <Object>
host <string> //主机地址，默认为pod的ip
port <string> //要探测的端口号
initialDelaySeconds <integer> //容器启动后等待多少秒进行第一次探测
timeoutSeconds <integer> //探测超时时间，默认1秒，最小1秒
periodSeconds <integer> //执行探测的频率，默认10秒，最小1秒
failureThreshold <integer> //连续探测失败多少次才被认定为失败，默认是3，最小值1
successThreshold <integer> //连续探测成功多少次才被认定为成功，默认是1
例子：
apiVersion: v1
kind: Pod
metadata:
name: pod-livereadiness
namespace: dev
spec:
containers:
- name: nginx
image: nginx:1.18
ports:
- name: nginx-port
containerPort: 80
livenessProbe: //存活性探测
tcpSocket: //使用TCPSocket探测方式
port: 80 //探测的端口
initialDelaySeconds: 30 //容器启动后30s以后开始探测
timeoutSeconds: 5 //探测超时时间
readinessProbe: //就绪性探测
httpGet: //使用HTTPGet探测方式
scheme: HTTP //协议使用http
port: 80 //应用端口
path: / //url路径Pod resource survivability health check practice
Use exec detection method to detect the viability of the container
When using exec detection, if the status code returned by the executed command is 1, it means that the health check has failed. K8s will always restart the current pod. Only when the status code of the executed command returns 0, the health check will be successful and the pod will run normally.
1.编写使用exec探测的yaml文件
#exec配置时，执行一个退出状态码为1的命令，观察pod的状态
[root@k8s-master ~]# vim pod-liveness-exec.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-liveness-exec
namespace: dev
spec:
containers:
- name: nginx
image: nginx:1.18
ports:
- name: nginx-port
containerPort: 80
livenessProbe: #定义存活性探测
exec: #使用exec命令方式
command: ["/bin/cat","/tmp/hello.txt"] #执行具体命令：查看下/tmp/hello.txt文件，这个文件一定是不存在的，返回的状态码一定是1
2.创建pod
[root@k8s-master ~]# kubectl create -f pod-liveness-exec.yaml
pod/pod-liveness-exec created
3.查看pod的运行状态
#可以看到restarts重启次数已经是3次了
[root@k8s-master ~]# kubectl get pod pod-liveness-exec -n dev
NAME READY STATUS RESTARTS AGE
pod-liveness-exec 1/1 Running 3 2m47s
4.查看pod的详细输出
[root@k8s-master ~]# kubectl describe pod pod-liveness-exec -n dev
····················
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m43s default-scheduler Successfully assigned dev/pod-liveness-exec to k8s-node1
Normal Pulled 76s (x4 over 2m44s) kubelet Container image "nginx:1.15" already present on machine
Normal Created 76s (x4 over 2m44s) kubelet Created container nginx
Normal Killing 76s (x3 over 2m16s) kubelet Container nginx failed liveness probe, will be restarted
Normal Started 75s (x4 over 2m44s) kubelet Started container nginx
Warning Unhealthy 66s (x10 over 2m35s) kubelet Liveness probe failed: /bin/cat: /tmp/hello.txt: No such file or directory
#在最后的输出里可以看到说找不到/tmp/hello.txt这个文件，命令执行失败
#根据输出的信息可以看到nginx容器在启动之后就进行了健康检查，健康检查失败之后，容器就会被kill掉，然后尝试重启pod，然而健康检查一直失败，容器就会一直重启Adjust the command executed by exec so that the status code is 0 after execution, which meets the requirements of the health check and check the effect.
1.修改yaml文件调整exec执行的命令
[root@k8s-master ~l]# vim pod-liveness-exec.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-liveness-exec
namespace: dev
spec:
containers:
- name: nginx
image: nginx:1.18
ports:
- name: nginx-port
containerPort: 80
livenessProbe:
exec:
command: ["/bin/ls","/tmp"] #健康检查命令调整为查看/tmp下的文件，该命令执行后状态码为0，健康检查也会成功
2.创建pod
[root@k8s-master ~]# kubectl delete -f pod-liveness-exec.yaml
pod "pod-liveness-exec" deleted
[root@k8s-master ~]# kubectl create -f pod-liveness-exec.yaml
pod/pod-liveness-exec created
3.观察pod的状态，多敲几次命令观察
# 一般在容器启动后的30秒左右进行监控检查
[root@k8s-master ~]# kubectl get pod pod-liveness-exec -n dev
NAME READY STATUS RESTARTS AGE
pod-liveness-exec 1/1 Running 0 46s
4.查看详细信息验证是否报错
[root@k8s-master ~]# kubectl describe pod pod-liveness-exec -n dev
# 已经不报错了，nginx容器正常运行Use TCPSocket to detect container survival
The TCPSocket method is to establish a connection with the specified container port. If the connection can be successfully established, it means that the health check is successful, otherwise the container will be restarted.
1.编写TCPSocket方式的yaml文件
[root@k8s-master ~]# vim pod-liveness-tcpsocket.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-liveness-tcpsocket
namespace: dev
spec:
containers:
- name: nginx
image: nginx:1.18
ports:
- name: nginx-port
containerPort: 80
livenessProbe: #定义存活性探测
tcpSocket: #类型为TCPSocket
port: 8080 #探测的端口号为8080,8080在nginx容器里是不存在的，因此会健康检查失败，观察容器状态
2.创建pod
[root@k8s-master ~]# kubectl create -f pod-liveness-tcpsocket.yaml
pod/pod-liveness-tcpsocket created
3.查看容器状态
#可以看到已经重启2次了
[root@k8s-master ~]# kubectl get pod pod-liveness-tcpsocket -n dev
NAME READY STATUS RESTARTS AGE
pod-liveness-tcpsocket 1/1 Running 2 108s
4.查看详细输出
[root@k8s-master ~]# kubectl describe pod pod-liveness-tcpsocket -n dev
····················
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m30s default-scheduler Successfully assigned dev/pod-liveness-tcpsocket to k8s-node1
Normal Pulled 30s (x4 over 119s) kubelet Container image "nginx:1.15" already present on machine
Normal Created 30s (x4 over 119s) kubelet Created container nginx
Normal Started 30s (x4 over 119s) kubelet Started container nginx
Normal Killing 30s (x3 over 90s) kubelet Container nginx failed liveness probe, will be restarted
Warning Unhealthy 20s (x10 over 110s) kubelet Liveness probe failed: dial tcp 10.244.1.35:8080: connect: connection refused
# 可以在最后看到提示 808 0端口连接失败
# 根据输出的信息可以看到 nginx 容器在启动之后就进行了健康检查，健康检查失败之后，容器就会被 kill 掉，然后尝试重启 pod，然而健康检查一直失败，容器就会一直重启Adjust the port detected by TCPSocket to the correct port and observe the status of the pod
1.修改yaml文件中探测端口为80
[root@k8s-master ~]# vim pod-liveness-tcpsocket.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-liveness-tcpsocket
namespace: dev
spec:
containers:
- name: nginx
image: nginx:1.18
ports:
- name: nginx-port
containerPort: 80
livenessProbe:
tcpSocket:
port: 80 #80是nginx的端口，调整为80，健康检查就会成功
2.创建pod
[root@k8s-master ~]# kubectl delete -f pod-liveness-tcpsocket.yaml
pod "pod-liveness-tcpsocket" deleted
[root@k8s-master ~]# kubectl create -f pod-liveness-tcpsocket.yaml
pod/pod-liveness-tcpsocket created
3.观察pod状态
[root@k8s-master ~]# kubectl get pod pod-liveness-tcpsocket -n dev
NAME READY STATUS RESTARTS AGE
pod-liveness-tcpsocket 1/1 Running 0 35s
# pod 运行成功，已经不会重启了Use HTTPGet method to detect the viability of the container
The HTTPGet method is to access the specified url link. If the access is successful, the health check is successful. If the access fails, the container will be restarted.
1.编写yaml
[root@k8s-master ~]# vim pod-liveness-httpget.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-liveness-httpget
namespace: dev
spec:
containers:
- name: nginx
image: nginx:1.18
ports:
- name: nginx-port
containerPort: 80
livenessProbe: #定义存活性探测
httpGet: #定义httpGet探测方式，如果下面不知道host参数，默认是pod的ip，地址连起来就是：scheme://podip:port/path
scheme: HTTP #使用的应用协议，默认是http，可以是http、https
port: 80 #应用端口号
path: /hello #访问的url地址，地址连起来就是：http://127.0.0.1/hello，默认的nginx，hello页面一定不存在，健康检查也会失败
2.创建pod
[root@k8s-master ~]# kubectl create -f pod-liveness-httpget.yaml
pod/pod-liveness-httpget created
3.观察 pod 状态
#可以看到使用 httpGet 方式的健康检查，如果健康检查失败状态会变为 CrashLoopBackOff，且容器一直会重启
[root@k8s-master ~]# kubectl get pod pod-liveness-httpget -n dev
NAME READY STATUS RESTARTS AGE
pod-liveness-httpget 0/1 CrashLoopBackOff 5 3m56s
4.查看详细输出信息
[root@k8s-master ~]# kubectl describe pod pod-liveness-httpget -n dev
·····················
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m6s default-scheduler Successfully assigned dev/pod-liveness-httpget to k8s-node1
Normal Pulled 4m39s (x4 over 6m3s) kubelet Container image "nginx:1.15" already present on machine
Normal Created 4m39s (x4 over 6m3s) kubelet Created container nginx
Normal Started 4m39s (x4 over 6m2s) kubelet Started container nginx
Normal Killing 4m39s (x3 over 5m39s) kubelet Container nginx failed liveness probe, will be restarted
Warning Unhealthy 59s (x19 over 5m59s) kubelet Liveness probe failed: HTTP probe failed with statuscode: 404
# 可以看到提示，健康检查的页面访问 404，健康检查失败
# 根据输出的信息可以看到 nginx 容器在启动之后就进行了健康检查，健康检查失败之后，容器就会被kill掉，然后尝试重启pod，然而健康检查一直失败，容器就会一直重启Adjust the HTTPGet health check page to an accessible page and continue to observe the status of the pod.
1.修改httpGet请求的页面为真实存在的
[root@k8s-master ~]# vim pod-liveness-httpget.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-liveness-httpget
namespace: dev
spec:
containers:
- name: nginx
image: nginx:1.18
ports:
- name: nginx-port
containerPort: 80
livenessProbe:
httpGet:
scheme: HTTP
port: 80
path: / #调整url路径为可以访问的，这样一来健康检查就是成功的
2.创建pod
[root@k8s-master ~]# kubectl delete -f pod-liveness-httpget.yaml
pod "pod-liveness-httpget" deleted
[root@k8s-master ~]# kubectl create -f pod-liveness-httpget.yaml
pod/pod-liveness-httpget created
3.观察pod状态
[root@k8s-master ~]# kubectl get pod pod-liveness-httpget -n dev
NAME READY STATUS RESTARTS AGE
pod-liveness-httpget 1/1 Running 0 52s
# 发现 pod 已经不会重启了，pod 运行成功Pod resource readiness health check practice
- 设置一个容器存活性探测和就绪性探测共存的健康检查
- 存活探测使用tcpSocket方式，检查端口是否可用
- 就绪性探测使用httpGet方式，检查是否可用提供服务
1.编写yaml
[root@k8s-master ~]# vim pod-livereadiness.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-livereadiness
namespace: dev
spec:
containers:
- name: nginx
image: nginx:1.18
ports:
- name: nginx-port
containerPort: 80
livenessProbe: #定义存活性探测
tcpSocket: #使用tcpSocket方式
port: 80 #探测的端口
readinessProbe: #定义就绪性探测
httpGet: #使用httpGet方式
scheme: HTTP #使用的协议
port: 80 #服务端口
path: / #url路径
2.创建pod
[root@k8s-master ~]# kubectl create -f pod-livereadiness.yaml
pod/pod-livereadiness created
3.查看pod的运行状态
[root@k8s-master ~]# kubectl get pod pod-livereadiness -n dev
NAME READY STATUS RESTARTS AGE
pod-livereadiness 1/1 Running 0 2m5s
4.查看健康检查是否生效
[root@k8s-master ~]# kubectl describe pod pod-livereadiness -n dev | grep -A 2 -B2 'Liveness'
Ready: True
Restart Count: 0
Liveness: tcp-socket :80 delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:80/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>