kubernetes problems (1)-Exploring the reasons and solutions for pod eviction

1 What is k8s evicted?

k8s evicted is a component in Kubernetes, mainly used to handle Pod eviction situations. In Kubernetes, when Node node resources are insufficient, in order to ensure the stable operation of the entire cluster, the Pods will be evicted according to certain priorities and policies. At this time, a component is needed to handle these evicted Pods and give them the opportunity to be rescheduled to other available Node nodes. And this component is k8s evicted.

2 reasons for k8s evicted

After analysis, the main reasons for pod eviction are as follows:

2.1 Insufficient resources

When the resources of a Node node are exhausted and can no longer be allocated to new Pods, Kubernetes will trigger the Pod eviction mechanism and delete some running Pods. This mechanism will first delete Pods with low priority based on the Pod's QoS (Quality of Service) policy. In this case, you can solve the problem of Pod eviction by horizontally expanding the cluster and increasing the number of Node nodes, or by vertically expanding the cluster and increasing the resources of Node nodes.

2.2 Node failure

A Node node may not work properly due to hardware failure, operating system error, etc. At this time, Kubernetes will consider that the node has been removed from the cluster and will no longer schedule the Pods on it. If there are Pods running on the node, these Pods may be forcibly evicted. The solution to this situation is to check the node's hardware, network, operating system, etc. to ensure that it is functioning properly.

2.3 Reasons for the Pod itself

In some cases, a Pod may be evicted for its own reasons. For example, when a serious error occurs when the Pod is running or the running time exceeds the maximum time limit set by Kubernetes, etc. At this time, the Pod will be marked as Failed and evicted. This requires monitoring and debugging the Pod to ensure its normal operation.

3 solutions to k8s evicted

In order to avoid Pod being evicted, you can use the following methods:

3.1 Appropriately set the resource requirements of Pod

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
    resources:
      requests:
        cpu: 200m
        memory: 200Mi
      limits:
        cpu: 500m
        memory: 500Mi

By setting the resource requirements of Pods, Kubernetes can better schedule and avoid Pods being evicted due to insufficient resources. It should be noted that when setting resource requirements, adjustments should be made based on actual conditions to avoid waste or insufficiency.

3.2 Perform a health check on the Pod

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: nginx:latest
    readinessProbe:
      httpGet:
        path: /
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10

By setting the health check of the Pod, Kubernetes can better monitor and maintain the running status of the Pod. Once an exception occurs, it can be handled in a timely manner to avoid the Pod being evicted. It should be noted that when setting up health checks, adjustments should be made according to the actual situation to avoid false negatives or false positives.

3.3 Optimize Node node resources

If the Node node resources in the cluster are insufficient, we can choose to expand horizontally or vertically. Horizontal expansion refers to increasing the number of Node nodes to share the load of the cluster; vertical expansion refers to increasing the resources of Node nodes to improve the performance of the cluster. Both methods can solve the problem of insufficient cluster resources, and you need to choose according to the actual situation.