For Kubernetes clusters, it is not a simple task to ensure that daily interruptions do not affect the availability of applications. Kubernetes v1.26, released last month, added a new feature: allowing unhealthy Pod eviction policies to be specified for PodDisruptionBudget (PDB) , which helps ensure application availability during node management operations.

PodDisruptionBudget (PDB)：https://kubernetes.io/zh-cn/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets

What problem does it solve?

Pod evictions initiated by the API are subject to PodDisruptionBudget (PDB) constraints. This means that voluntary interruptions requested by pod evictions should not interfere with daemonized applications, and the PDB's .status.currentHealthy should not be lower than .status.desiredHealthy. If the status of a running pod is Unhealthy, the pod is not counted in the PDB status, and these pods can only be evicted when the application is not disturbed. This helps to ensure the availability of interrupted or inactive applications as much as possible, ensuring no additional downtime due to eviction.

But this mechanism is problematic for cluster administrators who want to empty nodes without any manual intervention. This task can be further complicated if some applications misbehave because pods are in CrashLoopBackOff state (due to bugs or misconfigurations) or pods fail to go into ready state. This is because when all Pods for an application are unhealthy, all eviction requests fail due to PDB violations. Clearing the node will not have any effect in this case.

On the other hand, some users often rely on existing behavior to:

Prevent data loss caused by deleting Pods guarding basic resources or storage;

Get the best usability from your app.

Kubernetes 1.26 introduces a new experimental field for the PodDisruptionBudget API: .spec.unhealthyPodEvictionPolicy . Enabling this field will allow you to do both.

02 working principle

The eviction initiated by the API is a process that triggers the graceful termination of the Pod. This process can be initiated by calling the API directly, or using kubectl drain or other principals in the cluster. During this process, the removal of each Pod will be negotiated with the corresponding PDB to ensure that there are always enough Pods running in the cluster.

The eviction policy allows the PDB creator to further control how unhealthy Pods are handled. There are two eviction policies to choose from: IfHealthyBudget and AlwaysAllow.

The former IfHealthyBudget uses the existing behavior to achieve the best usability you can get by default. An unhealthy Pod will only be disturbed when the number of available Pods in its application reaches .status.desiredHealthy, which is the minimum available number.

By setting the PDB's spec.unhealthyPodEvictionPolicy field to AlwaysAllow, it is possible to indicate that the best possible availability is selected for the application. When adopting this strategy, it is always possible to evict unhealthy pods. This simplifies cluster maintenance and upgrades.

We think AlwaysAllow is generally a better choice, but for some critical workloads, you may still prefer to prevent unhealthy pods from being emptied from nodes, or other forms of API-initiated evictions.

specific usage

This is an Alpha feature, meaning you must enable the PDBUnhealthyPodEvictionPolicy feature gate for kube-apiserver with the command line parameter --feature-gates=PDBUnhealthyPodEvictionPolicy=true .

Feature Gates: https://kubernetes.io/zh-cn/docs/reference/command-line-tools-reference/feature-gates/

Here is an example. Assumptions:

This feature gate has been enabled in the cluster and a Deployment running a normal web server has been defined;

The Pod of the Deployment has been tagged app: nginx;

want to limit avoidable distractions, and you know that best-effort availability is adequate for this application;

You decide to allow eviction even if those web server pods are unhealthy;

You create a PDB to guard the application, and use the AlwaysAllow policy to evict unhealthy Pods.

The specific YAML is as follows:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nginx-pdb
spec:
  selector:
    matchLabels:
      app: nginx
  maxUnavailable: 1
  unhealthyPodEvictionPolicy: AlwaysAllow