- pod delete
Whenever you delete some Kubernetes resources such as namespace or pod, sometimes the resource status will be stuck in terminating and cannot be deleted for a long time. Sometimes even after adding the –force flag (forced deletion), it still cannot be deleted normally. At this time, you need to edit the resource and set the field finalizers to null, and then the Kubernetes resource will be deleted normally.
Sometimes it gets stuck when deleting a pod, the pod status changes to terminating, and the pod cannot be deleted.
(1) Forced deletion
kubectl delete pod xxx -n xxx --force --grace-period=0
(2) If forced deletion still does not work, set finalizers to empty
(If a container is already running, you need to modify some container attributes, but you do not want to delete the container, or it is inconvenient to update through replace. Kubernetes also provides a way to directly modify the container while the container is running. The method is the patch command.)
kubectl patch pod xxx -n xxx -p '{"metadata":{"finalizers":null}}'
This way the pod can be deleted.
Recommended content
- k8s deletion process
basic delete command status diagram:
Although this operation is simple, other factors may interfere with deletion, including finalizers and owner references.
The basic process of object deletion in K8s is as follows:
(1) The client submits a deletion request to the API Server (optional passing the GracePeriodSeconds parameter)
(2) The API Server performs a Graceful Deletion check (if the object implements
the RESTGracefulDeleteStrategy interface, the corresponding implementation will be called and returns whether Graceful deletion is required)
(3 ) API Server checks the Finalizers and determines whether to delete the object immediately based on whether it needs to be graceful deleted (if the object needs to be graceful deleted, update the
metadata.DeletionGracePeriodSecond and metadata.DeletionTimestamp fields, and do not delete the object from storage; if the object does not need to be graceful deleted) When deleting
: metadata.Finalizers is empty, delete directly. Metadata.Finalizers is not empty, do not delete, only update
metadata.DeletionTimestamp.
2.1 Introduction to finalizers
The Finalizers field belongs to the Kubernetes GC garbage collector. It is a deletion interception mechanism that allows the controller to implement asynchronous pre-delete (Pre-delete) callbacks. It exists in the Meta of any resource object, declared as []string in the k8s source code, and the content of the Slice is the name of the interceptor to be executed.
Common uses:
(1) The controller writes a custom string in metadata.finalizers when the object is first created
(2) APIServer will not delete the object when the metadata.finalizers array is not empty. This logic is hard and takes effect on all objects.
(3) When a client deletes an object, the controller can find that the object is in a deleted state, and then execute the corresponding pre-delete logic. After the execution is completed, the previously written custom string is removed.
(4) After all controllers have removed the strings written to metadata.finalizers. API Server will automatically delete the object.
Note: If the object also implements the graceful deletion policy, the deletion request needs to meet the condition GracefulPeriodSeconds = 0
2.2 Graceful Deletion
When APIServer processes a Pod deletion request, it will
comprehensively determine whether to perform Graceful deletion based on the pod.Spec.TerminationGracePeriodSeconds and deleteOptions.GracefulPeriodSeconds. If so, determine the final GracefulPeriodSeconds setting.
(1) When GracefulPeriodSeconds is set in the deletion request options, the options will prevail. If not, use
pod.Spec.TerminationGracePeriodSeconds. If pod.Spec.TerminationGracePeriodSeconds is not set, the default value of 0 is used.
(2) When the Pod is not scheduled or has ended (whether successful or failed). GracePeriodSeconds are all reset to 0.
GracePeriodSeconds of 0 means no graceful deletion. Non-zero means graceful deletion. The default graceful deletion time of Pod is 30 seconds, which is configured in the pod.Spec.TerminationGracePeriodSeconds field when the object is created
.
The purpose of graceful deletion is to give Kubelet a certain amount of time to gracefully exit the Pod. When a user performs a Delete operation on a Pod, the Pod object will not be deleted from the API Server immediately, but will only enter the Termination stage. Kubelet will send a TERM signal to the running Pod's container to notify it to exit. And execute the user-configured preStop hooks logic. After the graceful time has elapsed, start using the KILL signal to try to forcibly kill the container.
When kubelet cleans up the Pod, it will use the parameter GracefulPeriodSeconds==0 to perform the deletion operation.
Because the purpose of graceful deletion of Pod is to give Kubelet time to clean up resources, this is why during the GracePeriodSeconds setting stage, if the Pod has not been scheduled or has exited, immediate deletion can be directly allowed (GracePeriodSeconds = 0).
Note:
To understand this process, you can combine it with the Pod Termination description on the Kubernetes official website: https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
When the object performs a graceful deletion operation for the first time, the GracefulPeriodSeconds at that time will be Configured in the metadata.DeletionGracefulPeriodSeconds field.
Deletion operations are irreversible operations in API Server. metadata.DeletionTimestamp cannot be changed after it is set, and metadata.DeletionGracefulPeriodSeconds can only be decreased, not increased.
2.3 Reasons why objects cannot be deleted
After understanding the above mechanism, there are two reasons why objects cannot be deleted:
The object has finalizers, and the associated controller fails to execute or the finalizer function is stuck. For example, the namespace controller cannot delete all objects in the space. Especially when using aggregated apiserver, the third-party apiserver service failure prevents the deletion of its objects. At this time, it is necessary to restore the third-party apiserver service or remove the aggregation of the apiserver. The specific choice of the solution depends on the actual situation. The controller installed in the cluster added custom finalizers to some objects, and the controller was taken offline before deleting the fianlizers. As a result, these fianlizers did not have a controller to remove them. At this point, the controller needs to be restored and the finalizers will be manually removed. The specific option to choose depends on the actual situation.
The object needs to be deleted gracefully, but the executor cannot complete the deletion. For example, the Pod cannot be deleted because the kubelet cannot offline the node container and storage volume on the node. The more common reasons are as follows: kubelet cannot kill the process through the container runtime. For example, the process enters the D (Uninterruptible) state, the container runtime or the operating kernel encounters a bug, etc. As for the process entering the D state, if the faults that become D can be recovered, such as recovering the associated peripheral access, etc., the problem can be solved. If not, or because of the latter kernel bug, it is generally not possible to let the kubelet kill the process through the normal process. Generally, it is necessary to restart the operating system to solve the problem. The kubelet process stops or the node loses contact. In this case, no kubelet is running or the running kubelet has been disconnected from the apiserver, and the message that the pod needs to be deleted and offline cannot be received. So, there is no executor to perform graceful deletion. In this case, it is necessary to restore the operation of the node and kubelet.
Both mechanisms have forced skipping options:
Delete finalizers so that the associated logic does not need to be executed.
kubelet delete --force --grace-period 0 Delete directly.
However, this should only be selected when the associated cleanup work is no longer important or has been performed manually. Otherwise, it is easy to create data, status inconsistency, etc.