Troubleshooting Kubernetes Applications—Debugging Pods

1. Debug Pod

The first step in debugging a Pod is to view Pod information. View the current status and recent events of a Pod with the following command:

kubectl describe pods ${POD_NAME}

Take a look at the state of the containers in the Pod. Is the status of these containers all the  Running same? Have you rebooted recently?

Subsequent debugging depends on the status of the Pod.

Pod stuck in Pending state

If a pod is stuck in  Pending state, it means that the pod is not scheduled on the node. Usually this is due to insufficient resources of some type preventing scheduling. Looking at the output of the command above  kubectl describe ... , it should show why it wasn't scheduled. Common reasons are as follows:

• Insufficient resources : You may have exhausted all CPU or memory on the cluster. At this point, you need to delete pods, adjust resource requests, or add nodes to the cluster.

• Used hostPort : If the Pod is bound to  hostPort, the nodes that can run the Pod are limited. In most cases, hostPort it is not necessary, but the Service object should be used to expose the Pod. If you really need to use it  hostPort, then the number of nodes in the cluster is the upper limit of the number of Pods that can be created.

Pod stuck in Waiting state

If the pod is stuck in  Waiting state, it means that the pod has been scheduled to a worker node, but cannot run on that node. Also, kubectl describe ... the output of the command may be useful. Waiting The most common reason for the status is that the image pull failed. There are three areas to check:

• Make sure the image name is spelled correctly.
• Make sure the image has been pushed to the registry.
• Try to see if you can pull the image manually. For example, if you use Docker on your PC, run  docker pull <镜像>.

Pod is Crashing or otherwise unhealthy

Once a Pod is scheduled, it can be further debugged as described in Debugging a Running Pod.

Pod is in Running state but not working properly

If the Pod is not behaving as expected, it is likely that  mypod.yamlthere is a problem in the Pod description (eg on your local machine), and the error was ignored when the Pod was created without reporting an error. Usually, the nesting relationship of the section area in the Pod definition is wrong, and the misspelling of the field name will cause the corresponding content to be ignored. For example, if you mistakenly  command write  commnd, the Pod will be created, but it will not execute the command line you expect it to execute.

The first thing that can be done is to delete your pod and try to  --validate recreate with option. For example, run  kubectl apply --validate -f mypod.yaml. If  command it is misspelled  commnd, you will see the following error message:

I0805 10:43:25.129850   46757 schema.go:126] unknown field: commnd
I0805 10:43:25.129973   46757 schema.go:129] this may be a false alarm, see https://github.com/kubernetes/kubernetes/issues/6842
pods/mypod

The next thing to check is that the Pod on the API server matches what you expect to create (for example, you originally used a YAML file on your local machine to create the Pod). For example, after running  kubectl get pods/mypod -o yaml > mypod-on-apiserver.yaml, manually compare  mypod.yaml with the Pod description retrieved from the API server. It is normal for the YAML obtained from the API server to contain lines that do not exist in the YAML used to create the Pod. However, if some lines in the source file do not exist in the API server version, it means that the Pod specification is problematic.

Debug Replica Controller

The replica controller is relatively simple and straightforward. Either they can create pods, or they can't. If the pod cannot be created, see the instructions above to debug the pod.

You can also use  kubectl describe rc ${CONTROLLER_NAME} commands to view events related to the replica controller.

Debug Service

Services support load balancing across multiple Pods. There are some common problems that can prevent the service from working properly. The following instructions will help in debugging problems with the service.

First, verify that the service has an endpoint. For each Service object, the API server provides corresponding  endpoints resources.

The endpoints resource can be viewed by the following command:

kubectl get endpoints ${SERVICE_NAME}

Make sure that the number of Endpoints is consistent with the number of service member Pods. For example, if your Service is used to run 3 replicas of nginx containers, you should see 3 different IP addresses in the Service's Endpoints.

Service missing Endpoints

If you don't have Endpoints, try listing Pods with the labels used by the Service. Suppose your service contains the following tag selector:

...
spec:
  - selector:
     name: nginx
     type: frontend

You can use the following command to list the Pods that match the selector and verify that they belong to the created service:

kubectl get pods --selector=name=nginx,type=frontend

Verify that the Pod's   matches containerPort the service's  .targetPort