The pod has been in the ContainerCreating state, and kubectl describe pod sees the error Failed create pod sandbox
journalctl -u kubelet -n 1000
Check the kubelet log and find
Apr 27 16:14:00 tfl1st.prod.yiran.com kubelet[34090]: E0427 16:14:00.327761 34090 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "openmpi-worker-1_default(45008fb2-49ef-11e8-9e08-525400b26185)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 64.233.189.82:443: getsockopt: connection timed out
Apr 27 16:14:00 tfl1st.prod.yiran.com kubelet[34090]: E0427 16:14:00.327774 34090 kuberuntime_manager.go:647] createPodSandbox for pod "openmpi-worker-1_default(45008fb2-49ef-11e8-9e08-525400b26185)" failed: rpc error: code = Unknown desc = failed pulling image "gcr.io/google_containers/pause-amd64:3.0": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 64.233.189.82:443: getsockopt: connection timed out
Apr 27 16:14:00 tfl1st.prod.yiran.com kubelet[34090]: E0427 16:14:00.327835 34090 pod_workers.go:186] Error syncing pod 45008fb2-49ef-11e8-9e08-525400b26185 ("openmpi-worker-1_default(45008fb2-49ef-11e8-9e08-525400b26185)"), skipping: failed to "CreatePodSandbox" for "openmpi-worker-1_default(45008fb2-49ef-11e8-9e08-525400b26185)" with CreatePodSandboxError: "CreatePodSandbox for pod \"openmpi-worker-1_default(45008fb2-49ef-11e8-9e08-525400b26185)\" failed: rpc error: code = Unknown desc = failed pulling image \"gcr.io/google_containers/pause-amd64:3.0\": Error response from daemon: Get https://gcr.io/v1/_ping: dial tcp 64.233.189.82:443: getsockopt: connection timed out"
Execute docker images | grep pause-amd64 and find that there is no such image. Pull the image, restart kubelet, and that's it.