K8s common troubleshooting and solutions

1. After frequently restarting nodes in a k8s cluster, the following problems may occur

1. When checking the pod information, I found an error: open /run/flannel/subnet.env: no such file or directory

Check whether each node, including the master node, has /run/flannel/subnet.env. The content should be similar to the following:

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

If there is a node that does not have this file, make a copy and deploy it again, and this error should not be reported.

2. Deploy the service and find that the service cannot be exposed and calls between services are blocked.

First execute ifconfig to check the network status of the next node.
Insert image description here
You will find that there are two gateways. If there is a problem with the network, you can first check whether the two network segments are consistent.

When deploying K8s, I use the flannel network plug-in or the calico plug-in. However, there will be frequent restarts or the problem that the service cannot be discovered after deploying the service after a long running time. You can try to solve it in the following ways.

建议:在生产场景中,建议提前规划好k8s的网络,如果中途更换网络插件,产生的影响会比较大。

Redeploy the network plug-in. It is recommended not to replace it. You can still use the previous plug-in. You only need to redeploy it.

The first step is to delete flannel on the master node.
kubectl delete -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
The second step is to clean up the files left by the flannel network on the node node.
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/
rm -f /etc/cni/net.d/*

After performing the above operations, restart kubelet , and then restart all previously deployed applications. You will find that the service can be discovered. It's also a little trick. Anyway, my problem was solved.

Guess you like

Origin blog.csdn.net/bacawa/article/details/129122366