Kubernetes management experience

Recommended Tools

kubectx

kubectx: used to access the switch cluster

kubens: to switch the default namespace

kubectl-aliases

kubectlCommand aliases

Cluster management-related commands

kubectl get cs

# 查看节点
kubectl get nodes

kubectl get ing pdd --n java
# 不调度
kubectl taint nodes node1 key=value:NoSchedule
kubectl cluster-info dump


kubectl get svc --sort-by=.metadata.creationTimestamp
kubectl get no --sort-by=.metadata.creationTimestamp
kubectl get po --field-selector spec.nodeName=xxxx
kubectl get events  --field-selector involvedObject.kind=Service --sort-by='.metadata.creationTimestamp'

Reference links:

1. kubernetes node maintains cordon, drain, uncordon

• Application management-related

kubectl top pod
kubectl delete deployment,services -l app=nginx 
kubectl scale deployment/nginx-deployment --replicas=2
kubectl get svc --all-namespaces=true

• Forced to delete

There is a problem, use this command parameters have to add two sometimes delete pv / pvc--grace-period=0 --force

• Delete all failed pod

  kubectl get po --all-namespaces --field-selector 'status.phase==Failed'
  kubectl delete po  --field-selector 'status.phase==Failed'

• Some tips

k8s Currently there is no similar docker-compose the depends_ondependent startup mechanism recommended wait-for-it rewrite command mirror.

Cluster Management through (teaching) experience (training)

Node problem

taint with Freeze

kubectl taint nodes xx  elasticsearch-test-ready=true:NoSchedule
kubectl taint nodes xx  elasticsearch-test-ready:NoSchedule-

master node itself comes taint, so will lead us to release the container will not run in the master node above. However, if a custom taint, then we should pay attention! all DaemonSetand kube-system, will need to bring appropriate tolerations. Otherwise, the node All will be deported without the tolerationscontainer, even including network plug-ins, kube-proxy, very serious consequences, please note

taintWith tolerationsa pair corresponding to the present, the operator can not be used indiscriminately

NoExecute

      tolerations:
        - key: "elasticsearch-exclusive"
          operator: "Equal"
          value: "true"
          effect: "NoExecute"

kubectl taint node cn-shenzhen.xxxx elasticsearch-exclusive=true:NoExecute

NoExecute is not satisfied pod immediately deported to tolerate conditions, the operation is very dangerous, you should make sure there is a corresponding system components configuration tolerations.

Special attention to Existsthis operator is invalid, it must beEqual

NoSchedule

      tolerations:
        - key: "elasticsearch-exclusive"
          operator: "Exists"
          effect: "NoSchedule"
        - key: "elasticsearch-exclusive"
          operator: "Equal"
          value: "true"
          effect: "NoExecute"

kubectl taint node cn-shenzhen.xxxx elasticsearch-exclusive=true:NoSchedule

It is to try not to go above this dispatch, but in fact there was still a pod ran up there

ExistsAnd Existsfree to use, not very much influence

It is worth mentioning that the same key can be multiple simultaneous effect

Taints:             elasticsearch-exclusive=true:NoExecute
                    elasticsearch-exclusive=true:NoSchedule

Other reference links:

1. Taint and toleration of Kubernetes (stains and tolerance)
2. kubernetes scheduling mechanism

The step of isolating the correct node

# 驱逐除了ds以外所有的pod
kubectl drain <node name>   --ignore-daemonsets
kubectl cordon <node name>

This time get node to run commands, status becomes

node.xx   Ready,SchedulingDisabled   <none>   189d   v1.11.5

At last

kubectl delete <node name>

Maintain proper step node

kubectl drain <node name> --ignore-daemonsets
kubectl uncordon <node name>

A disk node pressure (DiskPressure)

--eviction-hard=imagefs.available<15%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5%

kubelet startup disk specified pressure to Ali cloud, for example, imagefs.available<15%means that the read-write layer container of less than 15% of the time, the node will be evicted. node expelled consequence is produced DiskPressure this situation, and the node can no longer run any mirroring, disk until the problem is resolved. If the node using the container home directory, this problem would be fatal, because you can not delete the directory, but really these directories host of accumulation, leading to a node They were expelled.

So, usually good to keep a good habit to write something Bite your tongue inside the container (vessel which is written files can take ephemeral-storage, ephemeral-storage pod will be expelled too much), and more use of non-state-type container, careful selection of storage, as far as possible do not use this storage hostpath

When the condition occurs, really a kind of feeling of tears.

Events:
  Type     Reason                 Age                   From                                            Message
  ----     ------                 ----                  ----                                            -------
  Warning  FreeDiskSpaceFailed    23m                   kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 5182058496 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed    18m                   kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 6089891840 bytes, but freed 0 bytes
  Warning  ImageGCFailed          18m                   kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 6089891840 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed    13m                   kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 4953321472 bytes, but freed 0 bytes
  Warning  ImageGCFailed          13m                   kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 4953321472 bytes, but freed 0 bytes
  Normal   NodeHasNoDiskPressure  10m (x5 over 47d)     kubelet, node.xxxx1     Node node.xxxx1 status is now: NodeHasNoDiskPressure
  Normal   Starting               10m                   kube-proxy, node.xxxx1  Starting kube-proxy.
  Normal   NodeHasDiskPressure    10m (x4 over 42m)     kubelet, node.xxxx1     Node node.xxxx1 status is now: NodeHasDiskPressure
  Warning  EvictionThresholdMet   8m29s (x19 over 42m)  kubelet, node.xxxx1     Attempting to reclaim ephemeral-storage
  Warning  ImageGCFailed          3m4s                  kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 4920913920 bytes, but freed 0 bytes

Reference links:

1. Eviction Signals
2. Figure 10 takes you deep understanding Docker container and mirror

Biao high CPU node

There may be a node during GC (container GC / image GC) , with a describe nodelook. I once encountered such a situation, a lot less container on the last node, but also a bit depressing

Events:
  Type     Reason                 Age                 From                                         Message
  ----     ------                 ----                ----
  Warning  ImageGCFailed          45m                 kubelet, cn-shenzhen.xxxx  failed to get image stats: rpc error: code = DeadlineExceeded desc = context deadline exceeded

reference:

kubelet source code analysis: Garbage Collect

The Object

under

pod frequently restart

There are a variety of reasons, can not be generalized

Resource reach limit set value

Increase the limit or check application

Readiness/Liveness connection refused

Readiness check failure will restart, but Readinesscheck fails is not necessarily a question of the application, if the node itself overloaded, there will also timeout or connection refused

To troubleshoot this problem on a node

pod expulsion (Evicted)

1. Node plus a stain lead pod expulsion

2. ephemeral-storage exceeds the limit deported

   1. EmptyDir usage exceeds his SizeLimit, then this pod will be deported
   2. Container usage (log, if there is no overlay zoning, include imagefs) exceed his limit, then the pod will be expelled
   3. Pod local temporary storage for total usage (all emptydir and container) over the pod in the limit of all container and then expelled pod

ephemeral-storage is used for temporarily storing a pod.

resources:
       requests: 
           ephemeral-storage: "2Gi"
       limits:
           ephemeral-storage: "3Gi"

After being expelled node can still see through get po, with the describe command, you can see the historical reasons expelled

Message: The node was low on resource: ephemeral-storage. Container codis-proxy was using 10619440Ki, which exceeds its request of 0.

reference:

1. Kubernetes pod ephemeral-storage configuration
2. Managing Compute Resources for Containers

kubectl exec failed to enter the container

I encountered this problem when building codis-server, when ready, and no configuration health check. But when acquiring pod description, display running. In fact, this time the container and not normal.

~ kex codis-server-3 sh
rpc error: code = 2 desc = containerd: container not found
command terminated with exit code 126

Solution: delete this pod, configurationlivenessProbe

pod of virtual host name

DeploymentDerived pod, virtual host namethat is pod name.

StatefulSetDerived pod, virtual host nameShi <pod name>.<svc name>.<namespace>.svc.cluster.localcompared to Deploymentappear more regularly few. And support other pod access

pod after another Crashbackoff

CrashbackoffThere are a variety of reasons.

Create a sandbox (FailedCreateSandBox) failed, mostly network plug problem cni

Mirroring pull, there is a problem of socialism with Chinese characteristics, may be too large, slow pull

There is also a container may be complicated by too high, resulting in an avalanche flow.

For example, there are three containers abc, a suddenly encountered traffic peak rush led to internal collapse, in turn Crashbackoff, then a will be serviceremoved out, the rest of the bc can not carry so much traffic, after another crash, and ultimately the site is not accessible. This case, more common in high concurrency website + inefficient web container.

In the case where the code does not change, the optimal solution is to increase the number of copies, and adds HPA, dynamic stretching capacity.

deploy

MinimumReplicationUnavailable

If deploythe configuration of the SecurityContext, but rejected api-server, this will be the case, the inside of the container api-server remove SecurityContextDenythe boot parameters.

See in particular Using Admission Controllers

service

We built a service, but there is no corresponding po, what happens?

When the request has been no response, until the request timeout

reference

1. Configure Out Of Resource Handling

service connection refuse

There may be reasons

1. pod is not provided readinessProbe, not ready to request pod
2. kube-proxy is down (kube-proxy is responsible for forwarding the request)
3. Network overload

no load balancing service

Check whether the use headless service. headless serviceIs not automatically load balancing ...

kind: Service
spec:
# clusterIP: None的即为`headless service`
  type: ClusterIP
  clusterIP: None

Specific performance service without their own virtual IP, nslookup ip will be all the pod, but only when the ping ip first pod appears

/ # nslookup consul
nslookup: can't resolve '(null)': Name does not resolve

Name:      consul
Address 1: 172.31.10.94 172-31-10-94.consul.default.svc.cluster.local
Address 2: 172.31.10.95 172-31-10-95.consul.default.svc.cluster.local
Address 3: 172.31.11.176 172-31-11-176.consul.default.svc.cluster.local

/ # ping consul
PING consul (172.31.10.94): 56 data bytes
64 bytes from 172.31.10.94: seq=0 ttl=62 time=0.973 ms
64 bytes from 172.31.10.94: seq=1 ttl=62 time=0.170 ms
^C
--- consul ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.170/0.571/0.973 ms

/ # ping consul
PING consul (172.31.10.94): 56 data bytes
64 bytes from 172.31.10.94: seq=0 ttl=62 time=0.206 ms
64 bytes from 172.31.10.94: seq=1 ttl=62 time=0.178 ms
^C
--- consul ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.178/0.192/0.206 ms

Ordinary type: ClusterIP service, nslookup will own the IP service

/ # nslookup consul
nslookup: can't resolve '(null)': Name does not resolve

Name:      consul
Address 1: 172.30.15.52 consul.default.svc.cluster.local

ReplicationController not updated

ReplicationController not used to apply updates, but kubectl rolling-update, but this directive is also abolished, replaced by a kubectl rolloutso should be used kubectl rolloutas a means of updating or a little lazy, then apply file, delete po.

Try to use deploy it.

StatefulSet update failed

StatefulSet is updated one by one, look at whether there is Crashbackoffa container, the container may be the cause of this update stuck, it can be deleted.

Advanced scheduling

Ensure the use of affinity node running on the target node

        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: elasticsearch-test-ready
                operator: Exists

Reference links:

1. advanced-scheduling-in-kubernetes
2. kubernetes-scheulder-Affinity

Use anti-affinity ensures that each node only run the same application

      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: 'app'
                operator: In
                values:
                - nginx-test2
            topologyKey: "kubernetes.io/hostname"
            namespaces: 
            - test

Tolerated run

The reason why the master node does not allow general image, because the master node with a stain, if desired forced to run above the master image, it is necessary to tolerate the appropriate stain.

      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
        - effect: NoSchedule
          key: node.cloudprovider.kubernetes.io/uninitialized
          operator: Exists

Ali cloud the issue Kubernetes

Modify the default ingress

Create a new type svc ingress points to load balance, and then change it kube-systemat nginx-ingress-controllerstartup parameters.

        - args:
            - /nginx-ingress-controller
            - '--configmap=$(POD_NAMESPACE)/nginx-configuration'
            - '--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services'
            - '--udp-services-configmap=$(POD_NAMESPACE)/udp-services'
            - '--annotations-prefix=nginx.ingress.kubernetes.io'
            - '--publish-service=$(POD_NAMESPACE)/<自定义svc>'
            - '--v=2'

LoadBalancer IP service has not been

Specific performance is EXTERNAL-IP has been displayed pending.

~ kg svc consul-web
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
consul-web   LoadBalancer   172.30.13.122   <pending>     443:32082/TCP   5m  

This problem with Alibaba Cloud Provider This component related, cloud-controller-managerthere are three components, they need an internal election primary, where can go wrong, and then I put a problem of poddeleted, just fine.

Cleanup Statefulset dynamic PVC

At present, Ali cloud Statefulsetdynamic use of PVC is nas.

1. For this store, you need to first copy the vessel will be zero, or the entire Statefulsetdeleted.
2. Delete PVC
3. The nas mount to any server above, and then delete the directory pvc nas correspondence.

After upgrading to v1.12.6-aliyun.1 fewer allocated memory node

The version of each node reserved 1Gi, equivalent to less entire cluster N GB (N is the number of nodes) for dispensing Pod.

If the node is a 4G, Pod request 3G, extremely easy to be deported.

It proposed to increase the node specifications.

Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-aliyun.1", GitCommit:"8cb561c", GitTreeState:"", BuildDate:"2019-04-22T11:34:20Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Add new nodes appear NetworkUnavailable

RouteController failed to create a route

Look kubernetes events, if there is a

timed out waiting for the condition -> WaitCreate: ceate route for table vtb-wz9cpnsbt11hlelpoq2zh error, Aliyun API Error: RequestId: 7006BF4E-000B-4E12-89F2-F0149D6688E4 Status Code: 400 Code: QuotaExceeded Message: Route entry quota exceeded in this route table  

This problem occurs because the reach of the custom route VPC entry restrictions , the default is 48, we need to raise vpc_quota_route_entrys_numquotas

Reference (scheduling related application):

1. Kubernetes of health examination and treatment services rely
2. How kubernetes resolution services depend on it?
3. Kubernetes Road 1 - Java application resource constraints myth
4. Control CPU Management Policies on the Node
5. Reserve Compute Resources for System Daemons
6. Configure Out Of Resource Handling