Recommended Tools
kubectx
kubectx: used to access the switch cluster
kubens: to switch the default namespace
kubectl-aliases
kubectl
Command aliases
Cluster management-related commands
kubectl get cs
# 查看节点
kubectl get nodes
kubectl get ing pdd --n java
# 不调度
kubectl taint nodes node1 key=value:NoSchedule
kubectl cluster-info dump
kubectl get svc --sort-by=.metadata.creationTimestamp
kubectl get no --sort-by=.metadata.creationTimestamp
kubectl get po --field-selector spec.nodeName=xxxx
kubectl get events --field-selector involvedObject.kind=Service --sort-by='.metadata.creationTimestamp'
Reference links:
- Application management-related
kubectl top pod
kubectl delete deployment,services -l app=nginx
kubectl scale deployment/nginx-deployment --replicas=2
kubectl get svc --all-namespaces=true
- Forced to delete
There is a problem, use this command parameters have to add two sometimes delete pv / pvc--grace-period=0 --force
- Delete all failed pod
kubectl get po --all-namespaces --field-selector 'status.phase==Failed'
kubectl delete po --field-selector 'status.phase==Failed'
- Some tips
k8s Currently there is no similar docker-compose the depends_on
dependent startup mechanism recommended wait-for-it rewrite command mirror.
Cluster Management through (teaching) experience (training)
Node problem
taint with Freeze
kubectl taint nodes xx elasticsearch-test-ready=true:NoSchedule
kubectl taint nodes xx elasticsearch-test-ready:NoSchedule-
master node itself comes taint, so will lead us to release the container will not run in the master node above. However, if a custom taint
, then we should pay attention! all DaemonSet
and kube-system, will need to bring appropriate tolerations
. Otherwise, the node All will be deported without the tolerations
container, even including network plug-ins, kube-proxy, very serious consequences, please note
taint
With tolerations
a pair corresponding to the present, the operator can not be used indiscriminately
NoExecute
tolerations:
- key: "elasticsearch-exclusive"
operator: "Equal"
value: "true"
effect: "NoExecute"
kubectl taint node cn-shenzhen.xxxx elasticsearch-exclusive=true:NoExecute
NoExecute is not satisfied pod immediately deported to tolerate conditions, the operation is very dangerous, you should make sure there is a corresponding system components configuration tolerations.
Special attention to Exists
this operator is invalid, it must beEqual
NoSchedule
tolerations:
- key: "elasticsearch-exclusive"
operator: "Exists"
effect: "NoSchedule"
- key: "elasticsearch-exclusive"
operator: "Equal"
value: "true"
effect: "NoExecute"
kubectl taint node cn-shenzhen.xxxx elasticsearch-exclusive=true:NoSchedule
It is to try not to go above this dispatch, but in fact there was still a pod ran up there
Exists
And Exists
free to use, not very much influence
It is worth mentioning that the same key can be multiple simultaneous effect
Taints: elasticsearch-exclusive=true:NoExecute
elasticsearch-exclusive=true:NoSchedule
Other reference links:
The step of isolating the correct node
# 驱逐除了ds以外所有的pod
kubectl drain <node name> --ignore-daemonsets
kubectl cordon <node name>
This time get node to run commands, status becomes
node.xx Ready,SchedulingDisabled <none> 189d v1.11.5
At last
kubectl delete <node name>
Maintain proper step node
kubectl drain <node name> --ignore-daemonsets
kubectl uncordon <node name>
A disk node pressure (DiskPressure)
--eviction-hard=imagefs.available<15%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5%
kubelet startup disk specified pressure to Ali cloud, for example, imagefs.available<15%
means that the read-write layer container of less than 15% of the time, the node will be evicted. node expelled consequence is produced DiskPressure this situation, and the node can no longer run any mirroring, disk until the problem is resolved. If the node using the container home directory, this problem would be fatal, because you can not delete the directory, but really these directories host of accumulation, leading to a node They were expelled.
So, usually good to keep a good habit to write something Bite your tongue inside the container (vessel which is written files can take ephemeral-storage, ephemeral-storage pod will be expelled too much), and more use of non-state-type container, careful selection of storage, as far as possible do not use this storage hostpath
When the condition occurs, really a kind of feeling of tears.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FreeDiskSpaceFailed 23m kubelet, node.xxxx1 failed to garbage collect required amount of images. Wanted to free 5182058496 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 18m kubelet, node.xxxx1 failed to garbage collect required amount of images. Wanted to free 6089891840 bytes, but freed 0 bytes
Warning ImageGCFailed 18m kubelet, node.xxxx1 failed to garbage collect required amount of images. Wanted to free 6089891840 bytes, but freed 0 bytes
Warning FreeDiskSpaceFailed 13m kubelet, node.xxxx1 failed to garbage collect required amount of images. Wanted to free 4953321472 bytes, but freed 0 bytes
Warning ImageGCFailed 13m kubelet, node.xxxx1 failed to garbage collect required amount of images. Wanted to free 4953321472 bytes, but freed 0 bytes
Normal NodeHasNoDiskPressure 10m (x5 over 47d) kubelet, node.xxxx1 Node node.xxxx1 status is now: NodeHasNoDiskPressure
Normal Starting 10m kube-proxy, node.xxxx1 Starting kube-proxy.
Normal NodeHasDiskPressure 10m (x4 over 42m) kubelet, node.xxxx1 Node node.xxxx1 status is now: NodeHasDiskPressure
Warning EvictionThresholdMet 8m29s (x19 over 42m) kubelet, node.xxxx1 Attempting to reclaim ephemeral-storage
Warning ImageGCFailed 3m4s kubelet, node.xxxx1 failed to garbage collect required amount of images. Wanted to free 4920913920 bytes, but freed 0 bytes
Reference links:
Biao high CPU node
There may be a node during GC (container GC / image GC) , with a describe node
look. I once encountered such a situation, a lot less container on the last node, but also a bit depressing
Events:
Type Reason Age From Message
---- ------ ---- ----
Warning ImageGCFailed 45m kubelet, cn-shenzhen.xxxx failed to get image stats: rpc error: code = DeadlineExceeded desc = context deadline exceeded
reference:
kubelet source code analysis: Garbage Collect
The Object
under
pod frequently restart
There are a variety of reasons, can not be generalized
Resource reach limit set value
Increase the limit or check application
Readiness/Liveness connection refused
Readiness check failure will restart, but Readiness
check fails is not necessarily a question of the application, if the node itself overloaded, there will also timeout or connection refused
To troubleshoot this problem on a node
pod expulsion (Evicted)
- Node plus a stain lead pod expulsion
-
ephemeral-storage exceeds the limit deported
- EmptyDir usage exceeds his SizeLimit, then this pod will be deported
- Container usage (log, if there is no overlay zoning, include imagefs) exceed his limit, then the pod will be expelled
- Pod local temporary storage for total usage (all emptydir and container) over the pod in the limit of all container and then expelled pod
ephemeral-storage is used for temporarily storing a pod.
resources:
requests:
ephemeral-storage: "2Gi"
limits:
ephemeral-storage: "3Gi"
After being expelled node can still see through get po, with the describe command, you can see the historical reasons expelled
Message: The node was low on resource: ephemeral-storage. Container codis-proxy was using 10619440Ki, which exceeds its request of 0.
reference:
kubectl exec failed to enter the container
I encountered this problem when building codis-server, when ready, and no configuration health check. But when acquiring pod description, display running. In fact, this time the container and not normal.
~ kex codis-server-3 sh
rpc error: code = 2 desc = containerd: container not found
command terminated with exit code 126
Solution: delete this pod, configurationlivenessProbe
pod of virtual host name
Deployment
Derived pod, virtual host name
that is pod name
.
StatefulSet
Derived pod, virtual host name
Shi <pod name>.<svc name>.<namespace>.svc.cluster.local
compared to Deployment
appear more regularly few. And support other pod access
pod after another Crashbackoff
Crashbackoff
There are a variety of reasons.
Create a sandbox (FailedCreateSandBox) failed, mostly network plug problem cni
Mirroring pull, there is a problem of socialism with Chinese characteristics, may be too large, slow pull
There is also a container may be complicated by too high, resulting in an avalanche flow.
For example, there are three containers abc, a suddenly encountered traffic peak rush led to internal collapse, in turn Crashbackoff
, then a will be service
removed out, the rest of the bc can not carry so much traffic, after another crash, and ultimately the site is not accessible. This case, more common in high concurrency website + inefficient web container.
In the case where the code does not change, the optimal solution is to increase the number of copies, and adds HPA, dynamic stretching capacity.
deploy
MinimumReplicationUnavailable
If deploy
the configuration of the SecurityContext, but rejected api-server, this will be the case, the inside of the container api-server remove SecurityContextDeny
the boot parameters.
See in particular Using Admission Controllers
service
We built a service, but there is no corresponding po, what happens?
When the request has been no response, until the request timeout
reference
service connection refuse
There may be reasons
- pod is not provided readinessProbe, not ready to request pod
- kube-proxy is down (kube-proxy is responsible for forwarding the request)
- Network overload
no load balancing service
Check whether the use headless service
. headless service
Is not automatically load balancing ...
kind: Service
spec:
# clusterIP: None的即为`headless service`
type: ClusterIP
clusterIP: None
Specific performance service without their own virtual IP, nslookup ip will be all the pod, but only when the ping ip first pod appears
/ # nslookup consul
nslookup: can't resolve '(null)': Name does not resolve
Name: consul
Address 1: 172.31.10.94 172-31-10-94.consul.default.svc.cluster.local
Address 2: 172.31.10.95 172-31-10-95.consul.default.svc.cluster.local
Address 3: 172.31.11.176 172-31-11-176.consul.default.svc.cluster.local
/ # ping consul
PING consul (172.31.10.94): 56 data bytes
64 bytes from 172.31.10.94: seq=0 ttl=62 time=0.973 ms
64 bytes from 172.31.10.94: seq=1 ttl=62 time=0.170 ms
^C
--- consul ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.170/0.571/0.973 ms
/ # ping consul
PING consul (172.31.10.94): 56 data bytes
64 bytes from 172.31.10.94: seq=0 ttl=62 time=0.206 ms
64 bytes from 172.31.10.94: seq=1 ttl=62 time=0.178 ms
^C
--- consul ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.178/0.192/0.206 ms
Ordinary type: ClusterIP service, nslookup will own the IP service
/ # nslookup consul
nslookup: can't resolve '(null)': Name does not resolve
Name: consul
Address 1: 172.30.15.52 consul.default.svc.cluster.local
ReplicationController not updated
ReplicationController not used to apply updates, but kubectl rolling-update
, but this directive is also abolished, replaced by a kubectl rollout
so should be used kubectl rollout
as a means of updating or a little lazy, then apply file, delete po.
Try to use deploy it.
StatefulSet update failed
StatefulSet is updated one by one, look at whether there is Crashbackoff
a container, the container may be the cause of this update stuck, it can be deleted.
Advanced scheduling
Ensure the use of affinity node running on the target node
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: elasticsearch-test-ready
operator: Exists
Reference links:
Use anti-affinity ensures that each node only run the same application
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: 'app'
operator: In
values:
- nginx-test2
topologyKey: "kubernetes.io/hostname"
namespaces:
- test
Tolerated run
The reason why the master node does not allow general image, because the master node with a stain, if desired forced to run above the master image, it is necessary to tolerate the appropriate stain.
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- effect: NoSchedule
key: node.cloudprovider.kubernetes.io/uninitialized
operator: Exists
Ali cloud the issue Kubernetes
Modify the default ingress
Create a new type svc ingress points to load balance, and then change it kube-system
at nginx-ingress-controller
startup parameters.
- args:
- /nginx-ingress-controller
- '--configmap=$(POD_NAMESPACE)/nginx-configuration'
- '--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services'
- '--udp-services-configmap=$(POD_NAMESPACE)/udp-services'
- '--annotations-prefix=nginx.ingress.kubernetes.io'
- '--publish-service=$(POD_NAMESPACE)/<自定义svc>'
- '--v=2'
LoadBalancer IP service has not been
Specific performance is EXTERNAL-IP has been displayed pending.
~ kg svc consul-web
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul-web LoadBalancer 172.30.13.122 <pending> 443:32082/TCP 5m
This problem with Alibaba Cloud Provider This component related, cloud-controller-manager
there are three components, they need an internal election primary, where can go wrong, and then I put a problem of pod
deleted, just fine.
Cleanup Statefulset dynamic PVC
At present, Ali cloud Statefulset
dynamic use of PVC is nas.
- For this store, you need to first copy the vessel will be zero, or the entire
Statefulset
deleted. - Delete PVC
- The nas mount to any server above, and then delete the directory pvc nas correspondence.
After upgrading to v1.12.6-aliyun.1 fewer allocated memory node
The version of each node reserved 1Gi, equivalent to less entire cluster N GB (N is the number of nodes) for dispensing Pod.
If the node is a 4G, Pod request 3G, extremely easy to be deported.
It proposed to increase the node specifications.
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-aliyun.1", GitCommit:"8cb561c", GitTreeState:"", BuildDate:"2019-04-22T11:34:20Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}
Add new nodes appear NetworkUnavailable
RouteController failed to create a route
Look kubernetes events, if there is a
timed out waiting for the condition -> WaitCreate: ceate route for table vtb-wz9cpnsbt11hlelpoq2zh error, Aliyun API Error: RequestId: 7006BF4E-000B-4E12-89F2-F0149D6688E4 Status Code: 400 Code: QuotaExceeded Message: Route entry quota exceeded in this route table
This problem occurs because the reach of the custom route VPC entry restrictions , the default is 48, we need to raise vpc_quota_route_entrys_num
quotas
Reference (scheduling related application):