Kubernetes 资源限制
- Kubernetes对单个容器的CPU及Memory实现资源限制
- Kubernetes对单个Pod的CPU及Memory实现资源限制
- Kubernetes对整个Namespace的CPU及Memory实现资源限制
1. Kubernets中资源限制
- CPU以核心为单位
- Memory以字节为单位
- requests 为Kubernetes scheduler执行pod调度时node节点至少需要拥有的资源.
- limits 为Pod运行成功后最多可以使用的资源上限.
2. Containers资源限制
限制内存最多使用512M,调度时内存最少需要100M
apiVersion: apps/v1
kind: Deployment
metadata:
name: limit-test-deployment
namespace: wework
spec:
replicas: 1
selector:
matchLabels: #rs or deployment
app: limit-test-pod
template:
metadata:
labels:
app: limit-test-pod
spec:
containers:
- name: limit-test-container
image: lorel/docker-stress-ng
resources:
limits:
memory: "512Mi"
requests:
memory: "100Mi"
args: ["--vm", "2", "--vm-bytes", "256M"]
生成pod
跑了这个压测Pod,2个进程,每个进程使用256M内存,合计512M左右.此时正好跑到上线512M
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case1-pod-memory-limit.yml
deployment.apps/limit-test-deployment created
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get pods -n wework|grep limit
limit-test-deployment-597c4f5466-x65f4 1/1 Running 0 8m43s
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl top pods -n wework|grep limit
limit-test-deployment-597c4f5466-x65f4 1863m 510Mi
修改yaml,将上限改为375M
limits:
memory: "375Mi"
再次生成pod,此时POD的内存只能跑到350Mi左右,无法超过375Mi,但CPU没有做限制,所以CPU跑到了1.8
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case1-pod-memory-limit.yml
deployment.apps/limit-test-deployment configured
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl top pods -n wework|grep limit
limit-test-deployment-7db68c5584-qg468 1808m 350Mi
加上CPU限制
limits:
memory: "200Mi"
cpu: "1.2"
此时再看pod状态,CPU最高也不会再超过1.2
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case1-pod-memory-limit.yml
deployment.apps/limit-test-deployment created
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl top pods -n wework|grep lim
limit-test-deployment-69d988487c-pz884 1199m 198Mi
这样就能避免1个pod占用资源过多,造成系统异常
3. Pod和Container资源限制
当没有限制时,创建一个2G,3C的容器
略
resources:
limits:
cpu: "3"
memory: "512Mi"
requests:
memory: "100Mi"
cpu: "500m"
略
Pod被正常创建且CPU为3
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get pods -n wework -o wide|grep lim
limit-test-deployment-69ffd466b9-rw75s 1/1 Running 0 105s 172.100.109.76 192.168.31.111 <none> <none>
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl describe pods limit-test-deployment-69ffd466b9-rw75s -n wework
Name: limit-test-deployment-69ffd466b9-rw75s
Namespace: wework
Priority: 0
Node: 192.168.31.111/192.168.31.111
Start Time: Fri, 19 Aug 2022 14:01:25 +0800
Labels: app=limit-test-pod
pod-template-hash=69ffd466b9
Annotations: <none>
Status: Running
IP: 172.100.109.76
IPs:
IP: 172.100.109.76
Controlled By: ReplicaSet/limit-test-deployment-69ffd466b9
Containers:
limit-test-container:
Container ID: docker://1200280f7dc02596773bf1d124428c47bb8cb4292495de8b7e16f1f9750d74b4
Image: lorel/docker-stress-ng
Image ID: docker-pullable://lorel/docker-stress-ng@sha256:c8776b750869e274b340f8e8eb9a7d8fb2472edd5b25ff5b7d55728bca681322
Port: <none>
Host Port: <none>
Args:
--vm
2
--vm-bytes
256M
State: Running
Started: Fri, 19 Aug 2022 14:01:44 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 3
memory: 512Mi
Requests:
cpu: 500m
memory: 100Mi
| 参数 | 作用 |
|---|---|
| type: Container | 针对单个容器进行限制 |
| type: Pod | 针对单个pod进行限制 |
| type: PersistentVolumeClaim | 针对namespace下pvc大小限制 |
| max | 限制最大 |
| min | 限制最小值 |
| default | limit默认值 |
| defaultRequest | request默认值 |
| maxLimitRequestRatio | limit/request的最大值 |
| cpu | CPU限制 |
| memory | 内存限制 |
| storage | 磁盘容量限制 |
限制后wework Namespace下:
容器:
最多使用2个cpu,2G内存.
小于CPU 500m和内存512Mi容器将不被创建.
默认Cpu 500m,内存512Mi.limit最多是request的2倍.
Pod:
最多使用4个cpu,4G内存.
pvc:
最多使用50Gi,最少使用30Gi.
apiVersion: v1
kind: LimitRange
metadata:
name: limitrange-wework
namespace: wework
spec:
limits:
- type: Container #限制的资源类型
max:
cpu: "2" #限制单个容器的最大CPU
memory: "2Gi" #限制单个容器的最大内存
min:
cpu: "500m" #限制单个容器的最小CPU
memory: "512Mi" #限制单个容器的最小内存
default:
cpu: "500m" #默认单个容器的CPU限制
memory: "512Mi" #默认单个容器的内存限制
defaultRequest:
cpu: "500m" #默认单个容器的CPU创建请求
memory: "512Mi" #默认单个容器的内存创建请求
maxLimitRequestRatio:
cpu: 2 #限制CPU limit/request比值最大为2
memory: 2 #限制内存limit/request比值最大为1.5
- type: Pod
max:
cpu: "4" #限制单个Pod的最大CPU
memory: "4Gi" #限制单个Pod最大内存
- type: PersistentVolumeClaim
max:
storage: 50Gi #限制PVC最大的requests.storage
min:
storage: 30Gi #限制PVC最小的requests.storage
配置生效
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case3-LimitRange.yaml
limitrange/limitrange-wework created
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get limitranges -n wework
NAME CREATED AT
limitrange-wework 2022-08-19T05:55:17Z
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl describe limitranges limitrange-wework -n wework
Name: limitrange-wework
Namespace: wework
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container memory 512Mi 2Gi 512Mi 512Mi 2
Container cpu 500m 2 500m 500m 2
Pod cpu - 4 - - -
Pod memory - 4Gi - - -
PersistentVolumeClaim storage 30Gi 50Gi - - -
此时再次创建刚才那个pod时虽然现实创建成功,但该pod并没有被创建.
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case2-pod-memory-and-cpu-limit.yml
deployment.apps/limit-test-deployment created
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get pods -n wework |grep lim
查看deployment时发现该deployment是0/1的状态
查看deployment的输出显示
CPU已经超出最大值了,最大值是2,我们的limit设置了3
limit和request的比值最大是2,我们已经是6了
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get deployments.apps limit-test-deployment -n wework
NAME READY UP-TO-DATE AVAILABLE AGE
limit-test-deployment 0/1 0 0 23m
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get deployments.apps limit-test-deployment -n wework -o yaml
略
- lastTransitionTime: "2022-08-19T06:31:19Z"
lastUpdateTime: "2022-08-19T06:31:19Z"
message: 'pods "limit-test-deployment-789fffc8f4-cq5wq" is forbidden: [maximum
cpu usage per Container is 2, but limit is 3, cpu max limit to request ratio
per Container is 2, but provided ratio is 6.000000]'
修改yaml后,重新部署
resources:
limits:
cpu: "2"
memory: "1000Mi"
requests:
memory: "1000Mi"
cpu: "1"
此时pod被正常创建了
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case2-pod-memory-and-cpu-limit.yml
deployment.apps/limit-test-deployment configured
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get deploy limit-test-deployment -n wework
NAME READY UP-TO-DATE AVAILABLE AGE
limit-test-deployment 1/1 1 1 6m25s
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get pod limit-test-deployment-d7cb67577-dtcp7 -n wework
NAME READY STATUS RESTARTS AGE
limit-test-deployment-d7cb67577-dtcp7 1/1 Running 0 43s
## 此时deployment也没有报错了
- lastTransitionTime: "2022-08-19T06:31:19Z"
lastUpdateTime: "2022-08-19T06:37:37Z"
message: ReplicaSet "limit-test-deployment-d7cb67577" has successfully progressed.
reason: NewReplicaSetAvailable
4. Pod资源限制
单个容器限制3C3G Pod限制4C4G
apiVersion: v1
kind: LimitRange
metadata:
name: limitrange-wework
namespace: wework
spec:
limits:
- type: Container #限制的资源类型
max:
cpu: "3" #限制单个容器的最大CPU
memory: "3Gi" #限制单个容器的最大内存
min:
cpu: "500m" #限制单个容器的最小CPU
memory: "512Mi" #限制单个容器的最小内存
default:
cpu: "500m" #默认单个容器的CPU限制
memory: "512Mi" #默认单个容器的内存限制
defaultRequest:
cpu: "500m" #默认单个容器的CPU创建请求
memory: "512Mi" #默认单个容器的内存创建请求
maxLimitRequestRatio:
cpu: 2 #限制CPU limit/request比值最大为2
memory: 2 #限制内存limit/request比值最大为1.5
- type: Pod
max:
cpu: "4" #限制单个Pod的最大CPU
memory: "4Gi" #限制单个Pod最大内存
- type: PersistentVolumeClaim
max:
storage: 50Gi #限制PVC最大的requests.storage
min:
storage: 30Gi #限制PVC最小的requests.storage
1个Pod里2个container最大Memory 3Gi共6Gi
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: wework-wordpress-deployment-label
name: wework-wordpress-deployment
namespace: wework
spec:
replicas: 1
selector:
matchLabels:
app: wework-wordpress-selector
template:
metadata:
labels:
app: wework-wordpress-selector
spec:
containers:
- name: wework-wordpress-nginx-container
image: nginx:1.16.1
imagePullPolicy: Always
ports:
- containerPort: 80
protocol: TCP
name: http
env:
- name: "password"
value: "123456"
- name: "age"
value: "18"
resources:
limits:
cpu: 1
memory: 3Gi
requests:
cpu: 500m
memory: 2Gi
- name: wework-wordpress-php-container
image: php:5.6-fpm-alpine
imagePullPolicy: Always
ports:
- containerPort: 80
protocol: TCP
name: http
env:
- name: "password"
value: "123456"
- name: "age"
value: "18"
resources:
limits:
cpu: 1
#cpu: 2
memory: 3Gi
requests:
cpu: 500m
memory: 2Gi
---
kind: Service
apiVersion: v1
metadata:
labels:
app: wework-wordpress-service-label
name: wework-wordpress-service
namespace: wework
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
nodePort: 30033
selector:
app: wework-wordpress-selector
生效配置
可以看到提示了Pod内存最大4Gi,但limit设置了6442450944
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case4-pod-RequestRatio-limit.yaml
deployment.apps/wework-wordpress-deployment configured
service/wework-wordpress-service created
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get deployments.apps wework-wordpress-deployment -n wework -o yaml
略
- lastTransitionTime: "2022-08-19T07:04:17Z"
lastUpdateTime: "2022-08-19T07:04:17Z"
message: 'pods "wework-wordpress-deployment-577c6775d8-bxwm6" is forbidden: maximum
memory usage per Pod is 4Gi, but limit is 6442450944'
reason: FailedCreate
将2个Container内存需求调小到2G再试,这次就创建成功了
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case4-pod-RequestRatio-limit.yaml
deployment.apps/wework-wordpress-deployment configured
service/wework-wordpress-service unchanged
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get deploy wework-wordpress-deployment -n wework
NAME READY UP-TO-DATE AVAILABLE AGE
wework-wordpress-deployment 1/1 1 1 5m27s
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get pod wework-wordpress-deployment-684df659bd-56ss8 -n wework
NAME READY STATUS RESTARTS AGE
wework-wordpress-deployment-684df659bd-56ss8 2/2 Running 0 74s
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get deploy wework-wordpress-deployment -n wework -o yaml
略
- lastTransitionTime: "2022-08-19T07:04:17Z"
lastUpdateTime: "2022-08-19T07:09:37Z"
message: ReplicaSet "wework-wordpress-deployment-684df659bd" has successfully
progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
5. Namespace资源限制
通过ResourceQuota限制整个Namespace的资源
主要限制CPU,内存还可以限制GPU,Pod数量,Service数量等
apiVersion: v1
kind: ResourceQuota
metadata:
name: quota-linux
namespace: linux
spec:
hard:
requests.cpu: "8"
limits.cpu: "8"
requests.memory: 4Gi
limits.memory: 4Gi
requests.nvidia.com/gpu: 4
pods: "2"
services: "6"
生效配置
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case6-ResourceQuota-ns.yaml
resourcequota/quota-linux created
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get resourcequotas -n linux
NAME AGE REQUEST LIMIT
quota-linux 63s pods: 2/2, requests.cpu: 0/8, requests.memory: 0/4Gi, requests.nvidia.com/gpu: 0/4, services: 2/6 limits.cpu: 0/8, limits.memory: 0/4Gi
测试创建更多的pod
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get resourcequotas -n linux
NAME AGE REQUEST LIMIT
quota-linux 3m9s pods: 2/2, requests.cpu: 0/8, requests.memory: 0/4Gi, requests.nvidia.com/gpu: 0/4, services: 2/6 limits.cpu: 0/8, limits.memory: 0/4Gi
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get pods -n linux
NAME READY STATUS RESTARTS AGE
linux-nginx-deployment-5cd9566d7f-npdw2 1/1 Running 0 7h37m
linux-tomcat-app1-deployment-6f8864d5d9-wdwv8 1/1 Running 0 7h37m
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get deploy -n linux
NAME READY UP-TO-DATE AVAILABLE AGE
linux-nginx-deployment 1/5 0 1 2d3h
linux-tomcat-app1-deployment 1/1 1 1 2d3h
## 可以看到创建失败了.原因是pod数量到上限了
- lastTransitionTime: "2022-08-19T07:31:01Z"
lastUpdateTime: "2022-08-19T07:31:01Z"
message: 'pods "linux-nginx-deployment-6477f56f7f-4pgnm" is forbidden: exceeded
quota: quota-linux, requested: pods=1, used: pods=2, limited: pods=2'
reason: FailedCreate
status: "True"
type: ReplicaFailure
调整最大Pod数量到20个,并修改deployment pod到3个,此时创建成功
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl apply -f case6-ResourceQuota-ns.yaml
resourcequota/quota-linux configured
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get resourcequotas -n linux
NAME AGE REQUEST LIMIT
quota-linux 10m pods: 4/20, requests.cpu: 1500m/8, requests.memory: 1536Mi/4Gi, requests.nvidia.com/gpu: 0/4, services: 2/6 limits.cpu: 3/8, limits.memory: 3Gi/4Gi
- lastTransitionTime: "2022-08-19T07:36:22Z"
lastUpdateTime: "2022-08-19T07:37:34Z"
message: ReplicaSet "linux-nginx-deployment-6477f56f7f" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 2
readyReplicas: 3
replicas: 3
updatedReplicas: 3
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get deploy -n linux
NAME READY UP-TO-DATE AVAILABLE AGE
linux-nginx-deployment 3/3 3 3 86s
linux-tomcat-app1-deployment 1/1 1 1 2d3h
再将deployment的Pod个数调整到5个时,可以看到limits.memory到了上限制
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get resourcequotas -n linux
NAME AGE REQUEST LIMIT
quota-linux 11m pods: 5/20, requests.cpu: 2/8, requests.memory: 2Gi/4Gi, requests.nvidia.com/gpu: 0/4, services: 2/6 limits.cpu: 4/8, limits.memory: 4Gi/4Gi
此时查看deployment的状态,确实是limits.memory=4Gi, limited了.
但deployment触发上限之前创建的Pod还是正常在运行的.
- lastTransitionTime: "2022-08-19T07:39:31Z"
lastUpdateTime: "2022-08-19T07:39:31Z"
message: 'pods "linux-nginx-deployment-6477f56f7f-vzht2" is forbidden: exceeded
quota: quota-linux, requested: limits.memory=1Gi, used: limits.memory=4Gi, limited:
limits.memory=4Gi'
reason: FailedCreate
status: "True"
type: ReplicaFailure
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get pods -n linux
NAME READY STATUS RESTARTS AGE
linux-nginx-deployment-6477f56f7f-4r5hg 1/1 Running 0 6m22s
linux-nginx-deployment-6477f56f7f-6cvdl 1/1 Running 0 6m22s
linux-nginx-deployment-6477f56f7f-dffz9 1/1 Running 0 3m13s
linux-nginx-deployment-6477f56f7f-v4sjp 1/1 Running 0 6m22s
linux-tomcat-app1-deployment-6f8864d5d9-wdwv8 1/1 Running 0 7h48m
root@k8s-master-01:/opt/k8s-data/yaml/limit# kubectl get deployment -n linux
NAME READY UP-TO-DATE AVAILABLE AGE
linux-nginx-deployment 4/5 4 4 6m31s
linux-tomcat-app1-deployment 1/1 1 1 2d4h
至此K8s资源限制常见的配置方法测试完毕