HPA自动伸缩
HorizontalPodAutoscaler（简称 HPA ） 自动更新工作负载资源（例如 Deployment 或者 StatefulSet）， 目的是自动扩缩工作负载以满足需求。
水平扩缩意味着对增加的负载的响应是部署更多的 Pod。 这与 “垂直（Vertical）” 扩缩不同，对于 Kubernetes， 垂直扩缩意味着将更多资源（例如：内存或 CPU）分配给已经为工作负载运行的 Pod。
如果负载减少，并且 Pod 的数量高于配置的最小值， HorizontalPodAutoscaler 会指示工作负载资源（Deployment、StatefulSet 或其他类似资源）缩减。
1. 手动调整副本数
1.1 修改yaml中的replicas个数
root@k8s-master-01:/opt/k8s-data/yaml/wework/tomcat-app1# kubectl get deployments.apps wework-tomcat-app1-deployment -n wework
NAME READY UP-TO-DATE AVAILABLE AGE
wework-tomcat-app1-deployment 1/1 1 1 5d
将replicas改为2
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: wework-tomcat-app1-deployment-label
name: wework-tomcat-app1-deployment
namespace: wework
spec:
replicas: 2
selector:
matchLabels:
app: wework-tomcat-app1-selector
template:
metadata:
labels:
app: wework-tomcat-app1-selector
spec:
containers:
- name: wework-tomcat-app1-container
image: harbor.intra.com/wework/tomcat-app1:v3
ports:
- containerPort: 8080
protocol: TCP
name: http
env:
- name: "password"
value: "123456"
- name: "age"
value: "18"
resources:
limits:
cpu: 1
memory: "512Mi"
requests:
cpu: 500m
memory: "512Mi"
volumeMounts:
- name: wework-images
mountPath: /usr/local/nginx/html/webapp/images
readOnly: false
- name: wework-static
mountPath: /usr/local/nginx/html/webapp/static
readOnly: false
volumes:
- name: wework-images
nfs:
server: 192.168.31.109
path: /data/k8s/wework/images
- name: wework-static
nfs:
server: 192.168.31.104
path: /data/k8s/wework/static
---
kind: Service
apiVersion: v1
metadata:
labels:
app: wework-tomcat-app1-service-label
name: wework-tomcat-app1-service
namespace: wework
spec:
type: NodePort
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
nodePort: 30092
selector:
app: wework-tomcat-app1-selector
生效后deployment的pod数量变成了2个
root@k8s-master-01:/opt/k8s-data/yaml/wework/tomcat-app1# kubectl apply -f tomcat-app1.yaml
deployment.apps/wework-tomcat-app1-deployment configured
service/wework-tomcat-app1-service unchanged
root@k8s-master-01:/opt/k8s-data/yaml/wework/tomcat-app1# kubectl get deployments.apps wework-tomcat-app1-deployment -n wework
NAME READY UP-TO-DATE AVAILABLE AGE
wework-tomcat-app1-deployment 2/2 2 2 5d
1.2 命令行调整
通过命令kubectl scale --replicas=指定deployment中pod的数量
root@k8s-master-01:/opt/k8s-data/yaml/wework/tomcat-app1# kubectl scale --replicas=3 deployments.apps wework-tomcat-app1-deployment -n wework
deployment.apps/wework-tomcat-app1-deployment scaled
root@k8s-master-01:/opt/k8s-data/yaml/wework/tomcat-app1# kubectl get deployments.apps wework-tomcat-app1-deployment -n wework
NAME READY UP-TO-DATE AVAILABLE AGE
wework-tomcat-app1-deployment 2/3 3 2 5d
root@k8s-master-01:/opt/k8s-data/yaml/wework/tomcat-app1# kubectl get deploy -n wework wework-tomcat-app1-deployment
NAME READY UP-TO-DATE AVAILABLE AGE
wework-tomcat-app1-deployment 3/3 3 3 5d
1.3 通过命令修改指定yaml创建pod数量
通过命令行kubectl scale --replicas=副本个数 -f 配置文件
root@k8s-master-01:/opt/k8s-data/yaml/wework/tomcat-app1# kubectl scale --replicas=4 -f tomcat-app1.yaml
deployment.apps/wework-tomcat-app1-deployment scaled
root@k8s-master-01:/opt/k8s-data/yaml/wework/tomcat-app1# kubectl get pods -n wework |grep app1
wework-tomcat-app1-deployment-98df9d6f8-2sf4d 1/1 Running 0 3m33s
wework-tomcat-app1-deployment-98df9d6f8-d9pn5 1/1 Running 0 4h17m
wework-tomcat-app1-deployment-98df9d6f8-fnjlx 1/1 Running 0 8s
wework-tomcat-app1-deployment-98df9d6f8-ghw76 1/1 Running 0 11m
1.4 其他
# Scale a replica set named 'foo' to 3
kubectl scale --replicas=3 rs/foo
# Scale a resource identified by type and name specified in "foo.yaml" to 3
kubectl scale --replicas=3 -f foo.yaml
# If the deployment named mysql's current size is 2, scale mysql to 3
kubectl scale --current-replicas=2 --replicas=3 deployment/mysql
# Scale multiple replication controllers
kubectl scale --replicas=5 rc/foo rc/bar rc/baz
# Scale stateful set named 'web' to 3
kubectl scale --replicas=3 statefulset/web
2. 通过HPA调整副本数
K8s从1.1版本开始支持了HPA(Horizontal Pod Autoscaler)控制器.用于基于Pod中CPU/Memory资源利用率进行Pod的自动扩缩容.使用Metrices Server进行数据采集,通过API将数据提供给HPA控制器,实现基于某个资源利用率对Pod进行扩缩容.
2.1 metrics-server
使用HPA需要先部署metrics-server
部署详见: https://github.com/kubernetes-sigs/metrics-server
部署完后就可以看到node的CPU,MEM使用率
root@k8s-master-01:~/metrics# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
192.168.31.101 89m 4% 929Mi 28%
192.168.31.102 212m 10% 2378Mi 73%
192.168.31.103 111m 5% 1897Mi 58%
192.168.31.111 128m 3% 1494Mi 20%
192.168.31.112 131m 6% 2524Mi 78%
192.168.31.113 153m 7% 1091Mi 33%
2.2 controller-manager启动参数
| 参数 | 含义 |
|---|---|
| –horizontal-pod-autoscaler-sync-period duration | 定义Pod水平伸缩时间间隔,默认15秒 |
| –horizontal-pod-autoscaler-cpu-initialization-period | Pod初始化时间,在此期间内Pod的Cpu资源指标将不会被采纳,默认5分钟 |
| –horizontal-pod-autoscaler-initial-readiness-delay | 用于设置pod准备时间,在此期间内Pod统统被认为未就绪不采集数据,默认为30秒 |
kube-controller的默认配置位于:/etc/systemd/system/kube-controller-manager.service ,可以按需求进行修改
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/bin/kube-controller-manager \
--bind-address=192.168.31.101 \
--allocate-node-cidrs=true \
--cluster-cidr=172.100.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
--leader-elect=true \
--node-cidr-mask-size=24 \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
--service-cluster-ip-range=10.200.0.0/16 \
--use-service-account-credentials=true \
--v=2
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
2.3 创建HPA
2.3.1 通过命令行创建
kubectl autoscale deployment -n wework wework-tomcat-app1 --min=2 --max=5 --cpu-percent=60
2.3.2 通过yml创建
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
namespace: wework
name: wework-tomcat-app1-podautoscaler
labels:
app: wework-tomcat-app1-selector
version: v2beta1
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: wework-tomcat-app1-deployment
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 30
部署hpa,正常情况下TARGETS是 ??%/30%
root@k8s-master-01:~/metrics# kubectl apply -f hpa.yaml
horizontalpodautoscaler.autoscaling/wework-tomcat-app1-podautoscaler created
root@k8s-master-01:~/metrics# kubectl get hpa -n wework
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 0%/30% 2 10 4 8m5s
如果是unknown状态,就是hpa.yaml配置有问题,需要再次核对标签和name等信息.
root@k8s-master-01:~/metrics# kubectl get hpa -n wework
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1 <unknown>/30% 2 10 0 2m14s
默认horizontal-pod-autoscaler-cpu-initialization-period 5分钟,所以暂时不会有影响Pod数量
root@k8s-master-01:~/metrics# kubectl get pods -n wework |grep app1
wework-tomcat-app1-deployment-98df9d6f8-2sf4d 1/1 Running 0 94m
wework-tomcat-app1-deployment-98df9d6f8-d9pn5 1/1 Running 0 5h48m
wework-tomcat-app1-deployment-98df9d6f8-fnjlx 1/1 Running 0 90m
wework-tomcat-app1-deployment-98df9d6f8-ghw76 1/1 Running 0 102m
root@k8s-master-01:~/metrics# kubectl get deploy -n wework wework-tomcat-app1-deployment
NAME READY UP-TO-DATE AVAILABLE AGE
wework-tomcat-app1-deployment 4/4 4 4 5d1h
5分钟之后,由于cpu使用率小于30%,于是hpa进行了收缩
root@k8s-master-01:~/metrics# kubectl get deployments.apps -n wework wework-tomcat-app1-deployment
NAME READY UP-TO-DATE AVAILABLE AGE
wework-tomcat-app1-deployment 2/2 2 2 5d1h
root@k8s-master-01:~/metrics# kubectl get pods -n wework |grep wework-tomcat-app1
wework-tomcat-app1-deployment-98df9d6f8-d9pn5 1/1 Running 0 5h52m
wework-tomcat-app1-deployment-98df9d6f8-ghw76 1/1 Running 0 106m
2.4 测试自动伸缩
2.4.1 自动扩展
进入一个pod,使用openssl命令跑高CPU
[root@wework-tomcat-app1-deployment-98df9d6f8-ghw76 /]# openssl speed
Doing md2 for 3s on 16 size blocks: 696894 md2's in 2.85s
Doing md2 for 3s on 64 size blocks: 355911 md2's in 2.86s
Doing md2 for 3s on 256 size blocks: 122354 md2's in 2.91s
Doing md2 for 3s on 1024 size blocks: 33355 md2's in 2.91s
Doing md2 for 3s on 8192 size blocks: 4360 md2's in 2.97s
Doing md4 for 3s on 16 size blocks: 22608001 md4's in 2.94s
Doing md4 for 3s on 64 size blocks: 16352809 md4's in 2.91s
Doing md4 for 3s on 256 size blocks: 9126019 md4's in 2.81s
Doing md4 for 3s on 1024 size blocks: 3463848 md4's in 2.96s
Doing md4 for 3s on 8192 size blocks: 484363 md4's in 2.85s
Doing md5 for 3s on 16 size blocks: 16087992 md5's in 2.80s
Doing md5 for 3s on 64 size blocks: 11538806 md5's in 2.94s
Doing md5 for 3s on 256 size blocks: 6081728 md5's in 2.97s
Doing md5 for 3s on 1024 size blocks: 2075725 md5's in 2.93s
此时HPA会自动扩展POD数量
root@k8s-master-01:~/metrics# kubectl get hpa -n wework --watch
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 2%/30% 2 10 2 46m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 8%/30% 2 10 2 46m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 5%/30% 2 10 2 47m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 5%/30% 2 10 2 48m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 102%/30% 2 10 2 49m
yawework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 102%/30% 2 10 4 49m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 102%/30% 2 10 7 49m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 2%/30% 2 10 7 50m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 22%/30% 2 10 7 51m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 34%/30% 2 10 7 52m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 34%/30% 2 10 7 53m
查看HPA详细信息可以看到,POD数量从2个扩展到4个,又从4个扩展到7个.
root@k8s-master-01:~# kubectl describe hpa -n wework wework-tomcat-app1-podautoscaler
Name: wework-tomcat-app1-podautoscaler
Namespace: wework
Labels: app=wework-tomcat-app1-selector
version=v2beta1
Annotations: <none>
CreationTimestamp: Tue, 23 Aug 2022 16:01:06 +0800
Reference: Deployment/wework-tomcat-app1-deployment
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 34% (171m) / 30%
Min replicas: 2
Max replicas: 10
Deployment pods: 7 current / 7 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 41m horizontal-pod-autoscaler New size: 2; reason: All metrics below target
Normal SuccessfulRescale 5m37s horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 5m22s horizontal-pod-autoscaler New size: 7; reason: cpu resource utilization (percentage of request) above target
root@k8s-master-01:~/metrics# kubectl get pods -n wework |grep wework-tomcat-app1
wework-tomcat-app1-deployment-98df9d6f8-6px56 1/1 Running 0 95s
wework-tomcat-app1-deployment-98df9d6f8-d9pn5 1/1 Running 0 6h30m
wework-tomcat-app1-deployment-98df9d6f8-dt29q 1/1 Running 0 95s
wework-tomcat-app1-deployment-98df9d6f8-fpzmp 1/1 Running 0 95s
wework-tomcat-app1-deployment-98df9d6f8-ghw76 1/1 Running 0 144m
wework-tomcat-app1-deployment-98df9d6f8-vxrlx 1/1 Running 0 110s
2.4.2 自动收缩
此时停止CPU负载
oing camellia-128 cbc for 3s on 8192 size blocks: 88823 camellia-128 cbc's in 2.97s
Doing camellia-192 cbc for 3s on 16 size blocks: 28821766 camellia-192 cbc's in 2.91s
Doing camellia-192 cbc for 3s on 64 size blocks: 8081136 camellia-192 cbc's in 2.92s
Doing camellia-192 cbc for 3s on 256 size blocks: ^C
[root@wework-tomcat-app1-deployment-98df9d6f8-ghw76 /]#
停止后由于hpa获取到CPU使用率的下降,也开始了收缩POD的数量
root@k8s-master-01:~/metrics# kubectl get hpa -n wework --watch
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
略
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 33%/30% 2 10 7 55m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 27%/30% 2 10 7 56m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 0%/30% 2 10 7 57m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 0%/30% 2 10 7 58m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 0%/30% 2 10 7 59m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 0%/30% 2 10 7 60m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 0%/30% 2 10 7 62m
wework-tomcat-app1-podautoscaler Deployment/wework-tomcat-app1-deployment 0%/30% 2 10 2 62m
root@k8s-master-01:~/metrics# kubectl get pods -n wework |grep wework-tomcat-app1
wework-tomcat-app1-deployment-98df9d6f8-d9pn5 1/1 Running 0 6h42m
wework-tomcat-app1-deployment-98df9d6f8-ghw76 1/1 Running 0 156m
此时,Pod数量又恢复到了2个.HPA自动伸缩测试完毕