OpenKruise
OpenKruise is a standard extension of Kubernetes. It can be used with native Kubernetes and provides more powerful and efficient capabilities for managing application containers, sidecars, and image distribution.
Core functions
-
In-place upgrade In-
place upgrade is an upgrade mirroring capability that can avoid deleting and creating new Pods. It is faster and more efficient than native Deployment/StatefulSet reconstruction Pod upgrades, and avoids interference with other containers in the Pod that do not need to be updated. -
Sidecar management
supports defining sidecar containers in a single CR. OpenKruise can help you inject these Sidecar containers into all eligible Pods. This process is very similar to Istio injection, but you can manage any Sidecar you care about. - Deploy
across multiple availability zones. Define a global workload and container across multiple availability zones. OpenKruise will help you create a corresponding subordinate workload in each availability zone. You can manage their number of copies, versions, and even adopt different release strategies for different availability zones.
CRD list
CloneSet
提供更加高效、确定可控的应用管理和部署能力，支持优雅原地升级、指定删除、发布顺序可配置、并行/灰度发布等丰富的策略，可以满足更多样化的应用场景。
Advanced StatefulSet
基于原生 StatefulSet 之上的增强版本，默认行为与原生完全一致，在此之外提供了原地升级、并行发布（最大不可用）、发布暂停等功能。
SidecarSet
对 sidecar 容器做统一管理，在满足 selector 条件的 Pod 中注入指定的 sidecar 容器。
UnitedDeployment
通过多个 subset workload 将应用部署到多个可用区。
BroadcastJob
配置一个 job，在集群中所有满足条件的 Node 上都跑一个 Pod 任务。
Advanced DaemonSet
基于原生 DaemonSet 之上的增强版本，默认行为与原生一致，在此之外提供了灰度分批、按 Node label 选择、暂停、热升级等发布策略。
AdvancedCronJob
一个扩展的 CronJob 控制器，目前 template 模板支持配置使用 Job 或 BroadcastJob。The above are all introduced in the official documents. This article mainly focuses on actual combat, first talk about CloneSet, and other controllers will be updated later. . .
Deploy Kruise to the Kubernetes cluster
Here use helm to install Kruise
1. Now kruise Chart
wget https://github.com/openkruise/kruise/releases/download/v0.7.0/kruise-chart.tgz
tar -zxf kruise-chart.tgz
cd kruise
[root@ kruise]# ls -l
total 16
-rw-r--r-- 1 root root 311 Dec 20 15:09 Chart.yaml
-rw-r--r-- 1 root root 4052 Dec 20 15:09 README.md
drwxr-xr-x 2 root root 4096 Dec 23 10:18 templates
-rw-r--r-- 1 root root 659 Dec 20 15:09 values.yaml2. Modify values.yaml, no need to modify by default.
3. Execute deployment
[root@qd01-stop-k8s-master001 kruise]# kubectl create ns kruise
namespace/kruise created
[root@qd01-stop-k8s-master001 kruise]# helm install kruise -n kruise -f values.yaml .
W1223 10:22:13.562088 1589994 warnings.go:67] apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
。。。。。。。
NAME: kruise
LAST DEPLOYED: Wed Dec 23 10:22:12 2020
NAMESPACE: kruise
STATUS: deployed
REVISION: 1
TEST SUITE: None
这里会看到一堆的deprecated信息，因为新版的kubernetes对CRD的版本会淘汰，可以根据自己的集群版本修改CRD的API版本即可4. Check kruise deployment status
[root@qd01-stop-k8s-master001 kruise]# helm ls -n kruise
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
kruise kruise 1 2020-12-23 10:22:12.963651877 +0800 CST deployed kruise-0.7.0
可以看到，集群中有的kruise crd类型
[root@qd01-stop-k8s-master001 kruise]# kubectl get crd|grep kruise
advancedcronjobs.apps.kruise.io 2020-12-23T02:22:13Z
broadcastjobs.apps.kruise.io 2020-12-23T02:22:13Z
clonesets.apps.kruise.io 2020-12-23T02:22:13Z
daemonsets.apps.kruise.io 2020-12-23T02:22:13Z
sidecarsets.apps.kruise.io 2020-12-23T02:22:13Z
statefulsets.apps.kruise.io 2020-12-23T02:22:13Z
uniteddeployments.apps.kruise.io 2020-12-23T02:22:13ZLet’s start using these managers
CloneSet
The CloneSet controller provides the ability to efficiently manage stateless applications. It can target native deployments, but CloneSet provides many enhancements.
1. Let's create a simple CloneSet first, yaml is as follows
apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
labels:
app: nginx-alpine
name: nginx-alpine
spec:
replicas: 5
selector:
matchLabels:
app: nginx-alpine
template:
metadata:
labels:
app: nginx-alpine
spec:
containers:
- name: nginx
image: nginx:alpine2. Deployment
[root@qd01-stop-k8s-master001 demo]# kubectl apply -f CloneSet.yaml
cloneset.apps.kruise.io/nginx-alpine created
[root@qd01-stop-k8s-master001 demo]# kubectl get po |grep nginx
nginx-alpine-29g7n 1/1 Running 0 45s
nginx-alpine-bvgqm 1/1 Running 0 45s
nginx-alpine-q9tlw 1/1 Running 0 45s
nginx-alpine-s2t46 1/1 Running 0 44s
nginx-alpine-sslvf 1/1 Running 0 44s
从输出结果看，和原生的Deployment没有啥区别
#注意，这里如果get deployment是看不到nginx-alpine这个应用的，需要get cloneset才能看到
[root@qd01-stop-k8s-master001 demo]# kubectl get deployment
[root@qd01-stop-k8s-master001 demo]# kubectl get cloneset
NAME DESIRED UPDATED UPDATED_READY READY TOTAL AGE
nginx-alpine 5 5 5 5 5 2m16sCloneSet allows users to configure PVC template volumeClaimTemplates to generate exclusive PVC for each Pod, which is not supported by Deployment. If the user does not specify this template, CloneSet will create a Pod without PVC.
3. Now let's create an example with PVC template
apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
labels:
app: nginx-2
name: nginx-2
spec:
replicas: 5
selector:
matchLabels:
app: nginx-2
template:
metadata:
labels:
app: nginx-2
spec:
containers:
- name: nginx
image: nginx:alpine
volumeMounts:
- name: data-vol
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: rbd
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: rbd
resources:
requests:
storage: 2Gideploy
[root@qd01-stop-k8s-master001 demo]# kubectl apply -f CloneSet.yaml
cloneset.apps.kruise.io/nginx-2 created
[root@qd01-stop-k8s-master001 demo]# kubectl get pv|grep data-vol
pvc-0fde19f3-ea4b-47e0-81be-a8e43812e47b 2Gi RWO Delete Bound default/data-vol-nginx-2-t55h8 rbd 83s
pvc-72accf10-57a6-4418-a1bc-c64633b84434 2Gi RWO Delete Bound default/data-vol-nginx-2-t49mk rbd 82s
pvc-8fc8b9a5-afe8-446a-9190-08fcee0ec9f6 2Gi RWO Delete Bound default/data-vol-nginx-2-jw2zp rbd 84s
pvc-c9fba396-e357-43e8-9510-616f698da765 2Gi RWO Delete Bound default/data-vol-nginx-2-b5fdd rbd 84s
pvc-e5302eab-a9f2-4a71-a5a3-4cd43205e8a0 2Gi RWO Delete Bound default/data-vol-nginx-2-l54dz rbd 84s
[root@qd01-stop-k8s-master001 demo]# kubectl get po|grep nginx
nginx-2-b5fdd 1/1 Running 0 97s
nginx-2-jw2zp 1/1 Running 0 97s
nginx-2-l54dz 1/1 Running 0 97s
nginx-2-t49mk 1/1 Running 0 96s
nginx-2-t55h8 1/1 Running 0 96sFrom the deployment results, we can see that each pod creates a PVC, which cannot be achieved by native Deployment.
note:
每个被自动创建的 PVC 会有一个 ownerReference 指向 CloneSet，因此 CloneSet 被删除时，它创建的所有 Pod 和 PVC 都会被删除。
每个被 CloneSet 创建的 Pod 和 PVC，都会带一个 apps.kruise.io/cloneset-instance-id: xxx 的 label。关联的 Pod 和 PVC 会有相同的 instance-id，且它们的名字后缀都是这个 instance-id。
如果一个 Pod 被 CloneSet controller 缩容删除时，这个 Pod 关联的 PVC 都会被一起删掉。
如果一个 Pod 被外部直接调用删除或驱逐时，这个 Pod 关联的 PVC 还都存在；并且 CloneSet controller 发现数量不足重新扩容时，新扩出来的 Pod 会复用原 Pod 的 instance-id 并关联原来的 PVC。
当 Pod 被重建升级时，关联的 PVC 会跟随 Pod 一起被删除、新建。
当 Pod 被原地升级时，关联的 PVC 会持续使用。4. Specify Pod shrinkage.
When a CloneSet is shrinked, sometimes users need to specify some Pods to delete. This is impossible for StatefulSet or Deployment, because StatefulSet needs to delete Pods according to the sequence number, and Deployment/ReplicaSet can only delete according to the order defined in the controller.
CloneSet allows users to specify the name of the Pod they want to delete while reducing the number of replicas.
Now let's modify the deployment file of the above example and specify to delete nginx-2-t55h8this Pod
apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
metadata:
labels:
app: nginx-2
name: nginx-2
spec:
replicas: 4
scaleStrategy:
podsToDelete:
- nginx-2-t55h8Then update the yaml file
[root@qd01-stop-k8s-master001 demo]# kubectl apply -f CloneSet.yaml
cloneset.apps.kruise.io/nginx-2 configured
[root@qd01-stop-k8s-master001 demo]# kubectl get po|grep nginx
nginx-2-b5fdd 1/1 Running 0 11m
nginx-2-jw2zp 1/1 Running 0 11m
nginx-2-l54dz 1/1 Running 0 11m
nginx-2-t49mk 1/1 Running 0 11mNow look at the input result. nginx-2-t55h8This Pod is no longer available .
This function is very useful. For example, a certain machine fails, or the load is too high, and you want to delete the specified pod.
5. Upgrade function
CloneSet 提供了和 Advanced StatefulSet 相同的 3 个升级方式，默认为 ReCreate：
ReCreate: 控制器会删除旧 Pod 和它的 PVC，然后用新版本重新创建出来。
InPlaceIfPossible: 控制器会优先尝试原地升级 Pod，如果不行再采用重建升级。目前，只有修改 spec.template.metadata.* 和 spec.template.spec.containers[x].image 这些字段才可以走原地升级。
InPlaceOnly: 控制器只允许采用原地升级。因此，用户只能修改上一条中的限制字段，如果尝试修改其他字段会被 Kruise 拒绝。Now let’s try to upgrade the Pod function in place. Upgrade the nginx image from nginx:alpine to nginx:latest.
First modify the yaml file, and paste the modified part of the file here.
apiVersion: apps.kruise.io/v1alpha1
kind: CloneSet
...
spec:
replicas: 4
updateStrategy:
type: InPlaceIfPossible
inPlaceUpdateStrategy:
gracePeriodSeconds: 10
......
spec:
containers:
- name: nginx
image: nginxPerform an upgrade
[root@qd01-stop-k8s-master001 demo]# kubectl apply -f CloneSet.yaml
cloneset.apps.kruise.io/nginx-2 configured
使用 kubectl describe查看升级过程
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 59m default-scheduler 0/22 nodes are available: 22 pod has unbound immediate PersistentVolumeClaims.
Warning FailedScheduling 59m default-scheduler 0/22 nodes are available: 22 pod has unbound immediate PersistentVolumeClaims.
Warning FailedScheduling 59m default-scheduler 0/22 nodes are available: 22 pod has unbound immediate PersistentVolumeClaims.
Normal Scheduled 59m default-scheduler Successfully assigned default/nginx-2-l54dz to qd01-stop-k8s-node007.ps.easou.com
Normal SuccessfulAttachVolume 59m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-e5302eab-a9f2-4a71-a5a3-4cd43205e8a0"
Normal Pulling 58m kubelet Pulling image "nginx:alpine"
Normal Pulled 58m kubelet Successfully pulled image "nginx:alpine" in 6.230045975s
Normal Killing 55s kubelet Container nginx definition changed, will be restarted
Normal Pulling 55s kubelet Pulling image "nginx"
Normal Pulled 26s kubelet Successfully pulled image "nginx" in 29.136659264s
Normal Created 23s (x2 over 58m) kubelet Created container nginx
Normal Started 23s (x2 over 58m) kubelet Started container nginxAs you can see from the output Container nginx definition changed, will be restarted, the Pod is not deleted and rebuilt, but directly updated the image file on the original basis, and restarted the service.
In-place upgrade reduces the deletion and reconstruction links, saving upgrade time and resource scheduling frequency. . .
6.
Partition The semantics of Partition is to keep the number or percentage of the old version of Pod, the default is 0. The partition here does not represent any order number.
在发布过程中设置了 partition:
如果是数字，控制器会将 (replicas - partition) 数量的 Pod 更新到最新版本。
如果是百分比，控制器会将 (replicas * (100% - partition)) 数量的 Pod 更新到最新版本。Now I update the image of the above example to nginx:1.19.6-alpine and set partition=3
kind: CloneSet
metadata:
labels:
app: nginx-2
name: nginx-2
spec:
replicas: 5
updateStrategy:
type: InPlaceIfPossible
inPlaceUpdateStrategy:
gracePeriodSeconds: 10
partition: 3
selector:
matchLabels:
app: nginx-2
template:
metadata:
labels:
app: nginx-2
spec:
containers:
- name: nginx
image: nginx:1.19.6-alpineView Results
Status:
Available Replicas: 5
Collision Count: 0
Label Selector: app=nginx-2
Observed Generation: 6
Ready Replicas: 5
Replicas: 5
Update Revision: nginx-2-7b44cb9c8
Updated Ready Replicas: 2
Updated Replicas: 2
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulUpdatePodInPlace 45m cloneset-controller successfully update pod nginx-2-l54dz in-place(revision nginx-2-5879fd9f7)
Normal SuccessfulUpdatePodInPlace 44m cloneset-controller successfully update pod nginx-2-t49mk in-place(revision nginx-2-5879fd9f7)
Normal SuccessfulUpdatePodInPlace 43m cloneset-controller successfully update pod nginx-2-b5fdd in-place(revision nginx-2-5879fd9f7)
Normal SuccessfulUpdatePodInPlace 43m cloneset-controller successfully update pod nginx-2-jw2zp in-place(revision nginx-2-5879fd9f7)
Normal SuccessfulCreate 22m cloneset-controller succeed to create pod nginx-2-zpp8z
Normal SuccessfulUpdatePodInPlace 5m22s cloneset-controller successfully update pod nginx-2-zpp8z in-place(revision nginx-2-7b44cb9c8)
Normal SuccessfulUpdatePodInPlace 4m55s cloneset-controller successfully update pod nginx-2-jw2zp in-place(revision nginx-2-7b44cb9c8)
[root@qd01-stop-k8s-master001 demo]# kubectl get pod -L controller-revision-hash
NAME READY STATUS RESTARTS AGE CONTROLLER-REVISION-HASH
nginx-2-b5fdd 1/1 Running 1 99m nginx-2-5879fd9f7
nginx-2-jw2zp 1/1 Running 2 99m nginx-2-7b44cb9c8
nginx-2-l54dz 1/1 Running 1 99m nginx-2-5879fd9f7
nginx-2-t49mk 1/1 Running 1 99m nginx-2-5879fd9f7
nginx-2-zpp8z 1/1 Running 1 19m nginx-2-7b44cb9c8From the output information, we can see that Update Revision已经更新为nginx-2-7b44cb9c8, and only two of the Pods have been upgraded.
Since we set partition=3, the controller has only upgraded 2 Pods.
Partition batch gray scale function completes the native Pod upgrade method, making the upgrade more flexible and enabling gray scale online. Awesome. . .
7. In the final demonstration, the
user can pause the release by setting paused to true, but the controller will still manage the number of replicas:
-
First, we change the image in the example to nginx:1.18.0 and set the number of copies to 10. After the modification, update the yaml. The results are as follows:
[root@qd01-stop-k8s-master001 demo]# kubectl get po -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' |sort nginx-2-7lzx9: nginx:1.18.0, nginx-2-b5fdd: nginx:1.18.0, nginx-2-jw2zp: nginx:1.18.0, nginx-2-l54dz: nginx:1.18.0, nginx-2-nknrt: nginx:1.18.0, nginx-2-rgmsc: nginx:1.18.0, nginx-2-rpr5z: nginx:1.18.0, nginx-2-t49mk: nginx:1.18.0, nginx-2-v2bpx: nginx:1.18.0, nginx-2-zpp8z: nginx:1.18.0, - Now we modify the yaml file, modify the image to nginx:alpine to perform the update, run as follows
[root@qd01-stop-k8s-master001 demo]# kubectl get po -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' |sort nginx-2-7lzx9: nginx:1.18.0, nginx-2-b5fdd: nginx:1.18.0, nginx-2-jw2zp: nginx:1.18.0, nginx-2-l54dz: nginx:1.18.0, nginx-2-nknrt: nginx:alpine, nginx-2-rgmsc: nginx:alpine, nginx-2-rpr5z: nginx:alpine, nginx-2-t49mk: nginx:1.18.0, nginx-2-v2bpx: nginx:alpine, nginx-2-zpp8z: nginx:1.18.0, - Now we see that the images of 4 pods have been updated to nginx:alpine, and then we modify the yaml file again and add
paused: truespec: replicas: 10 updateStrategy: paused: true type: InPlaceIfPossible inPlaceUpdateStrategy: gracePeriodSeconds: 10 - Execute apply again, update the yaml, check the update progress again, and find that the pod has not continued to update, and the image update has been suspended
[root@qd01-stop-k8s-master001 demo]# kubectl get po -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' |sort nginx-2-7lzx9: nginx:1.18.0, nginx-2-b5fdd: nginx:1.18.0, nginx-2-jw2zp: nginx:1.18.0, nginx-2-l54dz: nginx:1.18.0, nginx-2-nknrt: nginx:alpine, nginx-2-rgmsc: nginx:alpine, nginx-2-rpr5z: nginx:alpine, nginx-2-t49mk: nginx:1.18.0, nginx-2-v2bpx: nginx:alpine, nginx-2-zpp8z: nginx:1.18.0, - Finally,
paused: truecancel and apply the yaml file again, and the upgrade will continue. . .[root@qd01-stop-k8s-master001 demo]# kubectl get po -o=jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}' |sort nginx-2-7lzx9: nginx:alpine, nginx-2-b5fdd: nginx:alpine, nginx-2-jw2zp: nginx:alpine, nginx-2-l54dz: nginx:alpine, nginx-2-nknrt: nginx:alpine, nginx-2-rgmsc: nginx:alpine, nginx-2-rpr5z: nginx:alpine, nginx-2-t49mk: nginx:alpine, nginx-2-v2bpx: nginx:alpine, nginx-2-zpp8z: nginx:alpine,The above is the demo of the entire release suspension. The advantage of this function is that we can interrupt the upgrade at any time during the upgrade process.
In addition, CloneSet has many features, such as: MaxUnavailable maximum unavailable number, MaxSurge maximum elastic number, upgrade sequence, break-up strategy, life cycle hooks, etc. In view of the length of the article, these features are no longer demonstrated, there is a need You can view the official documentation.