Fighting k8s again (12): Pod upgrade and rollback under the guidance of Deployment

insert image description here

Article directory

Pod upgrades and rollbacks

When a service in the cluster needs to be upgraded , we need to stop all Pods currently associated with the service , then download the new version image and create a new Pod . This becomes a challenge if the cluster size is large , and stopping all and then escalating can result in service unavailability for an extended period of time .

Kubernetes provides a rolling upgrade feature to solve the above problems.

If a Pod is created through a Deployment , the user can modify the Deployment's Pod definition ( spec.template ) or image name at runtime and apply it to the Deployment object, and the system can complete the automatic update operation of the Deployment. If an error occurs during the update process, you can also restore the Pod's version with a rollback operation.

Deployment upgrade

# nginx-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

[root@k8s-master01 pod]# kubectl apply -f  nginx-deployment.yaml
deployment.apps/nginx-deployment created
[root@k8s-master01 pod]# kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
nginx-deployment-5bf87f5f59-h2llz   1/1     Running   0          3s    10.244.2.44   k8s-node01   <none>           <none>
nginx-deployment-5bf87f5f59-mlz84   1/1     Running   0          3s    10.244.2.45   k8s-node01   <none>           <none>
nginx-deployment-5bf87f5f59-qdjvm   1/1     Running   0          3s    10.244.1.25   k8s-node02   <none>           <none>

Now that the Pod image needs to be updated to Nginx:1.9.1 , we can set the new image name for the Deployment via the kubectl set image command:

kubectl set image deployment/nginx-development nginx=nginx:1.9.1

Another way to update is to use the kubectl edit command to modify the Deployment's configuration , changing spec.template.spec.containers[0].image from Nginx:1.7.9 to Nginx:1.9.1:

kubectl edit deployment/nginx-deployment

Once the image name (or Pod definition ) is modified , the system will be triggered to complete the rolling upgrade of all running Pods of the Deployment

You can use the kubectl rollout status command to view the update process of the Deployment:

Check the image used by Pod, which has been updated to Nginx:1.9.1:

[root@k8s-master01 pod]# kubectl describe pod nginx-deployment-678645bf77-4ltnc
Image:          nginx:1.9.1

Use the kubectl describe deployments/nginx-deployment command to closely observe the deployment's update process .

Events:
  Type    Reason             Age                   From                   Message
  ----    ------             ----                  ----                   -------
  Normal  ScalingReplicaSet  6m4s                  deployment-controller  Scaled down replica set nginx-deployment-5bf87f5f59 to 2
  Normal  ScalingReplicaSet  6m4s                  deployment-controller  Scaled up replica set nginx-deployment-678645bf77 to 2
  Normal  ScalingReplicaSet  6m3s                  deployment-controller  Scaled up replica set nginx-deployment-678645bf77 to 3
  Normal  ScalingReplicaSet  6m1s                  deployment-controller  Scaled down replica set nginx-deployment-5bf87f5f59 to 0
  Normal  ScalingReplicaSet  4m55s                 deployment-controller  Scaled up replica set nginx-deployment-5bf87f5f59 to 1
  Normal  ScalingReplicaSet  4m54s                 deployment-controller  Scaled down replica set nginx-deployment-678645bf77 to 2
  Normal  ScalingReplicaSet  4m53s (x2 over 8m4s)  deployment-controller  Scaled up replica set nginx-deployment-5bf87f5f59 to 3
  Normal  ScalingReplicaSet  103s (x2 over 6m5s)   deployment-controller  Scaled up replica set nginx-deployment-678645bf77 to 1
  Normal  ScalingReplicaSet  101s (x2 over 6m3s)   deployment-controller  Scaled down replica set nginx-deployment-5bf87f5f59 to 1
  Normal  ScalingReplicaSet  100s (x7 over 4m54s)  deployment-controller  (combined from similar events): Scaled down replica set nginx-deployment-5bf87f5f59 to 0

When the Deployment was initially created, the system created a ReplicaSet (nginx-deployment-5bf87f5f59) and created 3 Pod replicas according to the user's needs. When the Deployment is updated, the system creates a new ReplicaSet (nginx-deployment-678645bf77), expands its replica count to 1, and then shrinks the old ReplicaSet to 2. After that, the system continues to adjust the old and new ReplicaSets one by one according to the same update strategy. Finally, the new ReplicaSet runs 3 copies of the Pod of the new version, and the number of copies of the old ReplicaSet is reduced to 0. as the picture shows.

[root@k8s-master01 pod]# kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-5bf87f5f59   0         0         0       17m
nginx-deployment-678645bf77   3         3         3       15m

During the entire upgrade process, the system will ensure that at least two Pods are available, and at most four Pods are running at the same time. This is done by Deployment through a complex algorithm. Deployments need to ensure that only a certain number of Pods may be unavailable during the entire update process. By default, Deployment ensures that the total number of Pods available is at least the required number of replicas (DESIRED) minus 1, which is at most 1 unavailable (maxUnavailable=1). Deployments also need to ensure that the total number of Pods during the entire update process does not exceed the required number of replicas by too much. By default, Deployment ensures that the total number of Pods is at most 1 more than the required number of Pods, which is at most 1 surge value (maxSurge=1). Starting with Kubernetes version 1.6, the default values of maxUnavailable and maxSurge will be updated from 1, 1 to 25%, 25% of the desired number of replicas.

In this way, during the upgrade process, Deployment can ensure that the service is not interrupted, and the number of replicas is always maintained at the number specified by the user (DESIRED).

A description of the update strategy follows.

In the definition of Deployment, you can specify the strategy for Pod update through spec.strategy. Currently, two strategies are supported: Recreate (rebuild) and RollingUpdate (rolling update). The default value is RollingUpdate. The RollingUpdate strategy was used in the previous example.

Recreate: Set spec.strategy.type=Recreate, which means that when Deployment updates Pods, it will first kill all running Pods and then create new Pods.
RollingUpdate: Set spec.strategy.type=RollingUpdate, which means that Deployment will update Pods one by one in a rolling update manner. At the same time, the rolling update process can be controlled by setting two parameters (maxUnavailable and maxSurge) under spec.strategy.rollingUpdate.

The following descriptions of the two main parameters during rolling update are as follows.

spec.strategy.rollingUpdate.maxUnavailable: Used to specify the upper limit of the number of Pods in the Unavailable state of the Deployment during the update process. The value of maxUnavailable can be an absolute value (such as 5) or a percentage of the number of replicas that the Pod expects (such as 10%). If it is set as a percentage, the system will first calculate the absolute value (integer) by rounding down. . When another parameter maxSurge is set to 0, maxUnavailable must be set to an absolute value greater than 0 (from Kubernetes 1.6, the default value of maxUnavailable has been changed from 1 to 25%). For example, when maxUnavailable is set to 30%, the old ReplicaSet can shrink the number of replicas to 70% of the total number of required replicas immediately when the rolling update begins. Once a new Pod is created and ready, the old ReplicaSet will be further scaled down, and the new ReplicaSet will continue to expand. During the whole process, the system can ensure that the total number of Pods in the available state accounts for at least 70% of the total number of expected replicas of the Pod.
spec.strategy.rollingUpdate.maxSurge: It is used to specify the maximum value of the part where the total number of Pods exceeds the expected number of copies of Pods during the process of Deployment updating Pods. The value of maxSurge can be an absolute value (eg 5) or a percentage of the Pod's desired number of replicas (eg 10%). If set as a percentage, the system will first calculate the absolute value (integer) by rounding up. Starting with Kubernetes 1.6, the default value of maxSurge was changed from 1 to 25%. For example, when the value of maxSurge is set to 30%, the new ReplicaSet can expand the number of replicas immediately when the rolling update starts. It only needs to ensure that the sum of the number of Pod replicas of the old and new ReplicaSet does not exceed 130% of the expected number of replicas. Can. Once the old Pod is killed, the new ReplicaSet will be further expanded. During the whole process, the system can ensure that the sum of the total number of Pod replicas of the old and new ReplicaSets does not exceed 130% of the required number of replicas at any time.

Here you need to pay attention to the situation of multiple updates (Rollover). If the last update of the Deployment is in progress and the user initiates the update operation of the Deployment again, the Deployment will create a ReplicaSet for each update, and each time a new ReplicaSet is created successfully, it will increase the number of Pod replicas one by one, and at the same time Stop the expansion (update) of the ReplicaSet that was expanding before, add it to the list of ReplicaSets of the old version, and then start the operation of scaling down to 0.

For example, suppose we create a Deployment, which starts to create 5 Pod replicas of Nginx:1.7.9. When this Pod creation action has not been completed, we update the Deployment again, and the Pod template will be changed when the number of replicas remains unchanged. The image in the file is modified to Nginx:1.9.1, and assuming that Deployment has created 3 Pod copies of Nginx:1.7.9 at this time, Deployment will immediately kill the 3 Nginx:1.7.9 Pods that have been created, and start Create the Nginx:1.9.1 Pod. Deployment will not wait for 5 Pods of Nginx:1.7.9 to be created before updating.

You also need to be careful about updating the Deployment's Label Selector. In general, updating the Deployment's label selector is discouraged, as doing so will cause the Deployment's list of Pods to change and possibly conflict with other controllers. If you must update the tag selector, be careful to ensure that no other issues arise. A note about updates to the Deployment tag selector is as follows.

1) When adding a selector label, you must synchronously modify the label of the Pod configured by the Deployment, and add a new label to the Pod, otherwise the update of the Deployment will report a verification error and fail:

Adding label selectors is not backward compatible, which means that new label selectors will not match and use ReplicaSets and Pods created with old selectors, so adding selectors will result in all old ReplicaSets and ReplicaSets created by old ReplicaSets Pods are orphaned (not automatically deleted by the system and not controlled by the new ReplicaSet).

After adding a new label to the label selector and pod template (using the kubectl edit deployment command), the effect is as follows:

# kubectl get rs
NAME                            DESIRED     CURRENT     READY   AGE
nginx-deployment-4087004473     0           0           0       52m
nginx-deployment-3599678771     3           3           3       1m
nginx-deployment-3661742516     3           3           3       2s

You can see 3 new Pods created by the new ReplicaSet (nginx-deployment-3661742516):

# kubectl get pods
NAME                                READY   STATUS  RESTARTS    AGE
nginx-deployment-3599678771-01h26   1/1     Running 0           2m
nginx-deployment-3599678771-57thr   1/1     Running 0           2m
nginx-deployment-3599678771-s8p21   1/1     Running 0           2m
nginx-deployment-3661742516-46djm   1/1     Running 0           52s
nginx-deployment-3661742516-kws84   1/1     Running 0           52s
nginx-deployment-3661742516-wq30s   1/1     Running 0           52s

(2) Updating the tag selector, that is, changing the key or value of the tag in the selector, will also produce a similar effect to adding a selector tag.

(3) Delete the label selector, that is, delete one or more labels from the Deployment's label selector. The Deployment's ReplicaSet and Pod will not be affected in any way. Note, however, that deleted labels will still exist on existing Pods and ReplicaSets.

Rollback of Deployment

Sometimes (such as when the new Deployment is unstable) we may need to roll back the Deployment to an older version. By default, the release history of all Deployments is kept in the system, so that we can roll back at any time (the number of history can be configured).

Assuming that when updating the Deployment image, the container image name was mistakenly set to Nginx:1.91 (a non-existing image):

kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1

At this time, the deployment process of Deployment will be stuck:

kubectl rollout status deployments nginx-deployment
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...

Check the history of Deployment deployments

[root@k8s-master01 pod]# kubectl rollout history deployment/nginx-deployment
deployment.apps/nginx-deployment 
REVISION  CHANGE-CAUSE
5         <none>
6         <none>

Note that by using the --record parameter when creating a Deployment, you can see the commands used for each version in the CHANGE-CAUSE column. In addition, Deployment's update operation is triggered when Deployment is doing a rollout, which means that a new revision will be created if and only if the Deployment's Pod template (i.e. spec.template) is changed, such as updating the template Label or container image. Other update operations (such as expanding the number of replicas) will not trigger the update operation of the Deployment, which also means that when we roll back the Deployment to the previous version, only the Pod template part of the Deployment will be modified.
If you need to view the details of a specific version, you can add the --revision= <N>parameter:

To view information about a specific version, you can add the --revision= <N>parameter:

[root@k8s-master01 pod]# kubectl rollout history deployment/nginx-deployment --revision=5
deployment.apps/nginx-deployment with revision #5
Pod Template:
  Labels:   app=nginx
    pod-template-hash=5bf87f5f59
  Containers:
   nginx:
    Image:  nginx:1.7.9
    Port:   80/TCP
    Host Port:  0/TCP
    Environment:    <none>
    Mounts: <none>
  Volumes:  <none>

Now we decide to undo this release and roll back to the previous deployment version:

[root@k8s-master01 pod]# kubectl rollout undo deployment/nginx-deployment
deployment.apps/nginx-deployment rolled back

The deployment version number to roll back to can also be specified with the --to–revision parameter:

[root@k8s-master01 pod]# kubectl rollout undo deployment/nginx-deployment --to-revision=6
deployment.apps/nginx-deployment rolled back

Pause and resume Deployment's deployment operations, where complex modifications have been made

For a complex Deployment configuration modification, in order to avoid frequently triggering the Deployment update operation, you can first suspend the Deployment update operation, then modify the configuration, and then resume the Deployment, triggering the complete update operation at one time, so as to avoid unnecessary Deployment updates. operate

Take the Nginx created earlier as an example:

[root@k8s-master01 pod]# kubectl get deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           42m

[root@k8s-master01 pod]# kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-5bf87f5f59   0         0         0       43m
nginx-deployment-678645bf77   3         3         3       41m

Pause the update operation of the deployment through the kubectl rollout pause command

[root@k8s-master01 pod]# kubectl rollout pause deployment/nginx-deployment
deployment.apps/nginx-deployment paused

Then modify the image information of the Deployment:

kubectl set image deployment/nginx-deployment nginx=nginx：1.9.1

Look at the Deployment's history and find that no new Deployment deployment actions are triggered

[root@k8s-master01 pod]# kubectl rollout history deployment/nginx-deployment
deployment.apps/nginx-deployment 
REVISION  CHANGE-CAUSE
11        kubectl set image deployment/nginx-deployment nginx=nginx:1.7.9 --record=true
12        kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1 --record=true

After a Deployment is paused, configuration updates can be made as many times as needed. For example, update the container's resource limits again:

[root@k8s-master01 pod]# kubectl set resources deployment nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi
deployment.apps/nginx-deployment resource requirements updated

Finally, resume the deployment operation for this Deployment:

kubectl rollout resume deploy nginx-deployment

You can see that a new ReplicaSet is created

[root@k8s-master01 pod]# kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-56bbb744cc   3         3         3       12s
nginx-deployment-5bf87f5f59   0         0         0       50m
nginx-deployment-678645bf77   0         0         0       48m

Looking at the event information of the Deployment, you can see that the Deployment has completed the update:

[root@k8s-master01 pod]# kubectl describe deployment nginx-deployment
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:      nginx:1.9.1
    Port:       80/TCP
    Host Port:  0/TCP
    Limits:
      cpu:        200m
      memory:     512Mi

Note that a suspended Deployment cannot be rolled back until the suspended Deployment is resumed.

Use the kubectl rolling-update command to complete the rolling upgrade of RC

For the rolling upgrade of RC, kubernetes also provides a kubectl rolling-update command for implementation. This command creates a new RC, and then automatically controls the number of Pod replicas in the old RC to decrease to 0, while the number of Pod replicas in the new RC increases from 0 to the target value one by one to complete the Pod upgrade. Note that the system requires the new RC to be in the same namespace as the old RC.

# tomcat-controller-v1.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: tomcat
  labels:
    name: tomcat
    version: v1
spec:
  replicas: 2
  selector:
    name: tomcat
  template:
    metadata:
      labels:   #可以自己定义key-value
        name: tomcat
    spec:
      containers:
      - name: tomcat
        image: tomcat:6.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        env:             #环境变量
        - name: GET_HOSTS_FROM
          value: dns

[root@k8s-master01 pod]# kubectl apply -f tomcat-controller-v1.yaml

[root@k8s-master01 pod]# kubectl get rc -o wide
NAME     DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES       SELECTOR
tomcat   2         2         2       16s   tomcat       tomcat:6.0   name=tomcat

Do RC rolling upgrades

# tomcat-controller-v2.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: tomcat2
  labels:
    name: tomcat
    version: v2
spec:
  replicas: 2
  selector:
    name: tomcat2
  template:
    metadata:
      labels:   #可以自己定义key-value
        name: tomcat2
    spec:
      containers:
      - name: tomcat
        image: tomcat:7.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        env:             #环境变量
        - name: GET_HOSTS_FROM
          value: dns

[root@k8s-master01 pod]# kubectl rolling-update tomcat -f tomcat-controller-v2.yaml

There are two points to note in the configuration file:

The name of the RC cannot be the same as the name of the old RC.
There should be at least one Label in the selector that is different from the Label of the old RC to identify it as the new RC. In this example, a new Label named version is added to distinguish it from the old RC. That is to ensure that the same Key has at least one Value but not one.

After all new Pods are started, all old Pods are also destroyed, thus completing the update of the container cluster.

Another method is to use the kubectl rolling-update command directly without using the configuration file, and add the --image parameter to specify the name of the new version of the image to complete the rolling upgrade of the Pod

kubectl rolling-update tomcat --image=tomcat:6.0

Unlike using a configuration file, the result of the execution is that the old RC is deleted and the new RC will still use the name of the old RC.

It can be seen that kubectl completes the entire RC update by creating a new version of the Pod, stopping an old version Pod, and so on.

It can be seen that kubectl adds a Label with the key "deployment" to RC (the name of this key can be modified by the --deployment-label-key parameter), and the value of Label is the value of the content of RC after Hash calculation, which is quite In the signature, it is easy to compare whether the Image name and other information in the RC have changed.

If the configuration is found to be incorrect during the update process, the user can interrupt the update operation and complete the rollback of the Pod version by executing kubectl rolling-update --rollback:

kubectl rolling-update tomcat --image=tomcat:6.0 --rollback

It can be seen that the rolling upgrade of RC does not have functions such as the historical record of Deployment in the process of application version upgrade, and the fine control of the number of new and old versions. During the evolution of Kubernetes, RC will be gradually replaced by RS and Deployment. It is recommended that users give priority to Consider using Deployment to complete the deployment and upgrade of Pods.

Update policies for other managed objects

Since Kubernetes version 1.6, the update strategy for DaemonSet and StatefulSet also introduces a rolling upgrade similar to Deployment, which automatically completes the version upgrade of the application through different strategies.

1. DaemonSet update strategy

Currently, there are two types of upgrade strategies for DaemonSet: OnDelete and RollingUpdate.

(1) OnDelete: DaemonSet's default upgrade strategy, consistent with Kubernetes 1.5 and earlier. When OnDelete is used as the upgrade strategy, after the new DaemonSet configuration is created, the new Pod will not be automatically created, and the new operation will not be triggered until the user manually deletes the old version of the Pod.

(2) RollingUpdate: Introduced since Kubernetes version 1.6. When using RollingUpdate as the upgrade strategy to update the DaemonSet, the old version of the Pod will be automatically killed, and then automatically create a new version of the DaemonSet Pod. The whole process is as controllable as the rolling upgrade of a normal Deployment. However, there are two points that are different from the rolling upgrade of ordinary Pods: First, Kubernetes currently does not support viewing and managing the update history of DaemonSet; second, DaemonSet rollback (Rollback) cannot be implemented directly through the kubectl rollback command like Deployment. It must be achieved by resubmitting the old version of the configuration.

2. Update strategy of StatefulSet

Since Kubernetes version 1.6, the update strategy for StatefulSet is gradually aligned with the update strategy of Deployment and DaemonSet, and strategies such as RollingUpdate, Paritioned, and OnDelete will also be implemented to ensure that each Pod in the StatefulSet is updated in an orderly and one-by-one manner, and can Keep the update history, and can also roll back to a historical version.