OpenKruise: A cloud-native deployment base for Alibaba Double 11 full-link applications

Source| Alibaba Cloud Native Official Account

Author | Wang Siyu (Wine Toast)

OpenKruise is a cloud-native application automation engine open sourced by Alibaba Cloud in June 2019. It is essentially an application load project based on the Kubernetes standard. It can be used with native Kubernetes and is used to manage application containers, sidecars, image distribution, etc. Provide more powerful and efficient capabilities, so as to solve the problems of large-scale operation and maintenance and large-scale site construction of applications on Kubernetes in different dimensions, including deployment, upgrades, elastic expansion and contraction, Qos adjustment, health checks, and migration Repair and so on.

• OpenKruise official website :https://openkruise.io/

• GitHub project address :https://github.com/openkruise/kruise

Kruise is the homonym of Cruise, "K" for Kubernetes, which means navigation and automatic navigation of applications on Kubernetes. It is full of Alibaba's many years of large-scale application deployment, release and management best practices, and Alibaba Cloud Kubernetes serving thousands of customers Demand precipitation.

OpenKruise: Alibaba Double 11 Full Link Application Cloud Native Deployment Base

In the overall cloud nativeization process of the Alibaba economy, Alibaba's technical team has gradually precipitated a set of technical concepts and best practices that closely adhere to the upstream community standards and adapt to the scale of the Internet. Among them, the most important thing is undoubtedly how to automate the release, operation and management of applications. As a result, the Alibaba Cloud container team will feed back these capabilities to the community through OpenKruise, in order to guide the industry's best practices for cloud nativeization and avoid detours.

On Double 11 this year, Alibaba achieved full cloud nativeization of its core system. As of Double 11 in 2020, Alibaba has run nearly 100,000 OpenKruise workloads and manages millions of containers.

1. Over 95% of the internal code of OpenKruise version comes from community warehouse

The following figure shows the relationship between the OpenKruise version running inside Alibaba and the open source version:

As can be seen from the above figure: OpenKruise on Github is our main upstream warehouse, while the internal downstream warehouse only implements a very small number of internal coupling functions based on the public interface (this part of the code only occupies less than 5%), which means More than 95% of the code in OpenKruise run internally by Alibaba comes from community warehouses.

There are two points worth explaining:

• All common capabilities will be developed and submitted directly based on the open source warehouse, and then synchronized to the internal environment.

• Every line of code contributed by community members to OpenKruise will run in Ali's internal environment.

2. Automating application workload management on Kubernetes

Students who are doing upper-level business may lack the concept of "workload". Here is a brief introduction. I don’t know if you have been curious about how to implement each application expansion and release operation? In a cloud-native environment, we describe the deployment requirements of the application (the number of machines required, the mirror version, etc.) in a final state-oriented way, as shown in the figure below:

Application workload (workload) mainly refers to this YAML definition and the corresponding controller:

When the application expands and shrinks, the PaaS (Operation and Maintenance Platform) will modify the number of required machines in the above YAML object (for example, expand 10 machines to 110, and then shrink 5 machines to 105), and then the controller will follow the workload The desired number is adjusted to the number of Pods (containers) actually running. When the application triggers a release or rollback, PaaS (Operation and Maintenance Platform) will modify the image version and release strategy in the above YAML object, and the controller will rebuild all managed Pods (containers) according to the specified release strategy. Expected version (this is just a simplified description for easy understanding, the actual working mechanism will be much more complicated).

In other words, the application workload (workload) manages the life cycle of all containers in the application. Not only does the application expansion, shrinkage, and release depend on the work of the workload, but the workload is also responsible for continuously maintaining the number of Pods (containers) during application runtime to ensure that there are continuous running instances that meet the expected number. If a host machine fails and the application instance above is evicted, the workload will immediately expand a new container for the application.

3. Double 11 core applications are fully deployed based on OpenKruise

As Alibaba's economy goes to the cloud, e-commerce businesses and middleware applications related to Double Eleven entities have all migrated to the cloud native environment, using OpenKruise's application load capacity for deployment. OpenKruise provides a variety of different types of workloads to support different deployment methods:

• CloneSet: (Stateless application) This is the largest part, and most pan-e-commerce businesses are deployed and released through CloneSet.
• Advanced StatefulSet: (Stateful Application) is currently mainly used for the deployment of middleware in cloud native environments.
• SidecarSet: (sidecar lifecycle management) The defined sidecar container can be dynamically injected into the newly created Pod. The operation and maintenance container and mesh container on the cloud are added to the business Pod through this mechanism.
• Advanced DaemonSet: Deploy the host-level daemon to all nodes, including various basic components used to configure network and storage for business containers.

Therefore, we see that from the upper e-commerce business to the middleware, to the operation and maintenance container, and basic components, the entire upstream and downstream links rely on the workload provided by OpenKruise for deployment and operation. Whether it is the number of machines, version management, emergency expansion, release and other operations when the application is running, OpenKruise is maintaining it all the time.

As you can imagine, what happens if OpenKruise fails?

• If only the controller is hung up, the application expansion, shrinking, and publishing operations all fail.
• And if there are major bugs in the controller logic, such as incorrect calculation of the number or version number, it may even cause the business container to be deleted or upgraded to the wrong version on a large scale.

Of course, in response to the above-mentioned high-risk situations, we have taken many heavy protective measures to ensure the stability and availability of the business.

This is Alibaba's deployment base on the cloud, and OpenKruise is almost responsible for the deployment, management and operation and maintenance of the entire Double 11 business.

4. Main abilities

Where does OpenKruise come from? Or what problems or requirements prompted the birth of OpenKruise?

When going to the cloud became the trend and when the cloud native gradually became the standard, we found that the workload capabilities natively provided by Kubernetes simply could not meet the needs of Alibaba's ultra-large-scale business scenarios:

• When the application is released, all containers must be rebuilt: it is almost unacceptable for us. During the peak release period, if Alibaba’s large-scale applications are rebuilt on a large scale, this is whether for the business itself or other schedulers, middleware, Network/storage components are a catastrophic pressure.
• Stateless application load (Deployment) cannot upgrade containers in grayscale.
• StatefulSet cannot upgrade containers in parallel.
• There are many more, not to list them all...

In this context, OpenKruise appeared. Through either new development (CloneSet, SidecarSet) or enhanced compatibility (Advanced StatefulSet, Advanced DaemonSet), the upper-level business can finally land on cloud native smoothly.

The first feature of OpenKruise is "in-place upgrade". With this ability, we can finally make application releases do not need to rebuild the container, but only upgrade the image that needs to be updated on the original Pod. The benefits of this are too many:

• The publishing efficiency has been greatly improved. According to incomplete statistics, in-situ upgrades are at least 80% faster than full reconstruction upgrades in most business scenarios: it not only saves the time-consuming scheduling, network allocation, and remote disk allocation, but also pulls new versions. When mirroring, it all benefits from the old mirror on the node, and only a small incremental layer needs to be pulled.
• The IP remains unchanged before and after the release, the Pod network continues during the upgrade process, and all containers in the Pod except for the container being upgraded have been kept running normally.
• The volume remains unchanged, and the mount device of the original container is completely reused.
• To ensure the cluster certainty, the cluster topology after the full-link stress test is passed provides a guarantee for the big promotion.

Of course, in addition to this, we have also enhanced many other advanced capabilities to meet many business requirements for large-scale scenarios. This article will not introduce them one by one, but the following figure shows that the OpenKruise and Kubernetes native application load targets no Function comparison of state and stateful applications:

OpenKruise has officially entered the CNCF Sandbox

On November 11, 2020, at this special point in time, Alibaba technicians ushered in another major event: After voting by all members of the CNCF Technical Supervision Committee, they unanimously agreed to officially promote Alibaba Cloud's open source OpenKruise to a CNCF hosting project.

As mentioned at the beginning, OpenKruise has completed the community open source, and the internal and external versions are almost identical. In addition, we have also provided OpenKruise to the application catalog of Alibaba Cloud Container Service. Any customer on the public cloud can install and use OpenKruise with one click, and truly realize OpenKruise in Alibaba Group’s internal business, cloud products, and open source communities. The " Trinity ". At present, customers who use OpenKruise on ACK mainly include Douyu TV, Shentong, Youzan, etc., and companies such as Ctrip and Lyft in the open source community are also OpenKruise users and contributors.

OpenKruise opens up the cloud-native application load capacity based on Alibaba’s ultra-large-scale scenarios. It not only supplements the important section of the cloud-native community to expand application load, but also provides customers on the cloud with Alibaba’s years of application deployment management experience and The best practice results of the cloud native process. Since the official open source date, the OpenKruise project has established several key milestones: 

• Maintainer 5 members from Alibaba, Tencent, Lyft 
• 44 contributors
  • Domestic: Alibaba Cloud, Ant Group, Ctrip, Tencent, Pinduoduo...
  • Foreign: Microsoft, Lyft, Spectro Cloud, Dsicord...
• 1900+ GitHub Stars
• 300+ Forks

In the future, the focus of OpenKruise includes but not limited to the following goals:

• Continue to export the general cloud native application automation capabilities accumulated in Alibaba, and follow a sustainable three-in-one development strategy.
• In-depth mining of application load requirements in subdivisions. For example, we are exploring pooling capabilities for FaaS scenarios.
• Closer integration with open source products in other related fields, such as OAM/KubeVela, etc., to create a more complete cloud-native application system.

Welcome to join the OpenKruise family

If you have any suggestions for the development of OpenKruise, please post in the comment area below. In addition, you can search for the DingTalk group number: 23330762 to enter the "OpenKruise Community Exchange DingTalk Group". We sincerely welcome every open source enthusiast to participate in the construction of OpenKruise and jointly create the most mature and largest-scale applications in the cloud native field. Automation engine.