CNCF official ambassador Zhang Lei: Is Kubernetes a "database"?

Author | Zhang Lei, Alibaba Cloud senior technical expert, CNCF official ambassador, CNCF application delivery co-chair, Kubernetes project senior maintainer

Recently, there is a discussion about "Kubernetes is the new database" in the Kubernetes community, which has attracted a lot of attention. Of course, the more precise meaning of this discussion is that the Kubernetes project itself works like a database , not that you should use Kubernetes as a database.

At first glance, this "Kubernetes is a database" argument is rather bizarre. After all, the working principles of Kubernetes, such as the controller mode, declarative API, etc., as we usually call it, do not seem to be directly related to the "database". But in fact, there is a very essential meaning behind this statement. The reason for this has to start with one of the most basic theories in the Kubernetes project.

Kubernetes Declarative Application Management Theoretical Basis

When we discuss Kubernetes, we often refer to a concept called "Declarative Application Management". In fact, this is also a design that makes the Kubernetes project different from all other infrastructure projects. It is a unique capability of Kubernetes. So, have you ever thought about the specific performance of declarative application management in Kubernetes? Woolen cloth?

1. Declarative application management is more than "Declarative-style APIs"

If we review the core working principle of Kubernetes, it is not difficult to find the fact that most of the functions in Kubernetes, whether kubelet executes containers, kube-proxy executes iptables rules, or kube-scheduler executes Pod scheduling, and The process of Deployment management ReplicaSet, etc., in fact, the overall design follows the "controller" pattern that we often emphasize. That is: the user expresses the desired state he wants through YAML files, which is the final state (whether it is network or storage), and then various components of Kubernetes will make the state of the entire cluster approach the final state declared by the user , and finally reached a complete agreement between the two. The process of the actual state gradually approaching the desired state is called reconcile. And the same principle is also the core work of Operator and custom Controller.

This working form of approaching two states by driving the controller to execute reconcile through a declarative description file is the most intuitive embodiment of declarative application management. It should be noted that this process actually includes two meanings:

• ** The desired state of the declarative description. **This description must be the final state that the user wants in the strict sense. If you fill in an intermediate state in this description, or you want to dynamically adjust the desired state, it will destroy the accurate execution of this declarative semantics ;

• **Reconcile-based state approximation process. **The existence of the Reconcile process ensures the theoretical correctness of the system state being consistent with the final state. To be precise, the Reconcile process keeps executing the cycle of "check -> Diff -> execute", so that the system can always take necessary actions on the direct difference between the state of the system itself and the final state. By contrast, simply having a declarative description is not sufficient. It's easy to understand that the system achieved the desired state you wanted when you first submitted the description, and it doesn't mean, nor guarantee, that it will be the same an hour later. Many people confuse "Declarative Application Management" with "Declarative-style API", but they don't have a correct understanding of the necessity of Reconcile.

You may be curious, what are the benefits of adopting this declarative application management system for Kubernetes?

2. The essence of declarative application management: Infrastructure as Data

In fact, the theoretical basis behind the declarative application management system is an idea called Infrastructure as Data (IaD) . This idea believes that infrastructure management should not be coupled to a certain programming language or configuration, but should be pure, formatted, system-readable data that can fully characterize the system expected by users condition.

Note: Infrastructure as Data is sometimes called Configuration as Data, which means the same thing behind it.

The advantage of this is that any time I want to operate on the infrastructure, it is ultimately equivalent to "adding, deleting, modifying, and checking" these data. And more importantly, the way I "add, delete, modify, and check" these data has nothing to do with the infrastructure itself. So, my process of interacting with an infrastructure is not tied to a programming language, a remote call protocol, or an SDK. As long as I can generate the "data" in the corresponding format, I can do whatever I like with the infrastructure.

This benefit is embodied in Kubernetes, that is, if I want to do anything on Kubernetes, I only need to submit a YAML file, and then add, delete, modify and check this YAML file. Instead of having to use the Kubernetes project's Restful API or SDK. The content in this YAML file is actually the Data corresponding to the IaD system of Kubernetes.

Therefore, Kubernetes has defined all its functions as so-called "API objects" since its birth, which is actually defined as a piece of Data. In this way, Kubernetes users can achieve their desired goals by adding, deleting, modifying and checking these data, rather than being bound to a specific language or SDK. More importantly, compared to proprietary, imperative APIs or SDKs, declarative data in YAML can more easily shield the underlying implementation, making it easier to connect and integrate existing infrastructure capabilities , which is actually a secret weapon for the Kubernetes ecosystem to flourish at an amazing speed today: the declarative API and controller mode brought by the IaD idea makes the whole community more willing to write plug-ins and connect various capabilities for Kubernetes, and these The versatility and portability of plugins and capabilities is also very high, which is beyond the reach of other projects such as Mesos and OpenStack. It can be said that IaD is the core combat power of Kubernetes to achieve the goal of "The Platform for Platform".

At this point, everyone probably understands: the data in this IaD design is actually a declarative Kubernetes API object; and the control loop in Kubernetes is to ensure that the system itself can always be described by these data. state is always the same. From this point of view, Kubernetes is essentially a tuning system that expresses the set value of the system with data (Data) and maintains the system at the set value through the actions of the controller (Controller).

Wait a minute, this "keep the system at set point" theory sounds familiar?

In fact, the basic course behind Kubernetes, probably most readers with engineering background have learned, it is called "Control Theory".

Do you feel enlightened?

After understanding the essence of Kubernetes, we may look back at some of the original settings that were more difficult to understand, and it may be easier to understand some essential things.

For example, the reason why we have to write so many YAML files when using Kubernetes today is actually because we need a way to submit Data to the Kubernetes control system. In this process, YAML is just a carrier for writing Data in a format that humans can format. If we make an analogy, then YAML is like the "Tian Zi Ge" in our homework books when we were young, and the words written in the "Tian Zi Ge" are the data that Kubernetes really cares about and the core of the operation of the entire system.

Attentive readers should have thought at this point that since Kubernetes needs to process these data, shouldn't the data itself also have a fixed "format" so that Kubernetes can parse them? Yes, the format here is called the Schema of the API object in Kubernetes. If you often write custom Controller, you may have a deep sense of this Schema: CRD is a special API object specially used to define Schema.

YAML engineer? No, you are a database engineer!

**The nature of the IaD of Kubernetes above determines that its working principle is actually more similar to a "database" than a distributed system in the traditional sense. **This difference is also a fundamental reason for the steep learning cost of Kubernetes.

From this perspective, the various API objects exposed by Kubernetes for you are actually tables with predefined schemas. And the YAML files that we have racked our brains to write are actually the additions, deletions, and changes (CURD) to the data (Data) in these tables. The YAML tool itself, like SQL, is a tool and carrier that helps you operate on the data in the database. The only difference from traditional databases is that after Kubernetes gets the data, it does not aim to persist the data, but hopes to use the data to drive the Controller to perform certain operations, so as to make the entire system The state is gradually adjusted to be consistent with the final state declared in the data, which brings us back to the "control theory" part we talked about earlier.

It is precisely because the whole system of Kubernetes revolves around the setting of "data", the first-class citizen, that "writing and manipulating YAML files" has become almost the only daily work of Kubernetes engineers. However, after understanding the idea of ​​IaD introduced in this article today, you can actually compare yourself to a "database engineer", and this TItle is indeed more appropriate than a "YAML engineer".

The "view layer" of a Kubernetes project

As mentioned above, if you re-examine the Kubernetes design from a "database" perspective, it is not difficult to find that there are very subtle ideas behind many of Kubernetes' designs. for example:

• Data Model - Various API Objects and CRD Mechanisms of Kubernetes
• Data Interception Check and Modification Mechanism - Kubernetes Admission Hook
• Data Driven Mechanism - Kubernetes Controller/Operator
• Data Monitoring Change and Indexing Mechanism - Informer Mechanism of Kubernetes
• ……

On the other hand, as the Kubernetes infrastructure becomes more and more complex, and there are more and more third-party plug-ins and capabilities, community maintainers have also found that the built-in “data table” of the “database” of Kubernetes, in terms of scale and complexity, are experiencing explosive growth. Therefore, the Kubernetes community has been discussing how to design a "data view (View)" for Kubernetes for a long time, namely:

The benefits of such a "view layer" built on top of Kubernetes' built-in API resources to Kubernetes users are very similar to the "view" in the database, such as:

1. Simplify and change data format and presentation

The view layer of Kubernetes needs to be able to expose simpler, abstracted application layer API objects to R&D and O&M, instead of the original infrastructure layer API objects. As for how to define a view layer object, the degree of freedom should be completely in the hands of the user, and it does not need to be constrained to the Schema of the underlying Kubernetes built-in object.

2. Simplify complex data manipulation (simplify SQL)

View layer objects generated after abstraction not only need to be simpler in UI, but also need to be able to define and manage very complex underlying Kubernetes resource topologies, thereby reducing the complexity and mental burden of managing Kubernetes applications for users.

3. Protect the underlying datasheet

R&D and O&M directly operate on view layer objects, so the underlying Kubernetes primitive objects are protected. This allows these Kubernetes primitives to be arbitrarily changed and upgraded without the user's awareness.

4. Reuse data operations (reuse SQL)

Since the view layer objects are completely decoupled from the underlying infrastructure, an application or operation and maintenance capability declared through the view layer can drift in any Kubernetes cluster without worrying about whether the capabilities supported by these clusters are different.

5. Views are still tables, supporting standard table operations

The view layer objects of Kubernetes must still be standard Kubernetes objects, so that all Kubernetes operations and primitive pairs on API objects will be applicable to view layer objects. We cannot introduce additional mental burden on the Kubernetes API model.

Although the idea of ​​setting a view layer for Kubernetes did not finally land on the upstream side of Kubernetes, it became the mainstream practice of most large-scale players in the community. For example, Pinterest designed a CRD of PInterestService on top of Kubernetes to describe and define the application of Pinterest. This CRD is actually a view layer object. But this approach is still too rudimentary for most companies. You must know that the "view" of data is not just a simple abstraction and translation of data. To use the view layer on a large scale in a real production environment, at least several key issues need to be solved:

• How to define and manage the mapping relationship between view layer objects and underlying K8s objects? Note that this is by no means a simple one-to-one mapping, and a view layer object may correspond to multiple K8s objects.
• How to model and abstract "operation and maintenance capabilities"? A real application is not just a simple Deployment or Operator, it must be an organic combination of the program to be run and the corresponding operation and maintenance capabilities (such as a containerized application and its horizontal expansion strategy). How are these operation and maintenance capabilities reflected in the application definition? Is it feasible to define all annotations as annotations?
• How to manage the binding relationship between operation and maintenance capabilities and programs to be run? How to map this binding relationship to the real execution relationship in the underlying K8s?
• How to standardize the definition of cloud resources through view layer objects, such as an Alibaba Cloud RDS instance?
• ……

The above problems are one of the important reasons why Kubernetes upstream finally failed to implement the "view layer", and also  the main focus of Kubernetes application layer open source projects such as Open Application Model (OAM) . It should be pointed out that only relying on a "specification" such as OAM is still not enough to solve all the above problems. The establishment of the Kubernetes view layer must be guaranteed at the implementation layer with the help of the standard view layer dependency library, in order to truly enjoy the experience in Kubernetes to the advantages and convenience brought by the "Data View". At present, the most powerful Kubernetes view layer dependency library in the community is oam-kubernetes-runtime from the Crossplane team : https://github.com/crossplane/oam-kubernetes-runtime .

Summarize

Kubernetes, a "database"-like design with IaD as the core, is the important theoretical basis behind the prosperity and development of this community. However, the idea of ​​IaD itself is also a double-edged sword. On the other side of the thriving community it spawns, there are countless Controller/Operators that are "in their own right", and an extremely complex complex assembled by these Controllers. Kubernetes cluster. Such a production-level Kubernetes cluster is far from a cloud-native application management platform that is truly loved by R&D and O&M.

In the past 5 years, the great success of the Kubernetes project is actually a process of gradual standardization and unification of infrastructure capabilities (such as network, storage, containers) under the declarative API, and with the development of Kubernetes application layer technologies such as OAM With the gradual popularization, we have seen a standardized application layer ecology is paying off. More and more teams are experimenting with a more user-friendly data view layer that exposes APIs to end users, while providing infrastructure engineers with more powerful horizontal connectivity and modular platform capabilities.

At the same time, the other missing parts of Kubernetes, the "database", will surely emerge more and more in the community. For example, the Open Policy Agent (OPA) project, which is rapidly maturing today, can be considered as the result of the continuous evolution of the "data interception verification and modification mechanism" layer. Another example is Alibaba's internal management and control link performance tuning in the "10,000-node" cluster. Its theoretical basis and practice are similar to today's database performance optimization.

If you have any ideas about the IaD system, you are very welcome to communicate with us by scanning the code and adding a group!

Recommended activities

On May 27, 2020, US time, Crossplane, a well-known complex environment application delivery and infrastructure management project, will jointly hold the Crossplane Community Day in conjunction with the Open Application Model (OAM) community.

Kelsey Hightower , chief evangelist of Google's Kubernetes project , Li Xiang , senior technical expert of Alibaba Cloud, CNCF TOC, and Microsoft Distinguished Engineer, Kubernetes co-founder ** Brendan Burns ** will attend and give an important keynote speech on standardizing applications and infrastructure. .

On this Crossplane Community Day, you'll learn about the deep technical collaboration that the OAM community is working on with the Crossplane project, and how platform builders around the world use OAM and Crossplane to configure and manage applications, infrastructure, and cloud service.

Reply to the keyword crossplane in the background of Alibaba Cloud Native  Official Account , or click the link to directly get the online summit participation method.

Hiring!

The cloud native application platform team sincerely invites experts in the field of Kubernetes/Serverless/PaaS/application delivery (P7-P8) to join in the OAM construction:

• Working years: It is recommended to start from P7 for three years and from P8 for five years, depending on the actual ability;
• Working place: Domestic: Beijing, Hangzhou, Shenzhen;
• Positions include: architect, technical expert, full stack engineer, etc.

Resumes will be replied immediately, and the results will be available in 2~3 weeks. Resume delivery: jianbo.sjb AT alibaba-inc.com

" Alibaba Cloud Native focuses on microservices, serverless, containers, Service Mesh and other technical fields, focuses on cloud native popular technology trends, and cloud native large-scale implementation practices, and is the official account of cloud native developers that understands the most."

{{o.name}}

{{m.name}}