Author | Chen Xianlu Senior Technical Expert of Alibaba
This article is organized from the 23rd lecture of "CNCF x Alibaba Cloud Native Technology Open Course", click "Read the original text" to go to the course page. <br />Follow the "Alibaba Cloud Native" official account, reply to the keyword **"Getting Started"**, and you can download the PPT of the K8s series from zero entry.
Introduction : In Kubernetes, the API programming paradigm is the Custom Resources Definition (CRD). The CRD we often talk about actually refers to user-defined resources. Why is there a user-defined resource problem? This article will start from the source of its demand, and give a step-by-step in-depth explanation of this concept.
The source of demand
First, let's look at the sources of demand for the API programming paradigm.
In Kubernetes, the API programming paradigm is the Custom Resources Definition (CRD). The CRD we often talk about actually refers to user-defined resources.
Why is there a problem with user-defined resources?
With the increasing use of Kubernetes, the demand for user-defined resources will also increase. However, the function of aggregating various sub-resources provided by Kubernetes can no longer meet the increasing wide-ranging needs. The user wants to provide a user-defined resource that aggregates all sub-resources. However, the expansion and use of Kubernetes native resources is more complicated, so the function of user-defined resources was born.
2. Interpretation of use cases
An instance of CRD
Let's first specifically introduce what a CRD is.
The CRD function was introduced in Kubernetes version 1.7, and users can add custom Kubernetes object resources according to their needs. It is worth noting that the Kubernetes object resources added by the user here are all native and first-class citizens, and are the same object resources as the native Pods and Deployments that come with Kubernetes. From the perspective of Kubernetes' API Server, they are all first-class resources that exist in etcd.
At the same time, custom resources, like native built-in resources, can be created and viewed with kubectl, and also enjoy RBAC and security functions. Users can develop custom controllers to sense or manipulate changes in custom resources.
Let's look at a simple CRD example. The following figure is the definition of a CRD.
First of all, the top apiVersion refers to an apiVersion declaration of a CRD, declaring that it is a CRD requirement or a defined Schema.
kind is CustomResourcesDefinition, which refers to CRD. name is a user-defined name in a user-defined resource. Generally, we recommend using the format of "top-level domain name.xxx.APIGroup", such as foos.samplecontroller.k8s.io here.
spec is used to specify the group and version of the CRD. For example, when creating a Pod or Deployment, its group may be apps/v1 or apps/v1beta1, etc. Here we also need to define the CRD group.
- The group in the figure is samplecontroller.k8s.io;
- verison is v1alpha1;
- Names refers to what its kind is. For example, the kind of Deployment is Deployment, the kind of Pod is Pod, and the kind here is defined as Foo;
- The plural field is a nickname. For example, when the names of some fields or some resources are long, you can use this field to customize some nicknames to simplify its length;
- The scope field indicates whether the CRD is namespace managed. For example, ClusterRoleBinding is at the Cluster level. Another example is that Pod and Deployment can be created in different namespaces, so their scope is Namespaced. The CRD here is Namespaced.
The following figure is an example of the CRD defined in the above figure.
- Its apiVersion is the samplecontroller.k8s.io/v1alpha1 we just defined;
- kind is Foo;
- The name of metadata is the name of our example;
- In this example, the spec field is not actually defined in the CRD Schema. We can write it in the spec according to our own needs. The format is the format of key:value, such as deploymentName: example-foo, replicas: 1 in the figure . Of course, we can also do some tests or state resources to define what exactly is included in the spec.
CRD with checksum
Let's look at a CRD definition that includes validation:
It can be seen that this definition is more complicated, we will not repeat the fields before validation, and look at the validation section separately.
It is first the definition of an openAPIV3Schema, and what resources are defined in the spec. Take replicas as an example. Here, replicas are defined as an integer resource, the minimum value is 1, and the maximum value is 10. Then, when we use this CRD again, if the replicas we give is not an int value, or we write a value of -1, or a value greater than 10, the CRD object will not be submitted to the API Server, and the API Server will directly An error is reported, telling you that the defined parameter conditions are not met.
CRD with status field
Look again at the CRD definition with the status field.
When we use some Deployments or Pods, we may check the status of the current deployment, whether it is updated, etc. after the deployment is completed. These are achieved by increasing the state field. Also, before Kubernetes version 1.12, there was no state field.
The state is actually a sub-resource of a custom resource, and the nice thing about it is that an update to this field doesn't trigger a Deployment or Pod redeployment. We know that for some Deployments and Pods, as long as some specs are modified, it will recreate a new Deployment or Pod. But the state resource is not recreated, it is just used to reflect the entire state of the current Pod. The state of its sub-resources in the CRD declaration in the above figure is very simple, which is a key:value format. Anything written in "{}" is customized.
Take the status field of a Deployment as an example, it contains information such as availableReplicas, current status (such as which version has been updated to, when is the last version), and so on. When a user customizes a CRD, some complex operations can also be performed to tell other users what its current state is.
3. Operation Demonstration
Now let's demonstrate the CRD in detail.
We have two resources here: crd.yaml and example-foo.yaml.
First create the Schema of this CRD to let our Kubernetes Server know what the CRD looks like. The way to create is very simple, it is "kuberctl create -f crd.yaml".
Through "kuberctl get crd", you can see that the CRD just now has been created successfully.
At this time, we can create the corresponding resource "kuberctl create -f example-foo.yaml":
Let's see what's in it "kubectl get foo example-foo -o yaml" :
You can see that it is a Foo resource, the spec is what we just defined, and the selected part is basically all the metadata resources of Kubernetes will have. Therefore, there is not much difference between creating this resource and creating a Pod normally, but this resource is not a Pod, nor is it a built-in resource of Kubernetes itself, it is a resource created by ourselves. In terms of usage and experience, it is almost the same as the usage of built-in resources in Kubernetes.
4. Architecture Design
Controller overview
Defining just one CRD doesn't really do anything, it's simply counted into etcd by the API Server. How to do some complex operations based on the resources and Schema defined by this CRD is implemented by the Controller, that is, the controller. <br /> <br />Controller is actually a pluggable method provided by Kubernetes to extend or control declarative Kubernetes resources. It is the brain of Kubernetes and is responsible for the control operations of most resources. Take Deployment as an example, it is deployed through kube-controller-manager.
For example, if a Deployment is declared to have replicas and 2 Pods, then kube-controller-manager will create two copies of the corresponding Pod after receiving the request while observing etcd, and it will observe these in real time. The state of the Pod, if these Pods change, roll back, fail, restart, etc., it will do some corresponding operations.
Therefore, the Controller is the brain that controls the final state of the entire Kubernetes resource.
After the user declares the completion of the CRD, it is also necessary to create a controller to complete the corresponding goal. For example, the previous Foo, it wants to create a Deployment, and the replicas is 1, which requires us to create a controller to create the corresponding Deployment to truly realize the function of CRD.
Controller Workflow Overview
Here is an example of kube-controller-manager.
As shown in the figure above, there is an Informer on the left. Its mechanism is to watch kube-apiserver, and kube-apiserver will supervise the creation, update and deletion of all resources in etcd. Informer has two main methods: one is ListFunc; the other is WatchFunc.
- ListFunc is an operation like "kuberctl get pods", which lists all the current resources;
- WatchFunc will establish a long link with the apiserver. Once a new object is submitted, the apiserver will push it back, telling the Informer that a new object is created or updated.
After the Informer receives the requirements of the object, it will call the corresponding functions (such as the three functions AddFunc, UpdateFunc and DeleteFunc in the figure), and put them in a queue according to the format of the key value. The naming rule of the key value is "namespace/name", where name is the name of the corresponding resource. For example, if we created a resource of type foo in the default namespace, its key value is "default/example-foo". After the Controller gets an object from the queue, it will do the corresponding operation.
The following figure shows the workflow of the controller.
First, push events through kube-apiserver, such as Added, Updated, Deleted; then enter the ListAndWatch() loop of the Controller; there is a first-in, first-out queue in ListAndWatch, which will be Pop()ed out during operation; Then find the corresponding Handler. Handler will pass it to the corresponding function (such as Add(), Update(), Delete()).
A function generally has multiple workers. Multiple Workers means that for example, several objects come in at the same time, then the Controller may start five or ten such Workers at the same time to execute in parallel, and each Worker can handle different object instances.
After the work is completed, that is, after the corresponding object is created, the key is discarded, indicating that the processing has been completed. If there is any problem in the processing process, it will report an error directly, type an event, and then put the key back into the queue, and the next worker can receive it and continue the same processing.
V. Summary of this paper
The main content of this article ends here, here is a brief summary for you:
- CRD is the abbreviation of Custom Resources Definition, that is, user-defined resources, users can use this function to expand their own Kubernetes native resource information;
- Like ordinary Kubernetes resources, CRDs can be controlled by RBAC permissions and support the status field;
- CRD-controller is also the CRD controller, which can realize the user's own writing, and parse the CRD and turn it into the state expected by the user.
" Alibaba Cloud Native focuses on micro-services, serverless, containers, Service Mesh and other technical fields, focuses on popular cloud-native technology trends, and implements large-scale implementation of cloud-native, and is the technology circle that understands cloud-native developers best."