Getting started with K8s from scratch | Kubernetes API programming tool: Operator and Operator Framework

Author | Su Xing Senior Engineer at Alibaba

This article is organized from the 24th lecture of "CNCF x Alibaba Cloud Native Technology Open Course", click "Read the original text" to go to the course page.

Follow the "Alibaba Cloud Native" official account and reply to the keyword **"Getting Started"** to download the PPT of the K8s series of articles from scratch.

Introduction : This article will start from practice and combine cases to illustrate how to extend the Kubernetes API with the help of the Operator development framework. The content is mainly divided into three parts: first, it will briefly introduce the knowledge related to Operator; then it will introduce the Operator development framework and explain the whole development process in detail with the case; finally, it will re-explain how the Operator works with the workflow of the case.

1. An overview of the operator

basic concept

First introduce the basic concepts involved in this article.

• CRD (Custom Resource Definition) : allows users to customize Kubernetes resources, which is a type;

• CR (Custom Resourse) : a concrete instance of CRD;

• webhook : It is essentially an HTTP callback that will be registered with the apiserver. When a specific event of the apiserver occurs, the registered webhook will be queried and the corresponding message will be forwarded.

According to the different processing types, it can be generally divided into two categories: one may modify the incoming object, called mutating webhook; one type will read the incoming object, called validating webhook.

• Work Queue : The core component of the controller. It monitors resource changes in the cluster, and stores related objects, including its actions and keys, such as a Create action of a Pod, in the queue as an event;

• **controller **: It will process the above work queue cyclically, and push the cluster state to the expected state according to its own logic. Different controllers handle different types. For example, the replicaset controller focuses on the number of replicas and handles some Pod-related events;

• operator : operator is a set of mechanisms for describing, deploying and managing kubernetes applications. In terms of implementation, it can be understood as CRD combined with optional webhook and controller to implement user business logic, that is, operator = CRD + webhook + controller.

Common operator working modes

work process:

1. The user creates a custom resource (CRD);
2. The apiserver forwards the request of the CRD to the webhook according to a pass list registered by itself;
3. The webhook generally completes the default value setting and parameter verification of the CRD. After the webhook is processed, the corresponding CR will be written to the database and returned to the user;
4. At the same time, the controller will monitor the custom resource in the background, and process special operations associated with the custom resource according to business logic;
5. The above processing generally causes state changes in the cluster, and the controller monitors these associated changes and records these changes in the state of the CRD.

Here is a general introduction from High-Level, which will be reorganized later in combination with cases.

Second, operator framework combat

operator framework overview

Before we start, let's first introduce the operator framework. It actually provides users with the framework of webhook and controller. Its main significance is to help developers shield some common underlying details, without requiring developers to implement message notification triggering, failure re-queuing, etc. The operation and maintenance logic of the management application can be implemented.

There are two main mainstream operator frameworks: kubebuilder and operator-sdk .

There is actually no essential difference between the two, and their core is to use the official controller-tools and controller-runtime. However, the details are slightly different. For example, kubebuilder has more complete testing and deployment and code generation scaffolding, etc.; while operator-sdk supports upper-level operations such as ansible operator better.

kuberbuildere in action

The actual combat here is kuberbuilder. The case uses SidercarSet under the kruise project opened by Alibaba Cloud.

The function of SidercarSet is to insert sidecar containers (also called auxiliary containers) into the Pod. For example, some monitoring and log collection can be inserted to enrich the functions of the Pod, and then the SidercarSet can be updated according to the inserted state and the state of the Pod to record these auxiliary containers. The state of the container.

Step 1: Initialize

Operation : Create a new gitlab project and run "kubebuilder init --domain=kruise.io".

Parameter interpretation : domain specifies the Group domain name of the subsequently registered CRD object.

Interpretation of the effect : Pull dependent code bases, generate code frameworks, generate Makefile/Dockerfile and other tool files.

Let's take a look at the specific content it generates (for the convenience of demonstration, the actual generated file has been partially deleted):

The specific content can be confirmed by yourself in actual combat.

Step 2: Create API

Action : Running "kubebuilder create api --group apps --version v1alpha1 --kind SidecarSet --namespace=false" will actually not only create the API, that is, the CRD, but also generate the framework of the Controller.

Parameter interpretation : - group plus the previous domian is the Group of this CRD: apps.kruise.io;

• There are generally three types of versions, according to community standards:
  • v1alpha1: This api is unstable, CRD may be abandoned, fields may be adjusted at any time, do not rely on it;
  • v1beta1: The api has been stabilized, backward compatibility is guaranteed, and features may be adjusted;
  • v1: api and features are stable;
• kind: The type of this CRD, similar to the community's native Service concept;
• namespaced: Whether this CRD is globally unique or namespace unique, similar to node and pod.

Its parameters can be basically divided into two categories. group, version, kind basically correspond to the three important components of CRD meta information. Here are some common standards that you can refer to when you actually use them. namespaced is used to specify whether the CRD we just created is globally unique (eg node) or namespace unique (eg Pod). False is used here, that is, the SidecarSet is specified as globally unique.

Interpretation of the effect :

The framework of CRD and controller is generated, and the code needs to be filled manually later.

The actual result is shown below:

We focus on the blue font in the figure. sidecarset_types.go is where the CRD is defined and needs to be filled in. sidecarset_controller.go is used to define the controller, which also needs to be filled.

Step 3: Fill CRD

1. The generated CRD is located in "pkg/apis/apps/v1alpha1/sidecarset_types.go", which usually requires the following two operations :

(1) Adjustment comments

The code generator relies on annotations to generate code, so sometimes annotations need to be adjusted. The following lists the comments that SidecarSet needs to be adjusted in this actual combat:

+genclient:nonNamespaced : Generate non-namespace objects; +kubebuilder:subresource:status : Generate status subresources; +kubebuilder:printcolumn:name="MATCHED",type='integer',JSONPath=".status.matchedPods",description= "xxx": kubectl get sidecarset : related to subsequent presentations.

(2) Fill field

Populating fields is an important part of making a user's CRD actually work and actually make sense.

• SidecarSetSpec : Fill CRD description information;
• SidecarSetStatus : Fill CRD status information.

 2. After filling, run make to regenerate the code

It should be noted that developers do not need to participate in the underlying implementation of the CRD's grpc interface, codec and other controllers.

The actual padding looks like this:

The function of SidecarSet is to inject Sidecar into Pod. To complete this function, we define two main pieces of information in SidecarSetSpec (left): one is the Selector for selecting which Pods need to be injected; the other is the Containers that define the Sidecar container.

Status information is defined in SidecarSetStatus (right), MatchedPods reflects how many Pods actually matched by the SidecarSet, UpdatedPods reflects how many Pods have been injected, and ReadyPods reflects how many Pods have been working normally.

The complete content can refer to this document .

Step 4: Generate the webhook framework

1. To generate the mutating webhook, run :

"kubebuilder alpha webhook --group apps --version v1alpha1 --kind SidecarSet --type=mutating --operations=create"

"kubebuilder alpha webhook --group core --version v1 --kind Pod --type=mutating --operations=create"

2. To generate the validating webhook, run :

"kubebuilder alpha webhook --group apps --version v1alpha1 --kind SidecarSet --type=validating --operations=create,update"

Parameter interpretation :

• group/kind describes the resource object that needs to be processed;
• type describes what type of framework needs to be generated;
• operations describes which operations are concerned with the resource object.

Interpretation of the effect :

• The webhook framework is generated, and the code needs to be filled manually later

The actual generated result is shown in the following figure:

We executed three commands to generate three different handlers that need to be filled (the blue font part in the figure above). I will not mention it here, and I will explain it in detail in the next filling operation.

Step 5: Populate the webhook

The generated webhook handlers are located at:

• • pkg/webhook/default_server/sidecarset/mutating/xxx_handler.go
• • pkg/webhook/default_server/sidecarset/validating/xxx_handler.go
• • pkg/webhook/default_server/pod/mutating/xxx_handler.go

What needs to be rewritten and filled generally includes the following two parts: 

• Whether to inject K8s client : depends on whether other resources are required in addition to the incoming CRD. In this actual combat, not only need to pay attention to SidecarSet, but also inject Pod, so it is necessary to inject K8s client;

• The key method to populate the webhook : mutatingSidecarSetFn or validatingSidecarSetFn. Since the pointer of the resource object to be operated has been passed in, the hook's work can be completed by directly adjusting the properties of the object.

Let's take a look at the actual fill result.

Because in the fourth step, we defined three: sidecarset mutating, sidecarset mutaing, and pod mutating.

Let's first look at the sidecarset mutating on the left side of the figure above. First, setDefaultSidecarSet sets the default value, which is also the most common thing for mutaing.

The validating on the right side of the above figure is also very standard, and it also checks some fields of the SidecarSet.

Regarding pod mutaing, it is not shown here. There are some differences here. The mutaingSidecarSetFn here does not set the default value, but obtains the value of setDefaultSidecarSet and injects it into the Pod.

Step 6: Populate the controller

The generated controller skeleton is located in pkg/controller/sidecarset/sidecarset_controller.go. There are three main points that need to be modified: 

• **Modify permission comments. **The framework will automatically generate a comment in the form of //+kuberbuilder:rbac;groups=apps,resources=deployments/status,verbs=get;update;path, we can modify it according to our own needs, the comment will eventually generate rbac rules ;

 

• **Added enqueue logic. **The default code framework will fill the CRD's own queue logic (for example, additions, deletions, and modifications of SidecarSet objects will be added to the work queue). If you need to associate the trigger mechanism of resource objects (for example, SidecarSet also needs to pay attention to Pod changes), you need to manually Add its enqueue logic;

 

• ** Populate business logic. **Modify the Reconcile function to process the work queue in a loop. The Reconcile function mainly completes two parts: "completing business logic according to Spec" and "feeding back business logic results to status". It should be noted that if the Reconcile function returns err in error, it will be re-queued by default.

Let's take a look at the filling result of the Controller of the SidecarSet:

In addPod, the SidecarSet corresponding to the Pod is retrieved and added to the queue for processing by Reconcile.

After Reconcile takes out the SidercarSet, it selects the matching Pod according to the Selector, and finally calculates the cluster state according to the current state information of the Pod, and then backfills it into the CRD state.

3. Workflow of SidecarSet

Finally, let's reorganize the workflow of SidecarSet so that we can understand how the operator works.

1. The user creates a SidecarSet;
2. After the webhook receives the SidecarSet, it will perform default value setting and configuration item verification. After these two operations are completed, the real storage will be completed and returned to the user;
3. User creates a Pod;
4. The webhook will get back the corresponding SidecarSet, and take out the container from it and inject it into the Pod, so the Pod already has the sidecar just now when it is actually put into the library;
5. The controller keeps polling in the background to check the status changes of the cluster. The injection in step 4 will trigger the sidecarSet to be enqueued, and the controller will increment the SidecarSet's UpdatedPods by 1.

The above is a simple function implementation in the early stage of SidecarSet.

Here we add another question. In general webhook, the controller is used to complete the business logic and status update, but this is not certain, one of the two may not be necessary. In the above example, the main business logic is completed by the webhook without the participation of the controller.

4. Summary of this paper

The main content of this article ends here, here is a brief summary for you:

• Operator is a mechanism for CRD to cooperate with optional webhook and controller to extend user business logic under Kubernetes system;
• kubebuilder is an official, standardized Operator framework that is highly recognized by the community;
• According to the actual combat steps above, fill in the user-defined code, and you can easily implement an Operator.

" Alibaba Cloud Native focuses on micro-services, serverless, containers, Service Mesh and other technical fields, focuses on popular cloud-native technology trends, and implements large-scale implementation of cloud-native, and is the technology circle that understands cloud-native developers best."

{{o.name}}

{{m.name}}