Started from scratch K8s | Kubernetes API programming tool: Operator and Operator Framework

Author | Su Xing senior Alibaba

This paper finishing from "CNCF x Alibaba native cloud technology open class" Lecture 24, click on the "read the original" through the course page.

Concern "Alibaba Cloud native" No public replies keyword "Getting Started" , can be downloaded from the zero entry K8s series PPT.

REVIEW : This article from the practice, combined with the case to illustrate how to use Operator development framework to extend Kubernetes API. The content is divided into three parts: First, will brief the relevant knowledge Operator; Operator will then introduce development framework in conjunction with the case explained in detail throughout the development process; eventually combined to re-work process cases illustrate how the Operator work.

A, operator Overview

basic concepts

First, this article introduces the basic concepts involved.

• The CRD (the Custom the Resource Definition) : Kubernetes allow users to customize the resource, is a type;

• CR (the Custom Resourse) : A specific example of the CRD;

• WebHook : It is a HTTP callbacks on nature, will be registered to the apiserver. Apiserver when a specific event occurs, queries webhook registered, and forwards the appropriate messages in the past.

According to different types of treatment, it can be generally divided into two categories: one may modify the incoming object, called mutating webhook; will read only one incoming object class, called validating webhook.

• Work Queue : controller core components. It monitors changes in resources in the cluster, and the related object, including its operation with the key, for example a Pod Create action, as an event stored in the queue;

• Controller : It cyclically processing said work queues, according to their logic state of the cluster pushed intended state. Different treatment of different types of controller, such as replicaset controller is concerned that the number of copies, will deal with some of the Pod-related events;

• operator : operator is described, a mechanism kubernetes application deployment and management, from the implementation, it can be understood as optional webhook with CRD and business logic controller to achieve the user, i.e., operator = CRD + webhook + controller.

Common operator operating mode

work process:

1. The user creates a custom resource (CRD);
2. apiserver a pass according to their own list of registered forwards the request to the CRD webhook;
3. webhook usually complete set default values ​​and parametric tests of the CRD. After webhook processed, the corresponding CR is written to the database, returned to the user;
4. At the same time, the Controller will custom resource monitor in the background, according to the service logic, the operation of the special processing associated with the custom resource;
5. The above-described process will generally cause a state change in the cluster, the Controller will monitor changes associated, recording the changes to the state of the CRD.

Here is the High-Level outline the back will re-sort with case.

Two, operator framework combat

operator framework Overview

Before you begin, first of all introduce operator framework. It actually provides users with a framework webhook and controller, its main significance is to help developers to shield some common underlying details, developers do not need to go for message notification triggers, such as failed back into the team, only to be attention operation and maintenance management application logic can be realized.

Mainstream operator framework there are two: kubebuilder and operator-sdk .

Both really no distinction between the nature of their core are the official use of controller-tools and controller-runtime. But the details are slightly different, such as kubebuilder has a more comprehensive testing and deployment, and code generation of scaffolding; and operator-sdk support ansible operator of such upper operate better.

kuberbuildere combat

Here is a selection of real kuberbuilder. Case selection is SidercarSet under the cloud Ali opening kruise project.

SidercarSet function is responsible for Pod insert sidecar container (also known as secondary packaging), for example, can insert some monitoring, log collection functionality to enrich the Pod, then insert according to the state, the state Pod turn to record these auxiliary updated SidercarSet state of the container.

Step 1: Initialization

Operation : Create a gitlab project, run "kubebuilder init --domain = kruise.io".

Interpretation of parameters : domain domain name specifies the Group subsequent registration CRD object.

Interpretation of results : pulling dependent code base frame generating code generates Makefile / Dockerfile tools such as files.

Let's look at it specifically generated content _ (To facilitate the presentation, the actual generation of the file made some cut):

Specific content we can confirm detailed in combat when their own.

Step 2: Create an API

Operation : Run "kubebuilder create api --group apps --version v1alpha1 --kind SidecarSet --namespace = false"
in fact not only creates API, that is, CRD, the Controller will generate framework.

Parameter Interpretation : - before this group together with the CRD domian i.e. Group: apps.kruise.io;

• version generally in three, according to community standards:
  • v1alpha1: This api is unstable, CRD may be abandoned, fields may be adjusted at any time, do not rely on;
  • v1beta1: api has stabilized, it will ensure backward compatibility characteristics may be adjusted;
  • v1: api and characteristics are stable;
• kind: CRD of this type, similar to the concept of community native of Service;
• namespaced: This CRD is the only globally unique namespace or similar node and Pod.

Its parameters can be basically divided into two categories. group, version, kind substantially corresponding to the three major components CRD membered information. Here are some common standards, when we actually used for reference. When namespaced is used to specify the CRD we just created globally unique (such as node) is the only namespace (such as Pod). Here with the false, that is designated SidecarSet globally unique.

Interpretation of results :

CRD controller and generates the frame, behind the need to manually fill code.

Actual results as shown below:

我们重点关注是图中蓝色字体部分。sidecarset_types.go 就是定义 CRD 的地方，需要我们填充。sidecarset_controller.go 则用于定义 controller，同样需要进行填充。

Step 3: 填充 CRD

1. 生成的 CRD 位于 "pkg/apis/apps/v1alpha1/sidecarset_types.go"，通常需要进行如下两个操作：

(1) 调整注释

code generator 依赖注释生成代码，因此有时需要调整注释。以下列出了本次实战中 SidecarSet 需要调整的注释：

+genclient:nonNamespaced: 生成非 namespace 对象；
+kubebuilder:subresource:status: 生成 status 子资源；
+kubebuilder:printcolumn:name="MATCHED",type='integer',JSONPath=".status.matchedPods",description="xxx": kubectl get sidecarset： 后续展示相关。

(2) 填充字段

填充字段是令用户的 CRD 实际生效、实际有意义的重要部分。

• SidecarSetSpec: 填充 CRD 描述信息；
• SidecarSetStatus: 填充 CRD 状态信息。

2. 填充完运行 make 重新生成代码即可

需要注意的是，研发人员无需参与 CRD 的 grpc 接口、编解码等 controller 的底层实现。

实际的填充如下图所示：

SidecarSet 的功能是给 Pod 注入 Sidecar，为了完成该功能，我们在 SidecarSetSpec（左图） 定义了两个主要信息：一个是用于选择哪些 Pod 需要被注入的 Selector；一个是定义 Sidecar 容器的 Containers。

在 SidecarSetStatus（右图）中定义了状态信息，MatchedPods 反映的是该 SidecarSet 实际匹配了多少 Pod，UpdatedPods 反映的是已经注入了多少，ReadyPods 反映的则是有多少 Pod 已经正常工作了。

完整的内容可以参考该文档。

Step 4: 生成 webhook 框架

1. 生成 mutating webhook，运行：

"kubebuilder alpha webhook --group apps --version v1alpha1 --kind SidecarSet --type=mutating --operations=create"

"kubebuilder alpha webhook --group core --version v1 --kind Pod --type=mutating --operations=create"

2. 生成 validating webhook，运行：

"kubebuilder alpha webhook --group apps --version v1alpha1 --kind SidecarSet --type=validating --operations=create,update"

参数解读：

• group/kind 描述需要处理的资源对象；
• type 描述需要生成哪种类型的框架；
• operations 描述关注资源对象的哪些操作。

效果解读：

• 生成了 webhook 框架，后面需要手工填充代码

实际生成结果如下图所示：

我们执行了三条命令，分别生成了三个不同的需要填充的 handler 中（上图蓝色字体部分）。这里先不提，在下一步填充操作中再对其详细讲解。

Step 5: 填充 webhook

生成的 webhook handler 分别位于：

• • pkg/webhook/default_server/sidecarset/mutating/xxx_handler.go
• • pkg/webhook/default_server/sidecarset/validating/xxx_handler.go
• • pkg/webhook/default_server/pod/mutating/xxx_handler.go

需要改写、填充的一般包括以下两个部分：

• 是否需要注入 K8s client：取决于除了传入的 CRD 以外是否还需要其它资源。在本实战中，不仅要关注 SidecarSet，同时还要注入 Pod，因此需要注入 K8s client；

• 填充 webhook 的关键方法：即 mutatingSidecarSetFn 或 validatingSidecarSetFn。由于待操作资源对象指针已经传入，因此直接调整该对象属性即可完成 hook 的工作。

我们来看一下实际的填充结果。

因为第四步我们定义了三个：sidecarset mutating、sidecarset mutaing、pod mutating。

先来看上图左侧的 sidecarset mutating，首先是 setDefaultSidecarSet 把默认值设置好，这也是 mutaing 最常做的事情。

上图右侧 validating 也是非常的标准，也是对 SidecarSet 一些字段进行校验。

关于 pod mutaing 这里没有做展示，这里面有些不同，这里面的 mutaingSidecarSetFn 不是进行默认值设置，而是获取 setDefaultSidecarSet 的数值，然后注入到 Pod 里面。

Step 6: 填充 controller

生成的 controller 框架位于 pkg/controller/sidecarset/sidecarset_controller.go。主要有三点需要进行修改：

• 修改权限注释。框架会自动生成形如 //+kuberbuilder:rbac;groups=apps,resources=deployments/status,verbs=get;update;path 的注释，我们可以按照自己的需求修改，该注释最终会生成 rbac 规则；

• 增加入队逻辑。缺省的代码框架会填充 CRD 本身的入队逻辑（如 SidecarSet 对象的增删改都会加入工作队列），如果需要关联资源对象的触发机制（如 SidecarSet 也需关注 Pod 的变化），则需手工新增它的入队逻辑；

• 填充业务逻辑。修改 Reconcile 函数，循环处理工作队列。Reconcile 函数主要完成「根据 Spec 完成业务逻辑」和「将业务逻辑结果反馈回 status」两部分。需要注意的是，如果 Reconcile 函数出错返回 err，默认会重新入队。

我们来看一下 SidecarSet 的 Controller 的填充结果：

addPod 中先取回该 Pod 对应的 SidecarSet 并将其加入队列以便 Reconcile 进行处理。

Reconcile 将 SidercarSet 取出之后，根据 Selector 选择匹配的 Pod，最后根据 Pod 当前的状态信息计算出集群的状态，然后回填到 CRD 的状态中。

三、SidecarSet 的工作流程

Finally, we come again to sort out what workflow SidecarSet so that we can understand how the operator works.

1. Users create a SidecarSet;
2. After receiving the webhook SidecarSet, default values ​​will be set and verify the configuration item. After the completion of these two operations will be completed real storage, and returned to the user;
3. Users create a Pod;
4. webhook will back corresponding SidecarSet, Pod injection container is removed therefrom, and thus when the actual Pod storage of just had with the sidecar;
5. controller constantly polling in the background, change view cluster status. Step 4 will trigger SidecarSet injected into the team, controller will make UpdatedPods SidecarSet plus one.

These are the pre-SidecarSet a simple function to achieve.

Here we have to add a question. General webhook by the controller to complete the business logic, status updates, but this is not certain, one of the two may not be necessary. In the above example is completed by the webhook main business logic without controller involvement.

Fourth, the paper summarizes

The main content to stop here, and here we briefly summarize:

• Operator with the CRD is optional and webhook controller, a mechanism to extend the user's business logic at Kubernetes system;
• kubebuilder is a high degree of community recognition of one of the official, standardized Operator framework;
• Follow above steps actual filling custom code, it can easily achieve an Operator.

" Alibaba Cloud native concern micro service, Serverless, container, Service Mesh and other technical fields, focusing cloud native popular technology trends, cloud native large-scale landing practice, most do understand the developer's native cloud technology circles."