[K8S] Operator（入门篇）

1. 入门与环境准备

1.1 介绍

Operator是一种包装、运行和管理k8s应用的一种方式。它涵盖了 CRD(CustomResourceDeftination) + AdmissionWebhook + Controller ，并以 Deployment 的形式部署到K8S中。
- CRD 用来定义声明式API（yaml），程序会通过该定义一直让最小调度单元（POD）趋向该状态;
- AdmissionWebhook 用来拦截请求做 mutate（修改）提交的声明(yaml)和 validate(校验)声明式的字段；
- Controller 主要的控制器，监视资源的创建 / 更新 / 删除事件，并触发 Reconcile函数作为响应。整个调整过程被称作 Reconcile Loop(协调一致的循环)，其实就是让 POD 趋向CRD定义所需的状态；
Operator流程图
Kubebuilder 它是一个开发Operator的脚手架，可以生成 CRD、Webhook、Controller的代码与配置，并且提供了k8s go-client。

核心概念（引用）: juejin.cn/post/696287…

1.2 环境准备

go version v1.16+.
docker version 18.03+.
kubectl version v1.20.3+.

1.2.0 前置知识

需要熟练掌握k8s基础知识

1.2.1 k8s 环境（服务器）

准备一个可用的k8s环境，无论是单机或者集群都可以。推荐1.20以上
可以在服务器或者本地，建议在服务器使用

1.2.2 kubebuilder（本地）

Doc: book.kubebuilder.io/quick-start…

下载对应平台的的kubebuilder，如用mac开发就下载mac的；
- github.com/kubernetes-…

下载完成后，把二进制文件kubebuilder 加入到环境变量，如：

# mv kubebuilder_darwin_amd64 /usr/local/bin/kubebuilder
# chmod a+x /usr/local/bin/kubebuilder
# kubebuilder version
// Version: main.version{KubeBuilderVersion:"3.2.0", KubernetesVendor:"1.22.1", GitCommit:"b7a730c84495122a14a0faff95e9e9615fffbfc5", BuildDate:"2021-10-29T18:32:16Z", GoOs:"darwin", GoArch:"amd64"}
复制代码

1.2.3 kubectl（本地）

通过kubectl 控制远程服务器，务必加入到环境变量且使用的是默认kubeconfig，否则后面需要修改kubebuilder生成的Makefile。

1.2.4 Docker (本地or服务器)

Docker是什么就不解析了，主要用于发布项目时打包镜像的，也是在kubebuilder生成的Makefile中使用；
- ps: 个人考虑到本地 mac 没有位置安装Docker，所以在服务器安装，但需要注意的是linux和mac使用的controller-gen及kustomize 是不一样；需要在linux使用kubebuilder重新生成一个新的项目用以覆盖现有项目的bin目录。

2. 项目

参考地址: github.com/Shadow-linu…

2.1 项目创建

2.1.1 目录创建

不允许中文、空格、特殊符号，下划线，只允许中划线“-”

# mkdir -p /usr/local/k8s-operator
# cd /usr/local/k8s-operator
复制代码

2.1.2 创建项目

init 项目，创建域（domain）。

# kubebuilder init --domain shadow.com
复制代码

创建 API，创建group、version 和 kind。

# kubebuilder create api --group myapp --version v1 --kind Redis
复制代码

上面创建完成后，后面CRD的定义是以下的样子：test.yaml
```
apiVersion: myapp.shadow.com/v1
kind: Redis
...
复制代码
```

创建后项目结构

2.1.3 创建 CRD

可以选择设置资源的前缀（可选），config/default/kustomization.yaml
```
...
# 可以修改前缀
namePrefix: shadow-operator-
...
复制代码
```
在 k8s-operator/api/v1/redis_types.go，RedisSepc中的属性字段，在这我创建了 Name, Port, Replicas；
install crd
```
# make install
复制代码
```

查看 crd，可通过 kubectl describe crd redis.myapp.shadow.com 查看下crd的定义是否刚刚设置的字段；

# kubectl get crd
NAME                                       CREATED AT
...
redis.myapp.shadow.com                     2021-11-16T09:35:42Z
复制代码

若修改了CRD则需要重新安装

# make uninstall
# make install
复制代码

2.1.4 启动 Controller

我们主要的逻辑就在Reconcile 核心函数中去实现，该函数会被事件反复的触发，k8s-operator/controllers/redis_controller.go

...
func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
_ = log.FromContext(ctx)

    // TODO(user): your logic here
    
    redis := &myappv1.Redis{}
    if err := r.Get(ctx, req.NamespacedName, redis); err != nil {
        fmt.Println(err)
    } else {
        fmt.Println("得到对象", redis.Spec)
    }
}
...
复制代码

运行控制器，留意console端的输出
```
# make run
复制代码
```

2.1.5 测试 Yaml

编辑yaml文件，k8s-operator/config/samples/myapp_v1_redis.yaml

apiVersion: myapp.shadow.com/v1
kind: Redis
metadata:
  name: shadow
  namespace: default
spec:
  name: shadow
  port: 2378
  replicas: 3
复制代码

apply

# kubectl apply -f config/samples/myapp_v1_redis.yaml
复制代码

查看控制器的console输出

...
得到对象 {Name:shadow Port:2378 Replicas:3} 
...
复制代码

2.2 CRD 字段简单验证

Doc: book.kubebuilder.io/reference/m…

2.2.1 演示

k8s-operator/api/v1/redis_type.go，增加特定规范的注释用于限制了 Port 的取值大小，更多的注释作用参考上面的文档。

+kubebuilder:validation:Minimum:=2000
+kubebuilder:validation:Maximum:=2380

type RedisSpec struct {
   // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
   // Important: Run "make" to regenerate code after modifying this file

   // Foo is an example field of Redis. Edit redis_types.go to remove/update
   //Foo string `json:"foo,omitempty"`
   Name string `json:"name,omitempty"`

   // validation: https://book.kubebuilder.io/reference/markers/crd-validation.html

   //+kubebuilder:validation:Minimum:=2000
   //+kubebuilder:validation:Maximum:=2380
   Port int `json:"port,omitempty"`

   Replicas int `json:"replicas,omitempty"`
}
复制代码

重新安装CRD

# make install
# make uninstall
复制代码

验证效果

config/samples/myapp_v1_redis.yaml

apiVersion: myapp.shadow.com/v1
kind: Redis
metadata:
  name: shadow
  namespace: default
spec:
  port: 2390
  replicas: 3
  name: shadow
复制代码

执行后可以看到对应的限制报错

# kubectl apply -f config/samples/myapp_v1_redis.yaml 
The Redis "shadow" is invalid: spec.port: Invalid value: 2390: spec.port in body should be less than or equal to 2380
复制代码

2.3 Webhook 创建（修改与校验）

2.3.1 前置知识

什么是webhook？webhook是一种单独资源可以单独开发。
- kubernetes.io/zh/docs/ref…
我们通常是通过 MutatingWebhook 和 ValidatingWebhook 进行 API 请求拦截，这两个webhook 也是k8s默认开启的（若未开启，请手动开启）。这里的请求内容就是Yaml 文件内容。

2.3.2 创建 webhook

执行创建命令

# kubebuilder create webhook --group myapp --version v1 --kind Redis --defaulting --programmatic-validation
复制代码

创建完成后，生成 api/v1/redis_webhook.go

...
func (r *Redis) Default() {
   redislog.Info("default", "name", r.Name)

   // TODO(user): fill in your defaulting logic.
}

// TODO(user): change verbs to "verbs=create;update;delete" if you want to enable deletion validation.
//+kubebuilder:webhook:path=/validate-myapp-shadow-com-v1-redis,mutating=false,failurePolicy=fail,sideEffects=None,groups=myapp.shadow.com,resources=redis,verbs=create;update,versions=v1,name=vredis.kb.io,admissionReviewVersions=v1

var _ webhook.Validator = &Redis{}

// ValidateCreate implements webhook.Validator so a webhook will be registered for the type
func (r *Redis) ValidateCreate() error {
   redislog.Info("validate create", "name", r.Name)

   // 增加：如果是资源名字为 shadow 则不允许创建 
   if r.Name == "shadow" {
      return fmt.Errorf("error name.")
   }
   // TODO(user): fill in your validation logic upon object creation.
   return nil
}
...
复制代码

2.3.3 部署到k8s环境

因为 webhook 部署到线上我们通常会通过 CA 进行证书校验，所以这里一并配置一个可上线使用的webhook

创建一个 k8s 的cert-manager，它是一个k8s插件。

下载并应用 cert-manager.yaml

# wget https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml

# kubectl apply -f cert-manager.yaml
复制代码

查看是否创建完成

# kubectl  get  pods  -A
NAMESPACE                NAME                                                  READY   STATUS    RESTARTS   AGE
cert-manager             cert-manager-55658cdf68-9559b                         1/1     Running   0          7d18h
cert-manager             cert-manager-cainjector-967788869-hl472               1/1     Running   0          7d18h
cert-manager             cert-manager-webhook-7b86bc6578-spdct
...
复制代码

开启配置 config/default/kustomization.yaml

...
...
bases:
- ../crd
- ../rbac
- ../manager
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
# crd/kustomization.yaml
- ../webhook
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
- ../certmanager
# [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
#- ../prometheus

patchesStrategicMerge:
# Protect the /metrics endpoint by putting it behind auth.
# If you want your controller-manager to expose the /metrics
# endpoint w/o any authn/z, please comment the following line.
- manager_auth_proxy_patch.yaml

# Mount the controller config file for loading manager configurations
# through a ComponentConfig type
#- manager_config_patch.yaml

# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
# crd/kustomization.yaml
- manager_webhook_patch.yaml

# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
# 'CERTMANAGER' needs to be enabled to use ca injection
- webhookcainjection_patch.yaml

# the following config is for teaching kustomize how to do var substitution
vars:
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
- name: CERTIFICATE_NAMESPACE # namespace of the certificate CR
  objref:
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: serving-cert # this name should match the one in certificate.yaml
  fieldref:
    fieldpath: metadata.namespace
- name: CERTIFICATE_NAME
  objref:
    kind: Certificate
    group: cert-manager.io
    version: v1
    name: serving-cert # this name should match the one in certificate.yaml
- name: SERVICE_NAMESPACE # namespace of the service
  objref:
    kind: Service
    version: v1
    name: webhook-service
  fieldref:
    fieldpath: metadata.namespace
- name: SERVICE_NAME
  objref:
    kind: Service
    version: v1
    name: webhook-service
复制代码

进行部署(该步骤作者是在服务器上执行的，因为本地没有安装docker)，且需要预选安装好镜像仓库，harbor 或 docker registry。

# make install
# make docker-build  IMG=192.168.6.102:5000/shadow-redis:v1
# make docker-push  IMG=192.168.6.102:5000/shadow-redis:v1
# make deploy IMG=192.168.6.102:5000/shadow-redis:v1
复制代码

查看部署后的 webhook

# kubectl get  mutatingwebhookconfigurations
NAME                                             WEBHOOKS   AGE
shadow-operator-mutating-webhook-configuration   1          82m
...

# kubectl  get validatingwebhookconfigurations
NAME                                               WEBHOOKS   AGE
shadow-operator-validating-webhook-configuration   1          47h
...
复制代码

2.3.4 进行测试

由于已经部署了cert-manager，用于tls验证，所以本地就不在启动webhook，禁用webhook直接使用k8s环境的controller进行验证；
- main.go，找到 SetupWebhookWithManager 后注释掉
```
//if err = (&myappv1.Redis{}).SetupWebhookWithManager(mgr); err != nil {
// setupLog.Error(err, "unable to create webhook", "webhook", "Redis")
// os.Exit(1)
//}
复制代码
```
启动
```
# make run
复制代码
```

验证

# kubectl apply -f config/samples/myapp_redis_v1.yaml
Error from server (error name.): error when creating "config/samples/myapp_v1_redis.yaml": admission webhook "vredis.kb.io" denied the request: error name.
复制代码

2.4 控制POD资源

每个步骤都需要重启controller
```
# make run
复制代码
```

创建一个helper目录

# mkdir -p k8s-operator/helper
复制代码

2.4.1 增 / 删 / 改

主要实现的功能
- 创建资源；
- 删除资源；
- 副本伸缩；
- POD 被删除后自动重建；
- 记录事件；

helper/redis_helper.go


package helper

import (
   "context"
   "fmt"
   corev1 "k8s.io/api/core/v1"
   "k8s.io/apimachinery/pkg/runtime"
   "k8s.io/apimachinery/pkg/types"
   v1 "shadow.com/v1/api/v1"
   "sigs.k8s.io/controller-runtime/pkg/client"
   "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
)
// 生成 POD name
func GetRedisPodNames(redisConfig *v1.Redis) []string {
   podNames := make([]string, redisConfig.Spec.Replicas)
   fmt.Printf("%+v", redisConfig)
   for i := 0; i < redisConfig.Spec.Replicas; i++ {
      podNames[i] = fmt.Sprintf("%s-%d", redisConfig.Name, i)
   }

   fmt.Println("PodNames: ", podNames)
   return podNames
}

//  判断 redis  pod 是否能获取
func IsExistPod(podName string, redis *v1.Redis, client client.Client) bool {
   err := client.Get(context.Background(), types.NamespacedName{
      Namespace: redis.Namespace,
      Name:      podName,
   },
      &corev1.Pod{},
   )

   if err != nil {
      return false
   }
   return true
}
// 是否存在于finalizers，finalizers 是人为删除动作添加的，
// 只要finalizers有值则删除无法顺利进行，直到finalizers为空；
func IsExistInFinalizers(podName string, redis *v1.Redis) bool {
   for _, fPodName := range redis.Finalizers {
      if podName == fPodName {
         return true

      }
   }
   return false
}

func CreateRedis(client client.Client, redisConfig *v1.Redis, podName string, schema *runtime.Scheme) (string, error) {
   if IsExistPod(podName, redisConfig, client) {
      return "", nil
   }
   // 建立 POD 对象
   newPod := &corev1.Pod{}
   newPod.Name = podName
   newPod.Namespace = redisConfig.Namespace
   newPod.Spec.Containers = []corev1.Container{
      {
         Name:            podName,
         Image:           "redis:5-alpine",
         ImagePullPolicy: corev1.PullIfNotPresent,
         Ports: []corev1.ContainerPort{
            {
               ContainerPort: int32(redisConfig.Spec.Port),
            },
         },
      },
   }

   // set owner reference，使用ControllerManager为我们管理 POD
   // 这个就和ReplicateSet是一个道理
   err := controllerutil.SetControllerReference(redisConfig, newPod, schema)
   if err != nil {
      return "", err
   }
   // 创建 POD
   err = client.Create(context.Background(), newPod)
   return podName, err
}
复制代码

controllers/redis_controller.go

/*
Copyright 2021 shadow.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
        "context"
        "fmt"
        corev1 "k8s.io/api/core/v1"
        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
        "k8s.io/apimachinery/pkg/runtime"
        "k8s.io/apimachinery/pkg/types"
        "k8s.io/client-go/tools/record"
        "k8s.io/client-go/util/workqueue"
        "shadow.com/v1/helper"
        ctrl "sigs.k8s.io/controller-runtime"
        "sigs.k8s.io/controller-runtime/pkg/client"
        "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
        "sigs.k8s.io/controller-runtime/pkg/event"
        "sigs.k8s.io/controller-runtime/pkg/handler"
        "sigs.k8s.io/controller-runtime/pkg/log"
        "sigs.k8s.io/controller-runtime/pkg/reconcile"
        "sigs.k8s.io/controller-runtime/pkg/source"

        myappv1 "shadow.com/v1/api/v1"
)

// RedisReconciler reconciles a Redis object
type RedisReconciler struct {
        client.Client
        Scheme      *runtime.Scheme
        EventRecord record.EventRecorder
}

//+kubebuilder:rbac:groups=myapp.shadow.com,resources=redis,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=myapp.shadow.com,resources=redis/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=myapp.shadow.com,resources=redis/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Redis object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/[email protected]/pkg/reconcile
func (r *RedisReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        _ = log.FromContext(ctx)

        // TODO(user): your logic here

        redis := &myappv1.Redis{}
        if err := r.Get(ctx, req.NamespacedName, redis); err != nil {
                fmt.Println(err)
        } else {
                //  如果不为空 则正在删除
                if !redis.DeletionTimestamp.IsZero() {
                        return ctrl.Result{}, r.clearRedis(ctx, redis)
                }

                fmt.Printf("得到对象 %+v \n", redis.Spec)
                podNames := helper.GetRedisPodNames(redis)
                isEdit := false
                for _, podName := range podNames {
                        podName, err := helper.CreateRedis(r.Client, redis, podName, r.Scheme)

                        if err != nil {
                                return ctrl.Result{}, err
                        }
                        
                        if podName == "" {
                                continue
                        }
                        // 如果存在于 finalizers 证明已经创建了，跳过即可
                        if controllerutil.ContainsFinalizer(redis, podName) {
                                continue
                        }

                        redis.Finalizers = append(redis.Finalizers, podName)
                        isEdit = true
                }

                //  副本收缩
                if len(redis.Finalizers) > len(podNames) {
                        r.EventRecord.Event(redis, corev1.EventTypeNormal, "Upgrade", "副本收缩")
                        isEdit = true
                        err := r.rmIfSurplus(ctx, podNames, redis)
                        if err != nil {
                                return ctrl.Result{}, err
                        }
                }

                if isEdit {
                        r.EventRecord.Event(redis, corev1.EventTypeNormal, "Updated", "更新 shadow-redis")
                        err = r.Client.Update(ctx, redis)
                        if err != nil {
                                return ctrl.Result{}, err
                        }
                        // 下面说到增加RedisNum 来查看当前存在的POD数量，则需要打开下面的代码
                        // redis.Status.RedisNum = len(redis.Finalizers)
                        err = r.Status().Update(ctx, redis)
                        return ctrl.Result{}, err
                }

                return ctrl.Result{}, nil

        }

        return ctrl.Result{}, nil
}

// 收缩副本 ['redis0','redis1']   ---> podName ['redis0']
func (r *RedisReconciler) rmIfSurplus(ctx context.Context, podNames []string, redis *myappv1.Redis) error {
        for i := 0; i < len(redis.Finalizers)-len(podNames); i++ {
                err := r.Client.Delete(ctx, &corev1.Pod{
                        ObjectMeta: metav1.ObjectMeta{
                                Name: redis.Finalizers[len(podNames)+i], Namespace: redis.Namespace,
                        },
                })

                if err != nil {
                        return err
                }
        }
        redis.Finalizers = podNames
        return nil
}

func (r *RedisReconciler) clearRedis(ctx context.Context, redis *myappv1.Redis) error {
        podList := redis.Finalizers
        for _, podName := range podList {
                err := r.Client.Delete(ctx, &corev1.Pod{
                        ObjectMeta: metav1.ObjectMeta{
                                Name:      podName,
                                Namespace: redis.Namespace,
                        },
                })

                if err != nil {
                        fmt.Println("清除Pod异常：", err)
                }
        }

        redis.Finalizers = []string{}
        return r.Client.Update(ctx, redis)
}

func (r *RedisReconciler) podDeleteHandler(event event.DeleteEvent, limitInterface workqueue.RateLimitingInterface) {
        fmt.Println("被删除的对象名称是", event.Object.GetName())

        for _, ref := range event.Object.GetOwnerReferences() {
                // 因为会获取到所有被删除的pod，所以进行一次判断
                if ref.Kind == "Redis" && ref.APIVersion == "myapp.shadow.com/v1" {
                        // 重新推送队列，进行 reconcile
                        limitInterface.Add(reconcile.Request{
                                NamespacedName: types.NamespacedName{
                                        Name:      ref.Name,
                                        Namespace: event.Object.GetNamespace(),
                                },
                        })
                }
        }
}

// SetupWithManager sets up the controller with the Manager.
func (r *RedisReconciler) SetupWithManager(mgr ctrl.Manager) error {
        return ctrl.NewControllerManagedBy(mgr).
                For(&myappv1.Redis{}).
                // 监控资源，并对delete动作进行操作
                Watches(&source.Kind{Type: &corev1.Pod{}}, handler.Funcs{DeleteFunc: r.podDeleteHandler}).
                Complete(r)
}

复制代码

修改完成后进行测试，若遇到webhook拦截则可以通过 kubectl delete 删除即可（mutate, validate）；

myapp_redis_v1.yaml

apiVersion: myapp.shadow.com/v1
kind: Redis
metadata:
  name: shadow
  namespace: default
spec:
  port: 2379
  replicas: 3
  name: shadow
复制代码

创建

# kubectl apply -f config/samples/myapp_redis_v1.yaml
# kubectl get Redis
# kubectl get pods
复制代码

收缩副本，修改 myapp_redis_v1.yaml

apiVersion: myapp.shadow.com/v1
kind: Redis
metadata:
  name: shadow
  namespace: default
spec:
  port: 2379
  replicas: 2
  name: shadow
复制代码

# kubectl apply -f config/samples/myapp_redis_v1.yaml
复制代码

删除

# kubectl delete Redis shadow
复制代码

查看记录的事件

# kubectl describe Redis shadow
复制代码

2.4.2 查

kubectl get Redis 默认只展示 Name, Age 这两个字段，这里扩展一个RedisNum 字段用来统计创建了多少个POD

api/v1/redis_type.go，增加status

 type RedisStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
    RedisNum int `json:"redis_num"`

 }

 //下面两个注释对应的是 kubectl get Redis 看到的状态  
 //+kubebuilder:printcolumn:JSONPath=".status.redis_num",name=REDIS_NUM,type=integer
 //+kubebuilder:printcolumn:JSONPath=".metadata.creationTimestamp",name=AGE,type=date
复制代码

controllers/redis_controller.go，打开下面的注释

// redis.Status.RedisNum = len(redis.Finalizers)
复制代码

重新安装 CRD，并重启Controller
```
# make install
# make run
复制代码
```

查看

# kubectl get Redis
NAME     REDIS_NUM   AGE
shadow   3           21m
复制代码

3. 写在最后

后面等实战项目写完，会更新一个Opeartor实战篇加源码，点个赞再走吧。