kubernetes CRD Development Guide

Extended kubernetes two most common and need to know something: custom resource CRD and adminsion webhook, the teachings ten minutes you how to master the CRD development.

Pod went to create a composition kubernetes allow users to customize their resource object, just as the deployment statefulset, this one is widely used, such as Prometheus prometheus opterator on custom objects, controller plus a custom when listening to kubectl create Prometheus pormetheus cluster. rook and so on empathy.

I need kubernetes scheduling virtual machine, so there is a custom type VirtualMachine

kubebuilder

kubebuilder help us save a lot of work, so the development of CRD and adminsion webhook breeze.

installation

By source installation:

git clone https://github.com/kubernetes-sigs/kubebuilder
cd kubebuilder
make build
cp bin/kubebuilder $GOPATH/bin

Or download the binary:

os=$(go env GOOS)
arch=$(go env GOARCH)

# download kubebuilder and extract it to tmp
curl -sL https://go.kubebuilder.io/dl/2.0.0-beta.0/${os}/${arch} | tar -xz -C /tmp/

# move to a long-term location and put it on your path
# (you'll need to set the KUBEBUILDER_ASSETS env var if you put it somewhere else)
sudo mv /tmp/kubebuilder_2.0.0-beta.0_${os}_${arch} /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin

Also need to hold kustomize this is a rendering artifact yaml let helm trembling.

go install sigs.k8s.io/kustomize/v3/cmd/kustomize

use

Note that you have to have a kubernetes clusters, step installation you go

Creating CRD

kubebuilder init --domain sealyun.com --license apache2 --owner "fanux"
kubebuilder create api --group infra --version v1 --kind VirtulMachine

CRD install and start the controller

make install # 安装CRD
make run # 启动controller

Then we can see the creation of the CRD

# kubectl get crd
NAME                                           AGE
virtulmachines.infra.sealyun.com                  52m

To create a virtual machine:

# kubectl apply -f config/samples/
# kubectl get virtulmachines.infra.sealyun.com 
NAME                   AGE
virtulmachine-sample   49m

Look at yaml file:

# cat config/samples/infra_v1_virtulmachine.yaml 
apiVersion: infra.sealyun.com/v1
kind: VirtulMachine
metadata:
  name: virtulmachine-sample
spec:
  # Add fields here
  foo: bar

Here are just a deposit to the yaml etcd again, and we listen to when you create an event controller did not do what matter.

The controller is deployed to the cluster

make docker-build docker-push IMG=fanux/infra-controller
make deploy

I was not even a remote kubenetes, test to make life difficult when make docker-build, no etcd bin file, so the first test off.

Modify Makefile:

# docker-build: test
docker-build: 

Dockerfile where gcr.io/distroless/static:latestthis image may also pull you down, random change to change on the line, I changedgolang:1.12.7

When the code is also possible to construct some pull down, to enable it go mod vendor to rely on packaged into

go mod vendor
如果你本地有些代码拉不下来，可以用proxy:

export GOPROXY=https://goproxy.io

And then change the Dockerfile, commented download:

Modified:

# Build the manager binary
FROM golang:1.12.7 as builder

WORKDIR /go/src/github.com/fanux/sealvm
# Copy the Go Modules manifests
COPY . . 

# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o manager main.go

# Use distroless as minimal base image to package the manager binary
# Refer to https://github.com/GoogleContainerTools/distroless for more details
# FROM gcr.io/distroless/static:latest
FROM golang:1.12.7
WORKDIR /
COPY --from=builder /go/src/github.com/fanux/sealvm/manager .
ENTRYPOINT ["/manager"]

make deploy When the error: Error: json: cannot unmarshal string into Go struct field Kustomization.patches of type types.Patch

Put config/default/kustomization.yamlthe patches:change patchesStrategicMerge:to

kustomize build config/default This command will render the controller of yaml files, you can experience the next

See your controller has run up:

kubectl get deploy -n sealvm-system
NAME                        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
sealvm-controller-manager   1         1         1            0           3m
kubectl get svc -n sealvm-system
NAME                                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
sealvm-controller-manager-metrics-service   ClusterIP   10.98.71.199   <none>        8443/TCP   4m

Develop

Add Object data parameters

Look config / samples yaml the following documents:

apiVersion: infra.sealyun.com/v1
kind: VirtulMachine
metadata:
  name: virtulmachine-sample
spec:
  # Add fields here
  foo: bar

Here, there are parameters foo:barthat we add a virtual CPU, memory information:

Directly api/v1/virtulmachine_types.goto

// VirtulMachineSpec defines the desired state of VirtulMachine
// 在这里加信息
type VirtulMachineSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    // Important: Run "make" to regenerate code after modifying this file
    CPU    string `json:"cpu"`   // 这是我增加的
    Memory string `json:"memory"`
}

// VirtulMachineStatus defines the observed state of VirtulMachine
// 在这里加状态信息，比如虚拟机是启动状态，停止状态啥的
type VirtulMachineStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
}

Then make it:

make && make install && make run

Then go to render it yaml controller will find CRD has been put on the CPU and memory information:

kustomize build config/default

properties:
  cpu:
    description: 'INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
      Important: Run "make" to regenerate code after modifying this file'
    type: string
  memory:
    type: string

Modify yaml:

apiVersion: infra.sealyun.com/v1
kind: VirtulMachine
metadata:
  name: virtulmachine-sample
spec:
  cpu: "1"
  memory: "2G"

# kubectl apply -f config/samples 
virtulmachine.infra.sealyun.com "virtulmachine-sample" configured
# kubectl get virtulmachines.infra.sealyun.com virtulmachine-sample -o yaml 
apiVersion: infra.sealyun.com/v1
kind: VirtulMachine
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"infra.sealyun.com/v1","kind":"VirtulMachine","metadata":{"annotations":{},"name":"virtulmachine-sample","namespace":"default"},"spec":{"cpu":"1","memory":"2G"}}
  creationTimestamp: 2019-07-26T08:47:34Z
  generation: 2
  name: virtulmachine-sample
  namespace: default
  resourceVersion: "14811698"
  selfLink: /apis/infra.sealyun.com/v1/namespaces/default/virtulmachines/virtulmachine-sample
  uid: 030e2b9a-af82-11e9-b63e-5254bc16e436
spec:      # 新的CRD已生效
  cpu: "1"
  memory: 2G 

Status Similarly, it will not go up, for example, I put a status Riga Create, represents the controller to go to create virtual machines (mainly some control logic level), the status changed to create the finished Running

Reconcile The only interface implemented

controller in rotation with the event listener interfaces are encapsulated in this one it. You do not need to be concerned about how the event listener.

Gets the virtual machine information

func (r *VirtulMachineReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx = context.Background()
    _ = r.Log.WithValues("virtulmachine", req.NamespacedName)

    vm := &v1.VirtulMachine{}
    if err := r.Get(ctx, req.NamespacedName, vm); err != nil { # 获取VM信息
        log.Error(err, "unable to fetch vm")
    } else {
        fmt.Println(vm.Spec.CPU, vm.Spec.Memory) # 打印CPU内存信息
    }

    return ctrl.Result{}, nil
}

make && make install && make runThis time to create a virtual machine kubectl apply -f config/samples, it will log the output from the CPU memory. List interfaces Similarly, I will not go into details

r.List(ctx, &vms, client.InNamespace(req.Namespace), client.MatchingField(vmkey, req.Name))

update status

In the status field is added in the status structure:

type VirtulMachineStatus struct {
    Status string `json:"status"`
}

controller to go update status:

vm.Status.Status = "Running"
if err := r.Status().Update(ctx, vm); err != nil {
    log.Error(err, "unable to update vm status")
}

If there is: the server could not find the requested resourcethis error, then the CRD structure needs to add a comment // +kubebuilder:subresource:status:

// +kubebuilder:subresource:status
// +kubebuilder:object:root=true

type VirtulMachine struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   VirtulMachineSpec   `json:"spec,omitempty"`
    Status VirtulMachineStatus `json:"status,omitempty"`
}

That's all

After the compilation start to apply the discovery state has become a running:

# kubectl get virtulmachines.infra.sealyun.com virtulmachine-sample -o yaml
...
status:
  status: Running 

delete

time.Sleep(time.Second * 10)
if err := r.Delete(ctx, vm); err != nil {
    log.Error(err, "unable to delete vm ", "vm", vm)
}

After the 10s we will not GET

Delete collector Finalizers

If you do not use Finalizers, kubectl delete data directly deleted etcd, when the controller has been unable to get the CRD think'll get:

ERRO[0029] VirtulMachine.infra.sealyun.com "virtulmachine-sample" not foundunable to fetch vm  source="virtulmachine_controller.go:48"

So when we need to create CRD add Finalizer:

vm.ObjectMeta.Finalizers = append(vm.ObjectMeta.Finalizers, "virtulmachine.infra.sealyun.com")

Then when you remove it will only delete marked with a timestamp to the CRD for us to do the follow-up process, we deal with the end delete Finalizers:

如果 DeleteionTimestamp不存在
    如果没有Finalizers
        加上Finalizers,并更新CRD
要不然，说明是要被删除的
    如果存在Finalizers，删除Finalizers,并更新CRD

See a complete code example:

if cronJob.ObjectMeta.DeletionTimestamp.IsZero() {
        if !containsString(cronJob.ObjectMeta.Finalizers, myFinalizerName) {
            cronJob.ObjectMeta.Finalizers = append(cronJob.ObjectMeta.Finalizers, myFinalizerName)
            if err := r.Update(context.Background(), cronJob); err != nil {
                return ctrl.Result{}, err
            }
        }
    } else {
        if containsString(cronJob.ObjectMeta.Finalizers, myFinalizerName) {
            if err := r.deleteExternalResources(cronJob); err != nil {
                return ctrl.Result{}, err
            }

            cronJob.ObjectMeta.Finalizers = removeString(cronJob.ObjectMeta.Finalizers, myFinalizerName)
            if err := r.Update(context.Background(), cronJob); err != nil {
                return ctrl.Result{}, err
            }
        }
    }

webhook

kuberentes 有 三种 webhook, webhook admission, authorization and webhook CRD conversion webhook.

For example, here we give the CRD to set some default values, or is less filling some of the parameters the user to create, create, and so we have to ban these things.

Webhook use is also very simple, just give structure to achieve defined Defaulterand Validatorinterfaces can be.

Other interfaces

Reconcile polymerized structure Client interface, so all client are methods can be called directly, mostly related to the operation of the CRD object

type Client interface {
    Reader
    Writer
    StatusClient
}

// Reader knows how to read and list Kubernetes objects.
type Reader interface {
    // Get retrieves an obj for the given object key from the Kubernetes Cluster.
    // obj must be a struct pointer so that obj can be updated with the response
    // returned by the Server.
    Get(ctx context.Context, key ObjectKey, obj runtime.Object) error

    // List retrieves list of objects for a given namespace and list options. On a
    // successful call, Items field in the list will be populated with the
    // result returned from the server.
    List(ctx context.Context, list runtime.Object, opts ...ListOptionFunc) error
}

// Writer knows how to create, delete, and update Kubernetes objects.
type Writer interface {
    // Create saves the object obj in the Kubernetes cluster.
    Create(ctx context.Context, obj runtime.Object, opts ...CreateOptionFunc) error

    // Delete deletes the given obj from Kubernetes cluster.
    Delete(ctx context.Context, obj runtime.Object, opts ...DeleteOptionFunc) error

    // Update updates the given obj in the Kubernetes cluster. obj must be a
    // struct pointer so that obj can be updated with the content returned by the Server.
    Update(ctx context.Context, obj runtime.Object, opts ...UpdateOptionFunc) error

    // Patch patches the given obj in the Kubernetes cluster. obj must be a
    // struct pointer so that obj can be updated with the content returned by the Server.
    Patch(ctx context.Context, obj runtime.Object, patch Patch, opts ...PatchOptionFunc) error
}

// StatusClient knows how to create a client which can update status subresource
// for kubernetes objects.
type StatusClient interface {
    Status() StatusWriter
}

Scan code concern sealyun:

Discussion can add QQ group: 98,488,045