Strengthen Kubernetes capabilities: use CRD to define the implementation of multi-version resources

Canwu Yao, R&D engineer of Rancher China, has 7 years of experience in the field of cloud computing, is keen on open source technology, and has rich development and practical experience in the field of cloud native related technologies.

CRD, or Custom Resource Definition, is a powerful extension mechanism in the Kubernetes API. Through CRD, users can define their own resource types to extend the API and resource types of Kubernetes. CRD definitions can be used to create multiple versions of resources, which is useful for Kubernetes cluster evolution and upgrades.

In this article, we will introduce the version concept and usage of Kubernetes CRD; discuss how to define multiple versions of resources in CRD, and discuss the implementation of resource data storage and data conversion. Finally, we'll cover how to develop a CRD webhook for CRD version compatibility in Kubernetes.

Multi-version CRD

Multi-version CRD refers to defining multiple versions of API in CRD, and each version of API can have its own rules, so that the new version of the application can be updated and upgraded without affecting the old version of the application.

To create a multi-version CRD, we need to define a Custom Resource Definition object that contains multiple version fields. Each version field has a schema, which contains properties and rules for custom resources.

To illustrate by way of example, the following is an example single-version CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: foos.sample.webhook.io
spec:
  conversion:
    strategy: None
  group: sample.webhook.io
  names:
    kind: Foo
    listKind: FooList
    plural: foos
    shortNames:
    - foo
    - foos
    singular: foo
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
        type: object
    served: true
    storage: true

The file specifies version information, resource name, type and other information. Through the kubectl apply command, the CRD file can be applied to the Kubernetes cluster to create a new resource type Foo in the cluster, for example:

apiVersion: sample.webhook.io/v1
kind: Foo
metadata:
  name: foo-sample

The above custom resource Foo has only one v1 version. If we need to update a field in the custom resource, we need to update the version of Foo. For example, to update to v2 version, we need to add the definition of v2 version in the CRD file, for example:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
    name: foos.sample.webhook.io
spec:
    versions:
    - name: v1
      schema:
        openAPIV3Schema:
          properties:
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
          type: object
      served: true
      storage: false
    - name: v2
      schema:
        openAPIV3Schema:
          properties:
            alias:
              type: string
            apiVersion:
              type: string
            kind:
              type: string
            metadata:
              type: object
          type: object
      served: true
      storage: true

In the above CRD file, we defined two versions, v1 and v2. Compared with v1, v2 adds an alias field.

resource data store

To ensure that Kubernetes can correctly persist custom resources to storage, and can correctly read and restore custom resources from storage. The spec.versions.storage field is defined in the CRD, which is used to indicate which version of the CRD is persisted to the storage. In a CRD definition, only one version of the storage field has a value of true.

In the above example, we defined two versions, v1 and v2. The storage field of the v1 version is false, and the storage field of the v2 version is true, which indicates that the resource data of the v2 version is stored in Kubernetes etcd.

So how to store the data of the unselected version? Taking the above foo-sample as an example, let’s see how it is stored in etcd. The results are as follows:

/registry/sample.webhook.io/foos/default/foo-sample
{"apiVersion":"sample.webhook.io/v2","kind":"Foo","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"sample.webhook.io/v1\",\"kind\":\"Foo\",\"metadata\":{\"annotations\":{},\"name\":\"foo-sample\",\"namespace\":\"default\"}}\n"},"creationTimestamp":"2023-05-22T09:49:03Z","generation":1,"managedFields":[{"apiVersion":"sample.webhook.io/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2023-05-22T09:49:03Z"}],"name":"foo-sample","namespace":"default","uid":"3360f129-31f7-4a3f-8991-7694524d9a78"}}

The first line is foo-sample the key, and the second line is foo-sample the content expressed in json format, where apiVersion is sample.webhook.io/v2, which indicates that even v1 version resources are stored in v2 version format.

data conversion

When a custom resource supports multiple versions, because it can only be stored in a certain version format, it is necessary to convert between the stored version and the provided version to achieve multi-version compatibility.

If the transition involves schema changes and requires custom logic, it should be done using webhooks. An example of configuring webhook in CRD is as follows:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
...
spec:  
  conversion:    
      strategy: Webhook    
      webhook:      
          clientConfig:        
            caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJ2akNDQVdPZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQkdNUnd3R2dZRFZRUUtFeE5rZVc1aGJXbGoKYkdsemRHVnVaWEl0YjNKbk1TWXdKQVlEVlFRRERCMWtlVzVoYldsamJHbHpkR1Z1WlhJdFkyRkFNVFk0TkRjegpPRGswTXpBZUZ3MHlNekExTWpJd056QXlNak5hRncwek16QTFNVGt3TnpBeU1qTmFNRVl4SERBYUJnTlZCQW9UCkUyUjVibUZ0YVdOc2FYTjBaVzVsY2kxdmNtY3hKakFrQmdOVkJBTU1IV1I1Ym1GdGFXTnNhWE4wWlc1bGNpMWoKWVVBeE5qZzBOek00T1RRek1Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMERBUWNEUWdBRUZYWDQzaEZRdjRyOApKelR1UkJ0b3lqUmRtWjhIRkExbWZnOXJnTWVlNnVXZ20wMXBNR3lSRnFna3Z1RHF5RUlMTUtCRDduQ2IrVFp3CitBR3loVWhTV0tOQ01FQXdEZ1lEVlIwUEFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWUQKVlIwT0JCWUVGRzg2VnpCcW1wRUcrUWlrS0d1SGNIQlJwS2R3TUFvR0NDcUdTTTQ5QkFNQ0Ewa0FNRVlDSVFDaQp5SFFyeGNXN3dUM0dwRWhyNklQQWpXWDVJOSt4Y0dkUGQzQURKS0hwZ2dJaEFJQmhTc00xd1hxMU80VUlaWWZwCkNWVkxwaCtvUVMvMzI5OHMwS0VqWW9FTAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==        
            service:          
                name: webhook-sample          
                namespace: default          
                path: /v1/webhook/conversion          
                port: 443      
              conversionReviewVersions:      
              - v1

If there are no schema changes, the default conversion strategy can be used , where only fields are changed None when serving different versions .apiVersion

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
...
spec:
  ...
  conversion:
    strategy: None

CRD webhook development

Like Admission Webhook , CRD webhook is also an HTTP callback, and webhook service is an HTTP service. The interface structure is as follows:

// ConversionReview describes a conversion request/response.
type ConversionReview struct {
    metav1.TypeMeta `json:",inline"`
    // request describes the attributes for the conversion request.
    // +optional
    Request *ConversionRequest `json:"request,omitempty" protobuf:"bytes,1,opt,name=request"`
    // response describes the attributes for the conversion response.
    // +optional
    Response *ConversionResponse `json:"response,omitempty" protobuf:"bytes,2,opt,name=response"`
}

The Harvester project team added CRD webhook support to the harvester webhookConverter framework. Developers only need to implement the interface and register it in the webhook server, and then start the webhook server.

// Converter is a interface to convert object to the desired version
type Converter interface {
    GroupResource() schema.GroupResource
    Convert(Object *unstructured.Unstructured, desiredAPIVersion string) (*unstructured.Unstructured, error)
}

For specific development details, please refer to the sample webhook sample .

Summarize

In the process of actual product evolution, multi-version compatibility has always been an important issue. Kubernetes CRD adopts a strategy of selecting a certain version for persistence and using webhook to realize multi-version data conversion to solve multi-version compatibility issues. This strategy is also worth learning from in other projects outside of Kubernetes.

refer to

Versions in Custom Resource Definitions：https://kubernetes.io/zh-cn/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning