Source code interpretation: how KubeVela converts appfile into K8s specific resource objects

Author | Fan Dayong

KubeVela is a simple, easy-to-use and highly scalable cloud-native application management engine based on Kubernetes and the cloud-native application development model OAM jointly released by Alibaba Cloud and Microsoft Cloud.

KubeVela builds a specific implementation based on the OAM model, written in Golang, can build a cloud-native application platform for users end-to-end, providing a relatively complete solution.

The KubeVela project was launched in the community in July 2020. It has been welcomed by community volunteers including engineers from Alibaba, Microsoft, Crossplane and other companies, and they have joined the project development work together. They summed up various experiences and lessons in OAM practice into the KubeVela project.

The main purpose of this article is to explore how KubeVela converts an appfile into a specific resource object in K8s.

The process is generally divided into two stages:

1. Convert appfile to application in K8s
2. application is converted to the corresponding K8s resource object

# vela.yaml
name: test
services:
  nginx:
    type: webservice
    image: nginx
    env:
    - name: NAME
      value: kubevela

    # svc trait
    svc:
      type: NodePort
      ports:
      - port: 80
        nodePort: 32017

The deployment can be completed with the vela up command.

vela up command

Suggestion: Before looking at the vela command line tool code, let's take a brief look at the cobra framework.

// references/cli/up.go
// NewUpCommand will create command for applying an AppFile
func NewUpCommand(c types.Args, ioStream cmdutil.IOStreams) *cobra.Command {
  cmd := &cobra.Command{
    Use:                   "up",
    DisableFlagsInUseLine: true,
    Short:                 "Apply an appfile",
    Long:                  "Apply an appfile",
    Annotations: map[string]string{
      types.TagCommandType: types.TypeStart,
    },
    PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
      return c.SetConfig()
    },
    RunE: func(cmd *cobra.Command, args []string) error {
      velaEnv, err := GetEnv(cmd)
      if err != nil {
        return err
      }
      kubecli, err := c.GetClient()
      if err != nil {
        return err
      }

      o := &common.AppfileOptions{
        Kubecli: kubecli,
        IO:      ioStream,
        Env:     velaEnv,
      }
      filePath, err := cmd.Flags().GetString(appFilePath)
      if err != nil {
        return err
      }
      return o.Run(filePath, velaEnv.Namespace, c)
    },
  }
  cmd.SetOut(ioStream.Out)

  cmd.Flags().StringP(appFilePath, "f", "", "specify file path for appfile")
  return cmd
}

The source code above shows the entry point of the vela up command.

In the PresistentPreRunE function, the Kuberentes configuration information kubeconfig is injected by calling c.SetConfig().

In the RunE function:

• First, get the env variable of vela, velaEnv.Namespace corresponds to the namespace of Kubernetes.

• Second, get the Kubernetes client, kubectl.

• Next, use the Kubernetes client and vleaEnv to build the AppfileOptions needed to render the Appfile.

• Finally, call o.Run(filePath, velaEnv.Namespace, c).
  • This function requires three parameters, among which filePath is used to specify the location of the appfile, velaEnv.Namespace and c are used to create the rendered Application in the specified namespace.
    • filePath: path of appfile
    • velaEnv.Namespace: corresponding to the namespace of K8s
    • c: K8s client

How to convert an appfile to an Application in Kubernetes

• Starting point: appfile

• End: applicatioin

• 路径：appfile -> application (services -> component)
  • comp[workload, traits]

1. Starting point: AppFile

// references/appfile/api/appfile.go
// AppFile defines the spec of KubeVela Appfile
type AppFile struct {
  Name       string             `json:"name"`
  CreateTime time.Time          `json:"createTime,omitempty"`
  UpdateTime time.Time          `json:"updateTime,omitempty"`
  Services   map[string]Service `json:"services"`
  Secrets    map[string]string  `json:"secrets,omitempty"`

  configGetter config.Store
  initialized  bool
}

// NewAppFile init an empty AppFile struct
func NewAppFile() *AppFile {
  return &AppFile{
    Services:     make(map[string]Service),
    Secrets:      make(map[string]string),
    configGetter: &config.Local{},
  }
}

// references/appfile/api/service.go
// Service defines the service spec for AppFile, it will contain all related information including OAM component, traits, source to image, etc...
type Service map[string]interface{}

The above two pieces of code are the declarations of AppFile on the client side. Vela will read the yaml file in the specified path and assign it to an AppFile.

// references/appfile/api/appfile.go
// LoadFromFile will read the file and load the AppFile struct
func LoadFromFile(filename string) (*AppFile, error) {
  b, err := ioutil.ReadFile(filepath.Clean(filename))
  if err != nil {
    return nil, err
  }
  af := NewAppFile()
  // Add JSON format appfile support
  ext := filepath.Ext(filename)
  switch ext {
  case ".yaml", ".yml":
    err = yaml.Unmarshal(b, af)
  case ".json":
    af, err = JSONToYaml(b, af)
  default:
    if json.Valid(b) {
      af, err = JSONToYaml(b, af)
    } else {
      err = yaml.Unmarshal(b, af)
    }
  }
  if err != nil {
    return nil, err
  }
  return af, nil
}

The following is the data loaded into AppFile after reading the vela.yaml file:

# vela.yaml
name: test
services:
  nginx:
    type: webservice
    image: nginx
    env:
    - name: NAME
      value: kubevela

    # svc trait
    svc:
      type: NodePort
      ports:
      - port: 80
        nodePort: 32017

Name: test
CreateTime: 0001-01-01 00:00:00 +0000 UTC
UpdateTime: 0001-01-01 00:00:00 +0000 UTC
Services： map[
             nginx: map[
               env: [map[name: NAME value: kubevela]] 
               image: nginx 
               svc: map[ports: [map[nodePort: 32017 port: 80]] type: NodePort] 
               type: webservice
            ]
          ]
Secrets    map[]
configGetter: 0x447abd0 
initialized: false

2. End point: application

// apis/core.oam.dev/application_types.go
type Application struct {
  metav1.TypeMeta   `json:",inline"`
  metav1.ObjectMeta `json:"metadata,omitempty"`

  Spec   ApplicationSpec `json:"spec,omitempty"`
  Status AppStatus       `json:"status,omitempty"`
}

// ApplicationSpec is the spec of Application
type ApplicationSpec struct {
  Components []ApplicationComponent `json:"components"`

  // TODO(wonderflow): we should have application level scopes supported here

  // RolloutPlan is the details on how to rollout the resources
  // The controller simply replace the old resources with the new one if there is no rollout plan involved
  // +optional
  RolloutPlan *v1alpha1.RolloutPlan `json:"rolloutPlan,omitempty"`
}

The above code is the declaration of Application, combined with .vela/deploy.yaml (see the code below), it can be seen that to render an AppFile as an Application is mainly to convert AppFile's Services into Application's Components.

# .vela/deploy.yaml
apiVersion: core.oam.dev/v1alpha2
kind: Application
metadata:
  creationTimestamp: null
  name: test
  namespace: default
spec:
  components:
  - name: nginx
    scopes:
      healthscopes.core.oam.dev: test-default-health
    settings:
      env:
      - name: NAME
        value: kubevela
      image: nginx
    traits:
    - name: svc
      properties:
        ports:
        - nodePort: 32017
          port: 80
        type: NodePort
    type: webservice
status: {}

3. Path: Services -> Components

Combining the above content, it can be seen that converting Appfile into Application is mainly rendering Services into Components.

// references/appfile/api/appfile.go
// BuildOAMApplication renders Appfile into Application, Scopes and other K8s Resources.
func (app *AppFile) BuildOAMApplication(env *types.EnvMeta, io cmdutil.IOStreams, tm template.Manager, silence bool) (*v1alpha2.Application, []oam.Object, error) {
  ...
  servApp := new(v1alpha2.Application)
  servApp.SetNamespace(env.Namespace)
  servApp.SetName(app.Name)
  servApp.Spec.Components = []v1alpha2.ApplicationComponent{}
  for serviceName, svc := range app.GetServices() {
    ...
    // 完成 Service 到 Component 的转化
    comp, err := svc.RenderServiceToApplicationComponent(tm, serviceName)
    if err != nil {
      return nil, nil, err
    }
    servApp.Spec.Components = append(servApp.Spec.Components, comp)
  }
  servApp.SetGroupVersionKind(v1alpha2.SchemeGroupVersion.WithKind("Application"))
  auxiliaryObjects = append(auxiliaryObjects, addDefaultHealthScopeToApplication(servApp))
  return servApp, auxiliaryObjects, nil
}

The above code is where vela converts Appfile into Application code. Among them, comp, err := svc.RenderServiceToApplicationComponent(tm, serviceName) completes the transformation from Service to Component.

// references/appfile/api/service.go
// RenderServiceToApplicationComponent render all capabilities of a service to CUE values to KubeVela Application.
func (s Service) RenderServiceToApplicationComponent(tm template.Manager, serviceName string) (v1alpha2.ApplicationComponent, error) {

  // sort out configs by workload/trait
  workloadKeys := map[string]interface{}{}
  var traits []v1alpha2.ApplicationTrait

  wtype := s.GetType()
  comp := v1alpha2.ApplicationComponent{
    Name:         serviceName,
    WorkloadType: wtype,
  }

  for k, v := range s.GetApplicationConfig() {
    // 判断是否为 trait
    if tm.IsTrait(k) {
      trait := v1alpha2.ApplicationTrait{
        Name: k,
      }
      ....
      // 如果是 triat 加入 traits 中
      traits = append(traits, trait)
      continue
    }
    workloadKeys[k] = v
  }

  // Handle workloadKeys to settings
  settings := &runtime.RawExte nsion{}
  pt, err := json.Marshal(workloadKeys)
  if err != nil {
    return comp, err
  }
  if err := settings.UnmarshalJSON(pt); err != nil {
    return comp, err
  }
  comp.Settings = *settings

  if len(traits) > 0 {
    comp.Traits = traits
  }

  return comp, nil
}

4. Summary

Execute the vela up command, render the appfile as Application, write the data to .vela/deploy.yaml, and create it in K8s.

How is Application converted to corresponding K8s resource object

• Starting point: Application
• 中点：ApplicationConfiguration, Component
• End point: Deployment, Service
• path:
  • application_controller
  • applicationconfiguration controller

[Suggestion] > Learn about the content:>-client-to

• controller-runtime
• operator

1. Application

# 获取集群中的 Application
$ kubectl get application
NAMESPACE   NAME   AGE
default     test   24h

2. ApplicationConfiguration 和 Component

After the application controller obtains the Application resource object, it will create the corresponding ApplicationConfiguration and Component based on its content.

# 获取 ApplicationConfiguration 和 Component
$ kubectl get ApplicationConfiguration,Component
NAME                                         AGE
applicationconfiguration.core.oam.dev/test   24h

NAME                           WORKLOAD-KIND   AGE
component.core.oam.dev/nginx   Deployment      24h

Import Component by name in ApplicationiConfiguration:

3. application controller

Basic logic:

• Get an Application resource object.

• Render Application resource objects as ApplicationConfiguration and Component.

• Create ApplicationConfiguration and Component resource objects.

Code:

// pkg/controller/core.oam.dev/v1alpha2/application/application_controller.go

// Reconcile process app event
func (r *Reconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
  ctx := context.Background()
  applog := r.Log.WithValues("application", req.NamespacedName)

  // 1. 获取 Application
  app := new(v1alpha2.Application)
  if err := r.Get(ctx, client.ObjectKey{
    Name:      req.Name,
    Namespace: req.Namespace,
  }, app); err != nil {
    ...
  }

  ...

  // 2. 将 Application 转换为 ApplicationConfiguration 和 Component
  handler := &appHandler{r, app, applog}
  ...
  appParser := appfile.NewApplicationParser(r.Client, r.dm)
  ...
  appfile, err := appParser.GenerateAppFile(ctx, app.Name, app)
  ...
  ac, comps, err := appParser.GenerateApplicationConfiguration(appfile, app.Namespace)
  ...

  // 3. 在集群中创建 ApplicationConfiguration 和 Component 
  // apply appConfig & component to the cluster
  if err := handler.apply(ctx, ac, comps); err != nil {
    applog.Error(err, "[Handle apply]")
    app.Status.SetConditions(errorCondition("Applied", err))
    return handler.handleErr(err)
  }

  ...
  return ctrl.Result{}, r.UpdateStatus(ctx, app)
}

4. applicationconfiguration controller

Basic logic:

• Get the ApplicationConfiguration resource object.

• Loop through, get each Component and render workload and trait into corresponding K8s resource objects.

• Create the corresponding K8s resource object.

Code:

// pkg/controller/core.oam.dev/v1alpha2/applicationcinfiguratioin/applicationconfiguratioin.go

// Reconcile an OAM ApplicationConfigurations by rendering and instantiating its
// Components and Traits.
func (r *OAMApplicationReconciler) Reconcile(req reconcile.Request) (reconcile.Result, error) {
  ...
  ac := &v1alpha2.ApplicationConfiguration{}
  // 1. 获取 ApplicationConfiguration
  if err := r.client.Get(ctx, req.NamespacedName, ac); err != nil {
    ...
  }
  return r.ACReconcile(ctx, ac, log)
}

// ACReconcile contains all the reconcile logic of an AC, it can be used by other controller
func (r *OAMApplicationReconciler) ACReconcile(ctx context.Context, ac *v1alpha2.ApplicationConfiguration,
  log logging.Logger) (result reconcile.Result, returnErr error) {

  ...
  // 2. 渲染
  // 此处 workloads 包含所有Component对应的的 workload 和 tratis 的 k8s 资源对象
  workloads, depStatus, err := r.components.Render(ctx, ac)
  ...

  applyOpts := []apply.ApplyOption{apply.MustBeControllableBy(ac.GetUID()), applyOnceOnly(ac, r.applyOnceOnlyMode, log)}

  // 3. 创建 workload 和 traits 对应的 k8s 资源对象
  if err := r.workloads.Apply(ctx, ac.Status.Workloads, workloads, applyOpts...); err != nil {
    ...
  }

  ...

  // the defer function will do the final status update
  return reconcile.Result{RequeueAfter: waitTime}, nil
}

5. Summary

When vela up renders an AppFile as an Application, the subsequent process is completed by the application controller and application configuration controller.

About the Author

Dayong Fan, R&D engineer of Teamsun Tiancheng, GitHub ID: @just-do1.

Join OAM

• OAM official website:
  https://oam.dev

• KubeVela GitHub project address:
  https://github.com/oam-dev/kubevela