Analysis of the core architecture of KubeSphere

Others 2022-04-20 03:15:36 views: 0

About the author: Wan Hongming, the core contributor of KubeSphere, focuses on the field of cloud native security.

KubeSphere is a container hybrid cloud management system built on top of Kubernetes for cloud-native applications. It supports multi-cloud and multi-cluster management, provides full-stack automated operation and maintenance capabilities, helps enterprise users simplify DevOps workflow, and provides an operation-friendly wizard-style interface to help enterprises quickly build a powerful and feature-rich container cloud platform.

KubeSphere provides users with multiple functions required to build an enterprise-level Kubernetes environment, such as multi-cloud and multi-cluster management, Kubernetes resource management, DevOps, application lifecycle management, microservice governance (service mesh), log query and collection, service and Network, multi-tenant management, monitoring alarms, event and audit queries, storage management, access control, GPU support, network policies, image warehouse management, and security management.

Thanks to the excellent architecture and design of Kubernetes, KubeSphere adopts a lighter-weight architecture model and integrates resources flexibly, further enriching the K8s ecosystem.

KubeSphere Core Architecture

The core architecture of KubeSphere is shown in the following figure:

There are three main core components:

  • ks-console front-end service component
  • ks-apiserver backend service component
  • ks-controller-manager resource state maintenance component

The back-end design of KubeSphere follows the style of K8s declarative API, and all operable resources are abstracted as CustomResource as much as possible . Compared to the imperative API, the declarative API is more concise to use and provides better abstraction, telling the program the final desired state (what to do), not how to do it.

Typically, in a declarative API:

  1. Your API contains relatively few objects (resources) of small size.
  2. Objects define application or infrastructure configuration information.
  3. Object update operations are less frequent.
  4. Often a human is required to read or write objects.
  5. The main operations on objects are CRUD-style (create, read, update, and delete).
  6. Cross-object transaction support is not required: API objects represent the desired state rather than the exact actual state.

Imperative API (Imperative API) is different from declarative. The following are signs that your API may not be declarative:

  1. The client issues a "do this operation" command and then gets a synchronous response at the end of the operation.
  2. The client issues a "do this operation" command and obtains an operation ID, after which it needs to determine whether the request was successfully completed.
  3. You would compare your API to RPC.
  4. Store large amounts of data directly.
  5. Regular operations performed on objects are not CRUD-style.
  6. APIs are not easy to model with objects.

Use kube-apiserver and etcd to achieve data synchronization and data persistence, and maintain the state of these resources through ks-controller-manager to achieve the consistency of the final state. If you are familiar with K8s, you can well understand the benefits of declarative API, which is also the core part of KubeSphere.

For example, the configuration of pipelines, user credentials, user entities, and alarm notifications in KubeSphere can be abstracted into resource entities. With the mature architecture and tool chain of K8s, it can be easily combined with K8s to reduce the coupling between components and reduce the complexity of the system.

The core architecture of ks-apiserver

ks-apiserver is the core back-end component of KubeSphere, responsible for the interaction of front-end and back-end data, proxy distribution of requests, authentication and authentication. The following figure is the core architecture of ks-apiserver:

The development of ks-apiserver uses the go-restful framework, which can add multiple filters to the request link to dynamically intercept requests and responses, and implement authentication, authentication, audit logic forwarding and reverse proxy functions, KubeSphere's API style Also try to learn the K8s mode as much as possible to facilitate the use of RBAC for permission control.

Decoupling with the help of CRD + controller can greatly simplify the integration with third-party tools and software.

The K8s community also provides a rich toolchain, and with the help of controller-runtime and kubebuiler , development scaffolding can be quickly built.

API Aggregation and Permission Control

API aggregation can be realized by extending ks-apiserver, and functions such as function expansion and aggregation query can be further realized. The following specifications need to be followed in the API development process to integrate with KubeSphere's tenant system and resource system.

  • API Aggregation
  • Access control
  • CRD + controller

API Specification

# 通过 api group 进行分组
/apis/{api-group}/{version}/{resources}
# 示例
/apis/apps/v1/deployments
/kapis/iam.kubesphere.io/v1alpha2/users
# api core
/api/v1/namespaces

# 通过 path 区分不同的动作
/api/{version}/watch/{resources}
/api/{version}/proxy/{resources}/{resource}

# 通过 path 区分不同的资源层级
/kapis/{api-group}/{version}/workspaces/{workspace}/{resources}/{resource}
/api/{version}/namespaces/{namespace}/{resource}

Purpose of the canonical API:

  1. Better abstraction of resources, abstracting as Object is more suitable for declarative API
  2. Better management of API, version, grouping, layering, more convenient for API expansion
  3. Better integration with permission control, can easily obtain metadata from requests, apigroup, scope, version, verb

Access control

The core of KubeSphere permission control is RBAC role-based access control.

The key objects are: Role, User, RoleBinding.

Role defines the resources that a role can access.

Roles are divided according to resource levels. The roles at different levels of cluster role, workspace role, and namespace role define the resources that the role can access at the current level.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: role-grantor
rules:
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["rolebindings"]
  verbs: ["create"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["clusterroles"]
  verbs: ["bind"]
  # 忽略 resourceNames 意味着允许绑定任何 ClusterRole
  resourceNames: ["admin","edit","view"]
- nonResourceURLs: ["/healthz", "/healthz/*"] # nonResourceURL 中的 '*' 是一个全局通配符
  verbs: ["get", "post"]

RoleBinding can bind roles to a subject (Subject). Principals can be groups, users or service accounts.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-grantor-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: role-grantor
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: user-1

CRD + controller

Custom Resource is an extension to the Kubernetes API, which can expand the K8s API through dynamic registration. Users can use kubectl to create and access objects within it, just like manipulating built-in resources.

The resource is abstracted through CRD, and the resource state is maintained by monitoring the resource change through the controller. The core of the controller is Reconcile, which, like his meaning, maintains the resource state through passive and timed triggering until it reaches the declared state.

Taking the User resource as an example, we can define the CRD of the following structure to abstract the User:

apiVersion: iam.kubesphere.io/v1alpha2
kind: User
metadata:
  annotations:
    iam.kubesphere.io/last-password-change-time: "2021-05-12T05:50:07Z"
  name: admin
  resourceVersion: "478503717"
  selfLink: /apis/iam.kubesphere.io/v1alpha2/users/admin
  uid: 9e438fcc-f179-4254-b534-e913dfd7a727
spec:
  email: [email protected]
  lang: zh
  description: 'description'
  password: $2a$10$w312tzLTvXObnfEYiIrk9u5Nu/reJpwQeI66vrM1XJETWtpjd1/q2
status:
  lastLoginTime: "2021-06-08T06:37:36Z"
  state: Active

The corresponding API is:

# 创建
POST /apis/iam.kubesphere.io/v1alpha2/users

# 删除
DELETE /apis/iam.kubesphere.io/v1alpha2/users/{username}

# 修改
PUT /apis/iam.kubesphere.io/v1alpha2/users/{username}
PATCH /apis/iam.kubesphere.io/v1alpha2/users/{username}

# 查询
GET /apis/iam.kubesphere.io/v1alpha2/users
GET /apis/iam.kubesphere.io/v1alpha2/users/{username}

ks-apiserver is responsible for writing these data into K8s and then synchronize them to each replica by the informer.

ks-controller-manager maintains the resource status by monitoring data changes. Taking creating a user as an example, after POST /apis/iam.kubesphere.io/v1alpha2/userscreating a user, the user controller will synchronize the user resource status.

func (c *userController) reconcile(key string) error {
	// Get the user with this name
	user, err := c.userLister.Get(key)

        if err != nil {
            // The user may no longer exist, in which case we stop
            // processing.
            if errors.IsNotFound(err) {
                utilruntime.HandleError(fmt.Errorf("user '%s' in work queue no longer exists", key))
                return nil
            }
            klog.Error(err)
            return err
        }

	if user, err = c.encryptPassword(user); err != nil {
		klog.Error(err)
		return err
	}

	if user, err = c.syncUserStatus(user); err != nil {
		klog.Error(err)
		return err
	}

	// synchronization through kubefed-controller when multi cluster is enabled
	if c.multiClusterEnabled {
		if err = c.multiClusterSync(user); err != nil {
			c.recorder.Event(user, corev1.EventTypeWarning, controller.FailedSynced, fmt.Sprintf(syncFailMessage, err))
			return err
		}
	}

	c.recorder.Event(user, corev1.EventTypeNormal, successSynced, messageResourceSynced)
	return nil
}

Through the declarative API, the complex logic is placed in the controller for processing, which is convenient for decoupling. It can be easily integrated with other systems and services, such as:

/apis/devops.kubesphere.io/v1alpha2/namespaces/{namespace}/pipelines
/apis/devops.kubesphere.io/v1alpha2/namespaces/{namespace}/credentials
/apis/openpitrix.io/v1alpha2/namespaces/{namespace}/applications
/apis/notification.kubesphere.io/v1alpha2/configs

The corresponding permission control policy:

Define a role that can add, delete, modify, and query user resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: user-manager
rules:
- apiGroups: ["iam.kubesphere.io"]
  resources: ["users"]
  verbs: ["create","delete","patch","update","get","list"]

Define a role that can create pipeline resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: devops-manager
rules:
- apiGroups: ["devops.kubesphere.io"]
  resources: ["pipelines"]
  verbs: ["create","delete","patch","update","get","list"]

This article is published by OpenWrite , a multi-post blog platform !

{{o.name}}
{{m.name}}

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=324116879&siteId=291194637
Recommended
Ranking
Reason Codes enhanced accounting documents
The first test case appium
Force command and dispatch emergency communication plan
Interview - What is concurrent programming
21 classic Python interview questions [recommended collection]
[Dahua Data Structure-Introduction to Data Structure ①]
Spring @ConfigurationProperties
Why do we want to broadcast live on WeChat public account? What are the advantages?
How to bring up the toolbar under the desktop of Apple laptop
Dialog: Manually enter information
Daily
More
2024-05-21(35)
2024-05-20(5)
2024-05-19(0)
2024-05-18(31)
2024-05-17(6)
2024-05-16(23)
2024-05-15(5)
2024-05-14(9)
2024-05-13(8)
2024-05-12(28)

Code World

© 2018 codetd.com