Eastcom's cloud computing experience based on KubeSphere

Author: Zhou Feng Wu Changtai

Company Profile

Founded in 1958, Eastern Communication Co., Ltd. (hereinafter referred to as "Eastern Communication") is a total solution provider integrating hardware equipment, software and services. The company was successfully restructured and listed in 1996, becoming a state-controlled listed company that simultaneously issues A shares and B shares on the Shanghai Stock Exchange. The company's business mainly includes: private network communication and information security products and solutions, public network communication related products and ICT services, financial electronic equipment and software products, and intelligent manufacturing business.

During the 14th Five-Year Plan period, the company's main business focuses on the "information and communication industry" based on private network communication, public network communication, and ICT services, and the three major industries of "financial technology industry" and "smart manufacturing industry" based on financial electronics, and continues to innovate, transform and upgrade around the main business. Shouldering the mission of "creating value with science and technology, building a better life together", Eastern Communication adheres to the concept of "integrity, pragmatism, innovation, and win-win", and is committed to providing customers with high-quality products, convenient experience, perfect solutions and satisfactory services, and strives to become a leading enterprise with a dominant brand in the international market, continuous innovation and development!

Technology Status

Before using KubeSphere, the company has already begun to study cloud native and explore the way to apply to the cloud. However, due to the different time for each team to contact and learn cloud native technology, there are still certain thresholds for the use of native Kubernetes. And with the increase of the company's cloud infrastructure and usage teams, it also increases the difficulty for the cloud platform team to manage the cloud.
The delivery method of the project is still the traditional waterfall development model. There is no efficient process. In different environments, corresponding development or operation and maintenance personnel need to deploy manually, which increases the chance of differences and errors.

team size

The company has a national enterprise technology center and a post-doctoral workstation, undertakes a number of national key research and development projects, and has won the National Science and Technology Progress Award for many times. It has more than 2,500 employees, of which more than 75% are technical professionals.

background introduction

In recent years, cloud-native technology is no longer a new term. According to Gartner, an authoritative IT research company, by 2025, cloud-native platforms will become the basis of more than 95% of new digital plans, while in 2021 this proportion will be less than 40%. The huge advantages of cloud native technology in efficiency make it gradually become the mainstream trend of IT development.

Cloud-native technology can provide us with more choices for some of the business needs put forward by users. Taking a recent project of our company as an example, the user requires to provide active and standby instances for each service line, and also needs to provide disaster recovery backup. In the traditional deployment mode, the number of servers to be purchased is four times the number of service lines. However, using cloud-native technology to isolate business lines through containers can greatly reduce the cost of infrastructure deployment. Using Kubernetes as an orchestration engine to deploy services in the form of declarative scripts standardizes and simplifies the deployment process and improves repeatability and consistency.

The examples mentioned above are only part of the changes brought about by cloud-native technology. As the business undergoes more in-depth cloud transformation, it can also bring more value in terms of business elasticity, high availability, and agile development.

Selection instructions

The process of tool selection

The company's encounter with KubeSphere began with the selection of Dashboard for Kubernetes. Before using KubeSphere, we also researched and tried many other projects, such as the official Kubernetes Dashboard and similar products in other markets. The official Kubernetes Dashboard has relatively simple functions, and only provides management of cluster resources. Although similar products in the market have more functions, they still cannot fully meet our needs.

Reasons to choose KubeSphere

The reasons for finally choosing KubeSphere are as follows:

First of all, KubeSphere is not just a simple Dashboard. In addition to Kubernetes resource management, it also provides many additional functions such as unified cluster management, powerful observability including logging, auditing, and monitoring. At the same time, it provides a deployment method of pluggable components, which not only meets the functional diversity, but also meets the flexibility requirements of deployment.
Secondly, KubeSphere integrates DevOps automation process well, and DevOps automation process is the best practice method to improve software delivery speed, ensure quality and reliability.
Finally, what we value is KubeSphere's active open source community. As an open source product, a vibrant open source community can provide sufficient capabilities and technical knowledge support, so that everyone can share the dividends brought by the open source model.

practice process

Infrastructure and Deployment Architecture

In the process of using KubeSphere, we first adjusted the existing cluster resources and the overall organization structure on the cloud with the help of the multi-cluster and multi-tenant functions provided by KubeSphere. The figure below shows our current organization structure of Eastcom Cloud based on KubeSphere.

Using the multi-cluster management function, the existing cluster resources are managed in a unified manner. It not only realizes the isolation of different environments, but also realizes the unified management across clusters. Through a unified control plane, applications and their copies are distributed to different clusters, enabling centralized monitoring, log systems, events, and audit logs.

Secondly, with the help of the multi-tenant management function, create their own enterprise space (WorkSpace) for different business units, and control resource access rights in the enterprise space. Each enterprise space is divided into different departments according to the development, testing, and operation and maintenance groups, and a project space (NameSpace) is created for each project. As a logical unit of authority management, a department has the corresponding authority of the project by setting project roles, and then assigning users to the department simplifies the process of assigning authority to new members.

Introduce the DevOps workflow, use S2I and B2I to realize the rapid delivery of images, and use the Jenkins pipeline to realize CI/CD, helping the team to achieve rapid, safe and reliable delivery of software.

With the help of KubeSphere's application store function to manage the entire life cycle of the application (submission, review, testing, release, upgrade and delisting), we also deployed Harbor to manage container images.

storage implementation

In order to solve the problem of persistent storage, the storageclass based on nfs-subdir-external-provisioner is used as the storage class, and local storage is supported for businesses that are inconvenient to transform. Users create business Pods and PVCs to apply for a certain amount of storage resources. Cluster administrators manage storage classes and PVs. If NFS storage is used, nfs-subdir-external-provisioner automatically creates PVs based on the storage size requested by PVCs. If local storage is used, administrators are responsible for manually creating PVs.

Logging and Monitoring Solutions

In order to achieve persistent storage of log files, the logs read from the container are readable, and will not be lost after restart and power failure. Use EFK (Elasticsearch + Fluentd + Kibana) log collection and analysis solution to help effectively manage and analyze large-scale log data. Log collection targets include: Docker component logs, business container logs, and management plane component logs. Currently, the Elasticsearch component directly uses the built-in ES of KubeSphere. Later, in order to improve performance, it is considered to connect to the external Elasticsearch.

Introduce the Prometheus service as the monitoring center to regularly collect indicator data at three levels of business containers, control plane components, and physical servers, and display detailed information through the KubeSphere interface or Grafana interface. Use various indicators collected to judge whether the business or cluster is normal. For abnormalities, the alarm function of Prometheus can be used to forward the information, and the alarm information will be synchronized to the network management (internal management platform) or notified to the operation and maintenance personnel through email, SMS, DingTalk, etc.

multi-tenant management

Here we want to focus on sharing our analysis of the multi-tenant structure of KubeSphere. On the basis of the basic logical isolation capabilities provided by Kubernetes' RBAC and namespace, KubeSphere has added enterprise space to provide cross-cluster and cross-project (that is, the namespace in Kubernetes) shared resource capabilities and permission control. The following analysis is based on the KubeSphere V3.3.0 version used by our company.

multi-tenant element	Resource Type	source
user	`users.iam.kubesphere.io`	KubeSphere crds
platform role	`globalroles.iam.kubesphere.io`	KubeSphere crds
Platform Role Binding	`globalrolebindings.iam.kubesphere.io`	KubeSphere crds
corporate space	`workspaces.tenant.kubesphere.io`	KubeSphere crds
	`workspacesTemplate.tenant.kubesphere.io`	KubeSphere crds
Enterprise Space Role	`workspaceroles.iam.kubesphere.io`	KubeSphere crds
Enterprise space role binding	`workspacerolebindings.iam.kubesphere.io`	KubeSphere crds
project	`namespace`	K8s comes with
project role	`roles`	K8s comes with
project role binding	`rolebindings`	K8s comes with

UserUsers

A userusers.iam.kubesphere.io is an account instance of KubeSphere, an entity account that logs in to the KubeSphere console, and uses resources to abstract users in KubeSphere .

Information such as user name, password, and last login time are saved in the user information.

platform role globalroles

globalroles.iam.kubesphere.ioPlatform roles use resource abstraction in KubeSphere . After the installation is successful, 4 built-in platform roles will be pre-created, and each platform role is actually associated with one or more platform template roles (the permissions of the platform role are the set of associated template role permissions).

built-in role	describe
workspaces-manager	The enterprise space administrator manages all enterprise spaces on the platform.
users-manager	User administrator, manage all users of the platform.
platform-regular	Ordinary users of the platform do not have any resource operation authority before being invited to join the enterprise space or cluster.
platform-admin	Platform administrators can manage all resources within the platform.

Taking the users-manager role as an example, it is associated with four template roles of user view, role view, user management, and role management, which allows the users-manager role to have the corresponding resource access rights under these four role templates.

Platform role binding globalrolebindings

After creating a user, you need to assign a platform role to the user, and use globalrolebindings.iam.kubesphere.ioresources to abstract the binding relationship between the user and the platform role.

When creating a new platform role, editing permissions are required. At this time, the pre-created role template is associated with the newly created platform role.

Enterprise SpaceWorkSpace

An enterprise space is a logical unit used to manage projects, DevOps projects, application templates, and application warehouses in KubeSphere. Access to resources can be controlled within enterprise spaces, and resources can be securely shared within teams. The enterprise space is abstracted using workspaces.tenant.kubesphere.ioand resources in KubeSphere .workspacesTemplate.tenant.kubesphere.io

enterprise space roles workspaceroles

workspaceroles.iam.kubesphere.ioEnterprise space roles use resource abstraction in KubeSphere . After the installation is successful, 4 built-in enterprise space roles will be pre-created, and each enterprise space role is actually associated with one or more enterprise space template roles (the permissions of the enterprise space role are the set of permissions associated with the enterprise space template roles).

name	describe
workspace-viewer	The enterprise space observer can view all resources in the enterprise space.
workspace-self-provisioner	Ordinary members of the enterprise space can view enterprise settings, manage application templates, create projects and DevOps projects.
workspace-regular	Ordinary members of the enterprise space can view the settings of the enterprise space.
workspace-admin	The enterprise space administrator can manage all resources in the enterprise space.

Taking system-workspace-viewer as an example, it is associated with eight template roles: workspace setting view, role view, member view, department view, project view, DevOps project view, application template view, and application warehouse view.

enterprise space role bindings workspacerolebindings

In the enterprise space, you can invite users to join the enterprise space at the member of the enterprise space. When inviting users to join, you need to assign enterprise space roles to them, and use resources workspacerolebindings.iam.kubesphere.ioto abstract the binding relationship between users and enterprise space roles.

Here we take the default administrator role of the enterprise space as an example (the role binding of the administrator is bound when the administrator is selected when creating the enterprise space).

Project NameSpace

Projects in KubeSphere are namespaces in K8s, and NameSpace resources are used to abstract projects. The relationship between enterprise space and projects is one-to-many, that is, multiple projects can be created under one enterprise space. The default namespace of K8s and the namespace related to the KubeSphere system belong to the default enterprise space system-workspace.

demo-wsNext, create a new project under the enterprise space created earlier demo-ws-project-01.

Project role roles

In KubeSphere, project roles correspond to rolesthe resources that come with K8s. However, when creating a new project through KubeSphere, three built-in project roles are automatically created, and each project role is actually associated with one or more project template roles (the permissions of a project role are the set of permissions associated with project template roles).

built-in role	describe
viewer	Project observers can view all resources under the project.
operator	Project maintainers can manage resources under the project except users and roles.
admin	Project administrators can perform all operations on all resources under the project. This role has full control over all resources under the project.

The figure below shows the demo-ws-project-01project roles under the project. If you want to view the roles under other projects, you need to -nspecify the project name (namespace) with parameters.

Taking the viewer role as an example, the following project template roles are associated.

Project role binding rolebindings

Under the project, you can invite users to join the project from the project members. When inviting users to join, you also need to assign project roles to them. At this time, the resources that come with K8s are used rolebindings.

Take demo-ws-project-01the binding relationship under the project as an example.

As can be seen from the figure below, the admin user is bound to the admin role under this project.

Effect

Before using KubeSphere, we have the following feelings about the management and use of Kubernetes clusters:

After using KubeSphere, multiple clusters with original decentralized management logically form a unified environment. Many operations of cluster management do not need to be connected to different clusters for operation. The status monitoring of multiple clusters can also be obtained through a unified entrance, which greatly improves the efficiency of management.
Project delivery and deployment are more standardized, which reduces the differences that may arise due to the environment and different personnel operations. In particular, the application store makes applications as readily available as commodities on the shelf, and the deployment of applications can be installed with one click after only a few modifications.
The friendly user interface and visual operation mode lower the threshold for other teams in the company to use the cloud, so that even teams that are new to cloud native can get started quickly.

future plan

In the future, we plan to explore and use more KubeSphere component functions, such as audit logs, service mesh, etc. According to the internal needs of the Eastcom team, the source code of KubeSphere will be re-developed, and strive to give back to the community. Actively participate in community activities and have more in-depth exchanges with the community.

At the same time, we also have some suggestions for KubeSphere:

The monitoring data that can be provided on the KubeSphere page is only basic data such as CPU and memory, and the log retrieval function is not particularly convenient. We now use it by manually jumping to the Grafana and Kibana pages, hoping to have better logging and monitoring functions in the future.
At present, the official documentation is relatively simple, and some functions are only briefly mentioned. It is hoped that the documentation can be refined, and it is best to produce some video tutorials.
The annotation information in the source code is relatively small, which increases the difficulty of interpreting the source code. It is also hoped that the official can release some documents to guide the interpretation of the source code.

This article is published by OpenWrite, a multi-post platform for blogging !