A manufacturing company's cloud-native practice based on KubeSphere

background introduction

With the increase of business upgrades and software product projects, the conventional physical machine and virtual machine methods have gradually exposed some problems:

A large number of services are deployed on virtual machines, resource estimation and hardware waste are large;
A large number of services are deployed on virtual machines, the deployment time and difficulty are relatively large, and the degree of automation is low;
Developers and operation and maintenance personnel, due to the different development and deployment service environments, often report errors due to unstable services, resulting in many barriers and low efficiency;
It is inconvenient to troubleshoot the cause of the problem, the development does not have permission to access the production environment, and the service log and service monitoring status cannot be located.

In an increasingly competitive and ever-changing market environment, companies need to continuously iteratively develop products to meet business needs and quickly respond to changes, so it is becoming more and more urgent to solve the above problems.

Selection instructions

We investigated two open source products. After comprehensive evaluation and comparison, we finally chose KubeSphere. KubeSphere is positioned as an application-centric container platform that provides a simple and easy-to-use operation interface, which reduces learning costs to a certain extent. At the same time, it integrates native Istio and other functions, which is more in line with development habits.

practice process

Accelerate development response to application requirements, deliver value quickly, and respond quickly to changes. Agile development is a process of adapting to faster changes with a short iteration cycle and maintaining incremental continuous improvement. Kubernetes + Docker is a bridge for the integration of Dev and Ops. Conversely, agile development and automated operation and maintenance promote enterprise The implementation of DevOps provides end-to-end integration of development, operation and maintenance in the whole process from demand analysis to deployment monitoring.

Infrastructure and Deployment Architecture

The construction of KubeSphere is also very simple. Install Kubernetes through KubeAdmin, and then install KubeSphere in the way recommended by KubeSphere official website. Private internal cloud platform environment to build Kubernetes and KubeSphere. The basic server uses Linux Centos 7, and the kernel version is 5.6.

When building a Kubernetes cluster, we chose to use Keepalived and HAproxy to create a high-availability Kubernetes cluster master, including load balancing entries.

Deployment reference diagram:

storage and network

At present, we are mainly connected to the distributed storage of Ceph, which serves various persistent services. For example, we will make some mirror images of Harbor, mainly Rabbitmq, Redis, MySQL, etc. The production environment is mainly some stateless development services. For example, microservices developed by Springboot, SpringCloud, and Python services. The Python service is mainly used for simple analysis of AI models.

At the same time, NFS storage is also used to store some stateful data backups and log backup files.

The network chooses Calico, a pure three-layer BGP network.

Platform and application logs, monitoring, APM

We use ELK to collect logs of various basic services and business services, and perform log alarm monitoring.

We use Prometheus+grafana to collect and monitor OS, K8s system components and Pod services.

At the same time, we use SkyWalking to monitor the API full link performance of the service.

CI/CD

The DevOps module of KubeSphere we use integrates Jenkins and the construction of the pipeline to realize the one-click process of the project from pulling code, quality inspection to project deployment. In the DevOps module, a custom GitLab warehouse is used.

The reference graphics are as follows:

Stateful Service Management

We currently manage clusters for Redis, RabbitMQ, and Elasticsearch.

Uniqueness—For a StatefulSet with N replicas, each pod is assigned a [0，N)unique sequence number within a range.
Sequentiality—The startup, update, and destruction of pods in a StatefulSet are performed sequentially by default.
Stable network identity - the pod's hostname and DNS address will not change as the pod is rescheduled.
Stable persistent storage - when the pod is rescheduled, the original PersistentVolume can still be mounted, ensuring data integrity and consistency.

Effect

KubeSphere is a very popular container orchestration tool that helps users manage and deploy containerized applications. Using KubeSphere can improve application reliability, scalability, and security.

Developers spend almost no time on software deployment and monitoring, and do not need to care about too many underlying deployment details, which saves about 30% of the time, and the product iteration speed is faster.
Permissions are managed by role, making it easier for developers to troubleshoot service errors. It is very fast to view logs, indicator data, and monitoring reports directly on the platform, saving about 20% of time.
Resource utilization is optimized and costs are reduced. In the past, we deployed on VMs. Server resources were wasted a lot, and resource utilization was often reviewed. After KubeSphere, resource utilization increased by more than 30%.

future plan

In the future, we plan to further develop and improve our infrastructure environment and DevOps full-process efficiency, covering the automated testing process. We will continue to pay attention to new technology trends (service mesh, service governance, etc.) and best practices, and make corresponding upgrades and optimizations based on business needs. We will also continue to strengthen the team's training and technical capabilities to better support the company's business development.

This article is published by OpenWrite, a multi-post platform for blogging !