Author: Xu Peng, Shenzhen Haoshanghao Information (001298), deputy technical director, responsible for the architecture design and business development of the cloud server team, embracing cloud native, willing to share, and lifelong learning.
Company Profile
Hao Shanghao Information (001298) is a comprehensive service provider in mainland China dedicated to providing comprehensive support for China's intelligent manufacturing. Headquartered in Shenzhen, with more than 500 employees, it owns subsidiaries such as Beigaozhi, Tianwu, Doudou, Milian and Taige. The main business includes electronic component distribution, IoT product design and chip customization. Haoshanghao information adopts the operation mode of "group large platform + independent business of subsidiaries". Each subsidiary operates and manages independently at the business level, and fully shares back-end resources such as warehousing and logistics, capital credit, IT information system, and program design.
Its subsidiary Soybean Electronics is committed to the overall solution of Internet smart home, providing customized components such as Bluetooth modules and WIFI modules for the Internet of Things ecology, and Milian Technology provides the overall solution development of the Internet of Things, mainly to empower shared products, such as Shared bicycles, shared charging treasures, shared tissue machines, shared coffee machines, environmentally friendly plastic bag picking machines, etc.
background introduction
At the initial stage of introducing the Internet of Things business, each subsidiary was divided into two teams to develop their own business independently. The resource allocation was mainly to meet the current business needs. It was required to be able to quickly develop functions and go online quickly. The personnel investment was relatively large. Due to project development Earlier, the technology selection was independent of each other, and the system architecture was independently designed. Soybean Electronics mainly used Spring boot, Honeylink Technology mainly used Python Flask, and Golang was used for middleware message processing. With the increase in business overlap, the old system architecture There are the following disadvantages:
- Each subsidiary develops independently, and the business is directly deployed on ECS to run;
- The data layers are deployed independently of each other and used in separate ECS
- Associated business calls between subsidiaries interact through third-party cloud interfaces;
- Deployment requires manual packaging, online, no CI/CD;
- When adding a new ECS to provide services, the deployment operation is complicated;
- Resources cannot be dynamically allocated and utilized;
- Monitoring introduces Prometheus, and each part of the function is configured and realized by itself.
The old business structure is as follows:
Selection instructions
In order to solve the problems existing in the business in the current architecture, K8s + Docker is introduced to transform the existing container and expand new business at the same time.
When conducting K8s research and use, learn a lot of K8s related technologies, and build a complete set of K8s clusters for testing and comparison. The official management platform provided by K8s has complicated operation methods and complicated construction processes. In the process of researching K8s, Learn about the KubeSphere platform through network sharing.
After comparison, it was found that:
- KubeSphere is a distributed operating system for cloud-native applications built on top of Kubernetes, including all the functions that K8s can achieve;
- On the basis of K8s, KubeSphere advocates out-of-the-box use and built-in various configurable plug-ins to provide users with relatively optimal solutions;
- KubeSphere provides various monitoring functions such as multi-tenant management and monitoring alarms;
- Compared with K8s, the KubeSphere management interface is concise and clear, and the operation is convenient;
- KubeSphere provides Kubekey fast cluster construction, and only needs a few simple configuration modifications to complete many complex installation and deployment tasks such as K8s clusters and KubeSphere management pages;
- KubeSphere is a domestic open source project, providing rich sample documents, video tutorials, open source communities, etc., to find solutions more quickly when problems arise.
At present, my new business uses SpringCloud microservices to enter KubeSphere production clusters and KubeSphere test clusters to meet the development of our business, and use DevOps provided by GitLab+Harbor+KubeSphere to achieve CI/CD, rapid deployment, and efficient monitoring.
practice process
hardware resources
Alibaba Cloud ECS (8C 16G) 12 units. 1 Alibaba Cloud SLB.
Resource allocation
- Production cluster: 1 Alibaba Cloud SLB, 8 Alibaba Cloud ECS
- Test cluster: Alibaba Cloud ECS 3 sets
- Code and BUG management: 1 Alibaba Cloud ECS
deployment architecture
- Production environment: In order to meet the high availability requirements of K8s cluster services, it is necessary to ensure the HA of kube-apiserver. Alibaba Cloud SLB is used for high availability configuration. The specific deployment structure is shown in the figure;
- Test environment: use three ECSs to build a KubeSphere cluster, in which the Master is used as a worker node at the same time (not recommended);
- Code and defect tracking system: A single ECS is built with Docker, which is convenient for migration and maintenance, and provides Git Server and mantis services for all technical developers in the group;
- Data storage: The data storage layer uses the Ali NAS file system to facilitate data snapshot backup and capacity expansion;
- Internet file distribution: use Alibaba Cloud OSS + Alibaba Cloud CDN to distribute content.
System Architecture Diagram
The standard Spring Boot micro-service architecture is adopted. The business layer, middleware layer, data layer, and CI/CD are all deployed using KubeSphere, and the K8s standard storage class is used for data storage. The configuration data and encrypted data of the middleware and data layer are used K8s configuration dictionary and secret dictionary.
data storage class
The platform storage part uses OpenEBS provided during KubeSphere installation, and Ali NAS is used for business and middleware data, which is convenient for business data backup.
Cloud platform CI/CD practice
CI/CD flow chart:
Brief description of CI/CD:
- Developer submits code;
- Gitlab triggers a push event;
- GitLab calls the callback hook to trigger the Jenkins build task;
- Jenkins performs task execution according to the pipeline script in the build task, pulls code, Maven compiles, Docker builds, Docker pushes Harbor, executes deployment scripts, and notifies enterprises via WeChat.
CI/CD workflow:
API microservice deployment
Workload deployment
service deployment
confidential dictionary
middleware configuration
log system
- Part of it, the cluster operation log and the container operation log, are collected and stored using the ES provided by KubeSphere by default;
- Logstash + ElasticSearch is used to collect and place logs in the API business part, and kibana is used to read and view logs.
Q&A
1. Using the Aliyun Docker repository, CI/CD takes a long time.
Solution: Install Harbor from the app store, and change the warehouse from Aliyun Docker to Harbor.
2. Jekins restarts frequently.
Solution: Devops-jenkins in kubesphere-devops-system has opened the memory limit of 2G and changed it to 4G.
3. I don’t know how to trigger the combination of Devops and self-built Gitlab.
Solution: Enter Jenkins to trigger using a generic hook on the pipeline.
Effect
The current experience of using KubeSphere:
- It is very convenient to install the KubeSphere environment, which is basically out of the box;
- CI/CD greatly simplifies development and deployment costs;
- A variety of stable and efficient components built into KubeSphere ensure the stable operation of the cluster;
- KubeSphere can use the third-party Helm warehouse to facilitate the installation of Helm applications, and simplifies the original process of writing yml files when installing middleware.
future plan
- At present, the company's new business is running in the KubeSphere cluster, and it is planned to smoothly migrate the old business into the KubeSphere cluster in the future;
- At present, the microservice Spring Cloud uses sentinel for governance, and more new services may be developed with Golang, and the non-intrusive function of the gateway will be used at that time.
This article is published by OpenWrite, a multi-post platform for blogging !