The K8s-based container cloud Paas platform should be a must for every company using k8s. Today, we try to focus on the application and use building blocks to complete the design of a minimal version of the container cloud Paas platform. Let's Go
1. Basic functions
Our expectation is to achieve a self-service service as much as possible, so we do not consider some operations such as approval, in this part, we need to complete the key process of application from packaging to online
1.1 Image Packaging
After researching and writing the code, it is necessary to deploy the code in the production environment, and the smallest unit of deployment is usually the Docker image. Then we need to implement a self-service image packaging service to realize the delivery from source code to docker image.
After the R&D submits the code to the GIt code repository, the code repository administrator can set a callback hook to notify our deployment pipeline, follow the deployment pipeline to build the target image according to the previously set steps, and publish the built image. to our mirror repository
Among the deployment pipelines, we can directly use the old Jenkins, or we can choose a cloud-native deployment tool like Tekton.
1.2 Basic Services
If only from the application itself, in addition to the basic operating environment and code, it usually depends on some basic services (regardless of application layer dependencies), such as mysql, redis, kafka and other basic services, but such services are usually possible If it is not in k8s (except opeartor), then we need a self-service integration method. Here we integrate through the service catalog. Users only need to apply for the corresponding basic service resources.
1.3 Log Monitoring
After the application is launched, how can we obtain the corresponding health status? Usually logs and monitoring are needed for assistance. We hope that a method can be used for self-service log collection and monitoring item collection. At this time, we need a way to integrate with monitoring and log systems, and also involve the respective For monitoring alarms, we use EFK to collect logs for logs, and use Prometheus for monitoring. In addition to monitoring the basic resources of the application, the business can also expose the corresponding business indicators, so that we can realize the indicator level of the business layer. monitor
1.4 Load Balancing
After the application goes online, it usually needs to provide access to the outside world. In k8s, because of the network, it is usually necessary to expose the internal service of the network through ingress, and then it is necessary to provide users with a component that can automatically associate the service with the load balancing
There are usually two choices for load balancing components: old-fashioned load balancing components (Nginx/Kong/Haproxy) and microservice service gateways (Traefik), etc. The core of the choice is usually whether we want to transform, for example, we want to do something on ingress The implementation of basic verification, fusing, and current limiting requires secondary development, and it is necessary to select the ingress of the appropriate technology stack.
For these Ingresses, we usually need some dedicated nodes, that is, only responsible for the operation of the ingress, that is, the Proxy node.
1.5 Deploying Updates
Most applications will iterate the code with the iteration of the product or bug fixes. In k8s, it is usually what we call mirror update. This process can be done automatically with the help of k8s deployment.
When the application is updated, it is usually necessary to perform a grayscale test, that is, only some users are allowed to access. If there is an abnormality, it will be rolled back immediately. If it is normal, the entire application cluster will be rolled and updated, and the main package of this method is implemented in k8s. It is implemented through Deployment. Here we mainly create a new Deployment according to the proportion of the user's grayscale, and continue to update the old Deployment if successful.
1.6 Application offline
For offline applications, we usually need to release the corresponding resources. Here, a gc module may be needed to clean up various resources of the application.
So far, based on some basic requirements of k8s and applications, we have completed the full life cycle management of cloud-native applications based on code packaging, online monitoring, deployment updates, observability (log monitoring), etc.
2. Function realization of Paas platform based on K8s
In this section, we mainly focus on the realization of some Paas platform-related functions based on the K8s plane, mainly including: multi-tenant management, elastic scaling, capacity planning, configuration management, shared storage, cluster management, application market and other functions
2.1 Multi-tenant isolation
Multi-tenancy is an important mechanism based on the paas platform. The essence of multi-tenancy is to achieve resource isolation. Resource isolation usually includes physical isolation and software isolation. The so-called physical isolation means isolation on physical entities (such as servers). Software isolation refers to the access isolation of resources through admission control. Considering that most companies usually do not physically isolate k8s, we can directly use the namespace in k8s to isolate software several times.
2.2 Elastic scaling
On-demand billing for elastic scaling is a typical feature of the Paas platform, while K8s comes with HPA (horizontal automatic scaling), and realizes VPA (vertical automatic scaling) and Cluster automatic scaling through autoscaler. With these controllers, we can easily Provide users with the function of elastic scaling (generally more horizontally)
2.3 Capacity Planning
The main goal of capacity planning is to achieve resource isolation by limiting the resource quota of each business line, and at the same time, through capacity calculation, resource billing can be performed, and resource procurement can be made for future capacity planning decisions, and enterprise cost accounting and control. Relying on the ResourceQuota of k8s to realize the quota function, and do cost accounting by monitoring data
2.4 Configuration Management
In the process of application development, some configuration information is usually used, such as basic log, cache, database and other configuration information. In the previous environment, it was either based on env files or based on the configuration center for management, while in k8s Chinese name Basic configuration management can be achieved through two resources, configMap and Secret, that is, the configuration data is separated from the image to realize automatic configuration loading according to the environment.
2.5 Shared Storage
Shared storage in k8s mainly relies on PV/PVC to achieve. This is partly due to the relatively large differences in infrastructure of each company. It usually needs to be adjusted according to the company's existing technical capabilities. The specific implementation depends on CSI related implementation, which will not be described here
2.6 Cluster Management
In the company's internal environment, it is sometimes necessary to consider disaster recovery and backup, so it needs to be deployed in multiple computer rooms. Then our PAAS platform also needs to have this multi-cluster management capability, which is also suitable for production, testing, etc. Multi-environment cluster management, cluster management is mainly to solve the problem of platform multi-environment deployment, and manage all clusters of the entire group through a set of platforms
2.7 Application Market
The application market mainly refers to some applications such as redis, etcd, kafka middleware, etc. In addition to the integration method of the service catalog mentioned earlier, we also allow users to create some basic services through opeartor, thereby promoting the containerization of infrastructure, This part usually needs to be fine-tuned according to the current environment and the open source operator to adapt to the company's internal environment
2.8 User Center
In many companies, there are usually some internal user center services, which can be integrated to perform user authentication and even some permission control of the container cloud platform, so as to avoid repeated wheel building.
2.9 Basic functions
In addition to the above business functions, we usually need to perform basic functions, such as operation audit, permission control, security management and other basic functions. So far, we have a basic k8s-based cloud native Paas platform available.
3. Container Paas Platform Summary
Through the above infrastructure, we can usually get an application-centric container PaaS platform based on K8s, and from the perspective of various functions, we only need very little development work to complete various resources based on k8s. For the construction of the entire Paas platform, from the next section, we will start to develop some key parts, and carry out the development of some k8s operators, Let's Go. .
kubernetes study notes address: https://www.yuque.com/baxiaoshi/tyado3
WeChat account: baxiaoshi2020
, pay attention to the announcement number to read more source code analysis articles,
more articles follow www.sreguide.com