Author: Mao Wei, currently the system architect of the New Media Center of Wuxi Radio and Television Group, has been responsible for the design and construction of many provincial, municipal, district and county financial media platforms across the country, and has rich experience in system architecture design for new media industry construction. Now he is mainly engaged in the construction of Wuxi Bobao series of new media platforms, promoting the transformation of various business product lines to cloud native, and carrying out related evangelism work in this field.
Introduction
Established in 1999, Wuxi Radio and Television Group is the first radio and television group in the country. At the end of 2007, Wuxi Radio and Television Station was established (two brands and one team with Wuxi Radio and Television Group). As a mainstream media and a municipal cultural state-owned enterprise, the Group undertakes the dual functions of publicity and operation: on the one hand, it provides public opinion services for the central work of the Municipal Party Committee and the Municipal Government and the overall reform, development and stability of the city; Contribute to the development of the city's cultural industry. The group currently has 6 radio frequencies, 7 TV channels (including 1 bus and subway mobile TV channel), and a new media matrix led by "Wuxi Bo Bao".
background introduction
As one of the earliest urban radio and television media in China to put forward the concept of media integration development, put it into practice and continue to innovate, Wuxi Radio and Television began to deploy mobile terminal strategies and innovate continuously as early as 2012, and will upgrade and iterate its two major clients in 2021 The brand-new "Wuxi Bobao" client terminal has formed a series of "Bobao" WeChat public accounts, Weibo and video accounts, forming a new media matrix with stronger communication and influence. Recently, it won honors such as "National Radio and Television Media Integration Pilot Unit" and "New Era, New Brand and New Influence" by the State Administration of Radio, Film and Television in 2022.
In long-term practice, Wuxi Radio and Television has gradually explored ideas and experiences for the development of media integration suitable for urban radio and television. Guided by the construction of communication capabilities, we will actively build a new communication system externally, adhere to the "mobile first" strategy, expand and strengthen the mobile terminal platform, and occupy emerging communication positions. Guided by network-wide communication and local operation needs, we will continue to promote the reengineering and optimization of internal organizational structures, institutional mechanisms, business processes, and technology platforms. Promote the "main force into the main battlefield", transfer the team and production capacity of traditional radio and television to the new media end, and create a main public opinion position with urban media characteristics.
This requires Wuxi Radio and Television to quickly adapt to changing operational and market needs, and provide strong support for various geometrically growing businesses with efficient and agile application deployment and operation and maintenance.
Before container transformation, Wuxi Radio and Television mainly adopted an infrastructure environment based on virtualization technology. Each business application was deployed on an independent virtual machine according to its own needs. Over time, the scale of virtual machines became larger and larger. complex. Inadequacies in the structure have become increasingly prominent, as follows:
- When a certain scale is reached, the computing resources and storage resources that the virtual machine operating system itself needs to occupy are relatively wasteful.
- The old operating systems accumulated over a long period of time need to be maintained and upgraded. For example, a large number of existing CentOS systems need to be replaced by new release versions after the official stop of maintenance.
- Each application requires independent maintenance and management, making deployment and operation and maintenance costs increasingly high.
- The ability of elastic scaling is poor, and the deployment time is long.
- There is a lack of a test environment, and the development environment and production environment are not unified, and application updates rely on manual work.
- Lack of monitoring of business and resource utilization makes it impossible to discover potential problems in advance.
These problems lead to relatively low operation and maintenance efficiency, which cannot meet the needs of rapid business iteration. Therefore, the new media operation and maintenance team of Wuxi Radio and Television decided to carry out containerized transformation to improve the elasticity, flexibility and maintainability of the system, and realize the following functions:
- More efficient resource utilization: Containerization technology can share the operating system kernel, thereby reducing the computing resources and storage resources required by each application.
- Better maintainability: By using container orchestration tools, you can better manage and maintain containers, improve deployment and operation and maintenance efficiency, and reduce costs.
- Higher elasticity: Containerization technology can achieve rapid deployment and startup, and rapid scaling, so as to better meet the changing needs of the business.
- Higher consistency: Containerization technology can ensure the consistency of the development environment, test environment, and production environment, thereby reducing the risk of application updates.
- Better observability: Through distributed tracing, visualized traffic topology and monitoring, monitoring and alarming from nodes to containers to business resources can be realized, and problems can be discovered, located and solved in a timely manner.
- Better application lifecycle management: By integrating technologies such as application stores and DevOps systems and microservice governance, application release management can be made more agile, elastic, and scalable.
For this reason, embracing cloud native has become a trend in the entire industry, which can help reduce costs, improve efficiency, and enhance competitiveness.
Selection planning
Through the initial use of containerization and the accumulation of Kubernetes in the early stage, before deciding to fully transform into containerization, we have established some requirements in combination with our own requirements for the future management platform planning of Kubernetes:
- Able to manage multiple Kubernetes clusters, we will split multiple clusters according to business adaptation and install them on existing clusters.
- It can be integrated from the deployment, upgrade maintenance, and management of Kubernetes clusters, covering the life cycle management of clusters and applications.
- There is an API interface to facilitate the docking with its own CI/CD tools.
- Compatibility with non-CentOS systems (de-CentOSization is being promoted during model selection).
- It is convenient for future cluster upgrades, and can perfectly adapt to the containerd container runtime in cluster deployment.
- The deployed cluster is close to the original installation, so that the tool can maintain the cluster by itself later.
- There is a domestic installation mirror, which supports pure offline deployment.
During the selection period, we happened to be planning to deploy a self-developed business cluster. We found the KubeSphere and Kubekey solutions in the CNCF-certified Kubernetes deployment tool, and conducted in-depth tests on cluster deployment and life cycle management, mainly focusing on the following some dimensions:
Through testing, it is found that KubeSphere+Kubekey is more in line with the original needs of the management platform in all aspects. Therefore, KubeSphere+Kubekey is used to build a set of Kubernetes clusters and management platforms for self-developed business (mainly operations).
practice process
deployment architecture
The infrastructure is based on its own computer room virtualization cluster, and uses virtual machines to build Kubernetes clusters. In terms of cluster planning, it is divided into two production clusters, which are used for content production business and operation business respectively. For the content production business cluster, more emphasis is placed on stability, so version 1.21 is adopted. As for the operation business cluster, on the basis of pursuing relative stability, some new version features have been followed up, and version 1.23 has been adopted. At the same time, in the operation business cluster, some new version features will be practiced first to accumulate experience, so as to lay a solid foundation for upgrading the content production business cluster in the future. Of course, no matter which cluster it is, before each corresponding upgrade and maintenance, a temporary test cluster will be created for testing and verification of related operations.
cluster resources
Here we take the K8s cluster of version 1.23 as an example to introduce the deployment environment. Resource list:
| node name | configuration | Role | system |
|---|---|---|---|
| k8s-master1 | 4C 8G 200G | Control plane nodes, etcd | Debian 11 |
| k8s-master | 4C 8G 200G | Control plane nodes, etcd | Debian 11 |
| k8s-master3 | 4C 8G 200G | Control plane nodes, etcd | Debian 11 |
| k8s-node1 | 16C 32G 200G | work node | Debian 11 |
| k8s-node2 | 16C 32G 200G | work node | Debian 11 |
| k8s-node3 | 16C 32G 200G | work node | Debian 11 |
| k8s-node4 | 16C 32G 200G | work node | Debian 11 |
| k8s-node5 | 16C 32G 200G | work node | Debian 11 |
| k8s-harbor | 4C 8G 500G | mirror warehouse | Debian 11 |
| gitlab | 8C 16G 200G | code repository | Debian 11 |
| gitrunner | 8C 8G 200G | CI/CD | Debian 11 |
| k8s-lb1 | 4C 8G 100G | load balancing, DNS | Debian 11 |
| k8s-lb2 | 4C 8G 100G | load balancing, DNS | Debian 11 |
| proxy1 | 8C 16G 100G | Business Access Reverse Proxy | Debian 11 |
| proxy2 | 8C 16G 100G | Business Access Reverse Proxy | Debian 11 |
All servers are provided by means of virtual machines of the local virtualization platform.
External business exposure
In order to realize the external business access of the K8s cluster, two OpenResty servers are used, and the traffic is distributed to the Ingress NodePort port of each working node in the K8s cluster through the reverse proxy mode. In order to ensure high availability, the OpenResty server is deployed in active-active mode. At the same time, OpenResty is used to realize the hot update, current limiting and security protection functions of the configuration. In addition, the global SSL certificate management is unified on OpenResty to simplify the management complexity caused by the decentralized deployment of SSL certificates in the K8s cluster. Through these measures, the external business access of the Kubernetes cluster can be managed more efficiently.
In order to achieve high availability of K8s cluster management, a high-availability load balancing service is deployed using Keepalived and HAProxy to realize the unified external exposure of the three master node API servers at the back end. In addition, a set of dnsmasq is also set up to provide DNS resolution services for each node, so as to resolve the domain names of some internal services. In this way, it can be ensured that the API server of the Kubernetes cluster can continue to provide services, and the domain names of internal services can be correctly resolved.
storage implementation
According to business needs, more traditional virtual machine services need to be migrated to a containerized environment, so an in-depth understanding of the storage solution of the K8s cluster has been made. The goal is to fully utilize the existing hardware base while simplifying the architecture and reducing operation and maintenance costs as much as possible. Therefore, in terms of underlying storage, existing professional hardware NAS storage and vSphere-based Cloud Native Storage (CNS) are used to deal with different data persistence scenarios.
In order to solve the storage problem of multiple Deployments reading and writing applications at the same time, the storage class storage class based on nfs-subdir-external-provisioner is adopted, or the form of directly mounting nfs volumes in the Pod. However, we also realize that NFS storage may not be compatible and have performance issues in some application scenarios. Therefore, for data persistence scenarios that only require the ReadWriteOnce access type and have high performance requirements, such as databases and caches, the storageclass storage class is implemented by using the vSphere CNS that comes with the virtualization environment. This greatly simplifies the complexity of storage solutions.
Observability scheme
As a radio and television publicity application, it has high requirements for the stability of the entire platform, and pays more attention to observability in daily operation and maintenance. Initially, Prometheus-operator suite and Grafana were used for cluster resource monitoring, and Netdata was used for cooperation. For application logs, Loki, Promtail and Grafana are used for processing. However, in the application, it is found that this solution is not well integrated in the application management of the cluster, and there are some separations in use. After experiencing the overall monitoring and logging solution provided by KubeSphere, I decisively decided to switch to KubeSphere. This solves the previous separation problem between various systems, and realizes the integration of cluster + application management, monitoring and logging.
Devops solution
In terms of Devops, the GitLab CI/CD solution is adopted. R&D only needs to submit the code and tag it, and GitLab will automatically generate the corresponding jobs. Then, run the corresponding script through GitLab Runner to realize packaging, image push and other operations, and trigger the API to modify the image tag of the online application through a specific tag name, so as to realize automatic deployment.
apply effects
Compared with the previous Kubernetes cluster management method, after using KubeSphere, we have achieved:
- Ease of deploying and upgrading Kubernetes clusters
After applying KubeSphere, it is no longer necessary to manually install and configure Kubernetes clusters, because KubeSphere provides the KubeKey tool to implement one-click deployment and upgrade functions, which makes it possible to quickly create and manage clusters. In addition, KubeSphere also provides application management based on Helm and Operator, making it easier to deploy and manage applications.
- Unified management of multiple Kubernetes clusters
In actual application business, it is necessary to manage multiple Kubernetes clusters at the same time. After KubeSphere is applied, multiple Kubernetes clusters can be managed in a unified manner, making it easier to operate and monitor. In addition, KubeSphere also provides inter-cluster application image replication and scheduling, making it possible to flexibly deploy applications among multiple clusters.
- Realize enterprise space access control management and resource allocation in the form of tenants
In the business, access control management and resource allocation are required for different users and teams. After applying KubeSphere, tenants can be created to realize access control and resource allocation to enterprise space, so as to manage business more flexibly.
- Unification of logs and monitoring platforms for clusters and applications
Previously, different log and monitoring tool backgrounds were required to manage clusters and applications. After applying KubeSphere, we can use the unified log and monitoring platform provided by KubeSphere to manage clusters and applications, making it easier to view and analyze data.
- Simplified barriers to entry for application governance
Application governance is a very important part of our business. After applying KubeSphere, you can use the application governance components provided by KubeSphere, such as grayscale publishing and traffic management, to manage applications more conveniently. In this way, the cost of using application governance can be reduced and efficiency can be improved.
future plan
Combined with the actual application of radio and television, in terms of KubeSphere application, there are the following later plans:
- Try to use KubeSphere grayscale release in the test.
- Explore using KubeSphere's DevOps components to combine grayscale releases with CI/CD pipelines.
- Utilize the application governance component to realize the observability of business APM.
In the current process of using KubeSphere, I found that compared with other products, KubeSphere focuses more on application scenarios and simplified management background routes, which is more friendly to newcomers. However, some basic personnel may feel at a loss about the correspondence between some background configuration items and Kubernetes resources at the initial stage of contact, and the operation is cumbersome. In addition, it is found that the configuration of some Kubernetes resources cannot be fully controlled in the background at present, and needs to be realized by manually editing YAML. So hopefully these issues will be improved in future releases.
This article is published by OpenWrite, a multi-post platform for blogging !