Author: Zen and the Art of Computer Programming
1 Introduction
"Containerization" or "microserviceization" is becoming more and more popular, but the distributed data platform based on container clusters is still one of the most hated problems in the cloud native era. Enterprises want to quickly develop, test, and deploy applications on their data platforms, and ensure high availability and scalability of applications. The traditional way of managing data platforms mainly relies on manual operations, but the efficiency is low, the operation is complicated, it is difficult to track changes, and it does not conform to the concept of cloud native.
Today, there are many open source tools that can help enterprises automate data management, including open source declarative orchestration tools such as ArgoCD and Terraform, and big data analysis engines such as Kubeflow. These tools can provide the basic capabilities of the management platform, but how to combine actual needs to realize a more flexible management system has become an important and complicated issue.
This article shares how to effectively manage modern data platforms by introducing several existing Kubernetes data management tools and how to use these tools to build a data platform. The article will elaborate on the following aspects:
- Data platform architecture and functional division;
- Use Argo CD to manage the workflow of the data platform;
- Use Terraform to manage resource configuration;
- What aspects need to be considered to realize a complete enterprise-level data platform;
- Selection and comparison of more data management tools.
I hope readers can get inspiration from it and improve their cloud native technical level. In addition, this article will lead readers to realize an enterprise-level data platform through case practice.
2. Explanation of basic concepts and terms
2.1 Kubernetes
Kubernetes (K8s) is an open source platform for automatically deploying, scaling, and managing containerized applications. Its design goal is to make it easy to deploy containerized applications without having to care about the underlying infrastructure. It provides application