Yunjuan Yunshu: Introduction to distributed cloud-native Kosmos

Kosmos isMobile Cloud a collection of open source distributed cloud-native federated cluster technologies, its name Kosmos: K stands for kubernetes, and Cosmos means the universe (Greek), implying the infinite expansion of Kubernetes.

What is Kosmos?

Kosmos integrates infrastructure resources at the bottom and provides high-level capabilities for platform products at the top, realizing unified management and orchestration, network connectivity, and multi-level scheduling of multi-cloud and multi-cluster. Kosmos is fully compatible with the k8s API. Users can use federated clusters like a single cluster, especially stateful applications such as databases. For example, MySQL-Operator can arrange the active and standby MySQL instances in a cross-domain cluster managed by kosmos without any modification. At present, Kosmos carries a large number of services in the mobile cloud, and it has experienced the test of long-term production.

What can Kosmos do?

Currently, kosmos mainly includes three major modules: multi-cluster network, multi-cluster management and orchestration, and multi-cluster scheduling. In addition, kosmos is also equipped with a kosmosctl tool, which can quickly deploy kosmos components, add clusters, test network connectivity, etc.

multi-cluster network

The goal of the Kosmos network is to open up the network between multiple k8s clusters. This module can be deployed independently. The Kosmos network enables Pod to access Pods and Services across clusters as if they were in the same cluster. Currently, this module mainly has the following functions:

Multi-mode support: For the added cluster, you can choose P2P or Gateway mode, where mode is more compatible and suitable for hybrid cloud and multi-cloud scenarios. P2P mode is suitable for underlay network interoperability, with shorter network paths and better performance. Gateway
Cross-cluster PodIP and ServiceIP mutual access: Based on Linux tunnel technology, L3 network interoperability of multiple Kubernetes clusters is realized, that is, users can perform operations within the federated cluster access. Pod-to-Pod, Pod-to-Service
Network segment conflicts, dual-stack support, etc.: The Kosmos network was designed with common problems in production, such as dual-stack support, cluster network segment conflicts, etc., in order to facilitate Implemented in complex production environments. Network segment conflict: The Kosmos network allows two or more clusters to use the same network segment in a federated cluster, making it easier for users to manage existing clusters. Pod/Service

The Kosmos multi-cluster network module currently contains the following key components:

Controller-Manager: Used to collect network information of the cluster where it is located and monitor changes in network settings;
Network-manager: Used to calculate the network configuration required by each node;
Agent: It is a Daemonset used to configure the host network, such as tunnel creation, routing, NAT, etc.;
Multi-Cluster-Coredns: Implement multi-cluster service discovery;
Elector: Responsible for gateway node election;

Multi-cluster management and orchestration

The Kosmos multi-cluster management and orchestration module implements Kubernetes tree expansion and cross-cluster orchestration of applications. Currently, it mainly supports the following functions:

Fully compatible with k8s api: Users can use kubectl, client-go and other tools with the host cluster as usual ofkube-apiserverinteraction, andPodactually distributed across the entire multi-cloud multi-cluster.
Stateful applications, k8s-native application support: In addition to stateless applications, Kosmos also supports stateful applications and k8s-native (with kube-apiserverOrchestration of interactive) applications. Kosmos will automatically detectPod dependent storage and permission resources, such as pv/pvc, sa, etc., and automatically perform two-way synchronization.
Diverse Pod topology distribution constraints: Users can easily control the distribution of Pods in the federated cluster, such as: Region, Availability Zone, cluster or node , helping to achieve high availability and improve resource utilization.

Multi-cluster scheduling (under construction)

The Kosmos scheduling module is developed based on the extension of the Kubernetes scheduling framework and is designed to meet the container management needs in mixed node and sub-cluster environments. This scheduler has been carefully designed and customized to provide the following core functions to enhance the flexibility and efficiency of container management:

Flexible node and cluster hybrid scheduling: The Kosmos scheduling module allows users to easily and intelligently schedule workloads between real nodes and sub-clusters based on customized configurations. This enables users to fully utilize the resources of different nodes to ensure optimal performance and availability of workloads. Based on this function, Kosmos can enable flexible cross-cloud and cross-cluster deployment of workloads.
Refined container distribution strategy: By introducing custom resource definitions (CRD), users can precisely control the topological distribution of workloads. The configuration of CRD allows users to explicitly specify the number of workload pods in different clusters and adjust the distribution ratio according to needs.
Fine-grained fragmentation resource defragmentation: The Kosmos scheduling module can intelligently sense fragmented resources in sub-clusters, effectively avoiding insufficient sub-cluster resources when pods are deployed after being scheduled. This helps ensure that workloads are more evenly distributed across different nodes, improving system stability and performance.

Whether building a hybrid cloud environment or requiring flexible deployment of workloads in different clusters, the Kosmos scheduling module can be used as a reliable solution to help users manage containerized applications more efficiently.

Kosmos carrying capacity analysis

KOSMOS multi-cluster management can stably support 100 large-scale clusters and manage more than 500,000 nodes and 500,000 Pods, which can meet the needs of users in large-scale production.

APIResponsivenessPrometheus analysis results:

The value of Kosmos open source

If a project wants to flourish, our power is limited. We need more like-minded developers to join us and brainstorm to make kosmos develop better. At the same time, we also hope that more developers will point out the areas that we urgently need to improve.

KOSMOS-committed to creating a one-stop resource federation cluster system solution. We believe that open source can guide the development of Kosmos in the right direction and make Kosmos better. We look forward to discussing implementation ideas and technical details with you, and welcome your issues and PRs.