All resource management systems need to solve the three basic problems of effective resource utilization, effective response to tasks, and flexible configuration of scheduling strategies. So in a distributed scenario, how do YARN and Kubernetes solve it? This article introduces.
— Apache YARN —
The full name of YARN is (Yet Another Resource Negotiator), which is a cluster-shared scheduling framework with good scalability and very high reliability of the scheduler itself. The architecture of YARN is shown in the figure below, in which ResourceManager controls the entire cluster and manages the allocation of basic computing resources by applications. It schedules individual resource parts (compute, memory, bandwidth, etc.) to the underlying NodeManager (YARN's per-node agent). The ResourceManager also works with the Application Master to allocate resources and with the NodeManager to start and monitor their underlying applications. In this context, the Application Master assumes some of the responsibilities of the previous TaskTracker, and the ResourceManager assumes the role of the JobTracker.
Application Master manages each instance of an application running in YARN, and is responsible for coordinating resources from ResourceManager, and monitoring container execution and resource usage (resource allocation of CPU, memory, etc.) through NodeManager. From the perspective of YARN, the Application Master is user code, so there are potential security issues. NodeManager manages each node in a YARN cluster. The NodeManager provides services for each node in the cluster, from overseeing the lifetime management of a container to monitoring resources and tracking node health. The NodeManager manages abstract containers that represent per-node resources available to a particular application. Container is an abstraction of resources in YARN. It encapsulates a certain amount of resources (memory, CPU) on a node. The operation of Container is initiated by the Application Master to the NodeManager where the resources are located.
The scheduling process of a MapReduce Job is shown in the figure below. It generally includes four steps: submitting the Job, starting the Application Master, requesting resource requirements, and processing data through the Container after passing the job. This process is also applicable to computing engines such as Spark and Flink. Through this resource management system of YARN, all short-term and medium-term computing tasks can be effectively managed and scheduled in a unified manner.
Scheduling capability is the core capability of YARN. The YARN community provides three scheduling models: FIFO, Fair, and Capacity. Users can also inherit the ResourceScheduler interface to implement a custom scheduler. As the name implies, FIFO Scheduler is the simplest scheduler. The submitted jobs are put into the corresponding position of the linear queue according to the order of submission time or according to the order of priority. When scheduling resources, they are scheduled according to the order of queues and first-in-first-out and resource allocation. This kind of scheduler is too simple. In actual production, there are not many applications. After all, jobs that need to be scheduled have different priorities.
In some multi-user scenarios, such as a large group running batch data processing tasks required by different applications through different users every night, the number of applications may be as many as dozens, and the fairness of cluster resource allocation among users is relatively important. In response to the needs of multi-tenancy, the community has launched the Capacity Scheduler, which allows different organizations to use their own resources without affecting each other, while improving the utilization and throughput of the entire cluster. Capacity Scheduler divides resources into multiple queues, and each queue allocates a part of resources. Applications of different organizations or users run in their respective queues, thereby achieving resource isolation. When a situation permits, in order to improve cluster throughput, resource preemption between queues is also allowed.
Fair Scheduler divides resources into multiple resource pools. Each resource pool sets the minimum guarantee and maximum limit for resource allocation. Administrators can also specify the priority of resource pools. Resource pools with higher priorities will be allocated more resources. Resources. When a resource pool has surplus resources, the remaining resources can be temporarily shared with other resource pools. Fair Scheduler first mounts the user's tasks to the leaf nodes of the tree queue as shown in the figure below, and waits for subsequent resource scheduling. After each scheduling cycle starts, the Scheduler selects a node in the cluster. Starting from the root node of the tree queue, each queue selects a sub-queue according to the priority of the job or according to the fairness strategy, and finally selects a sub-queue on the leaf node according to A fair strategy is used to select an app, and then allocate suitable resources for the app on the corresponding node to start computing tasks.
In order to better support production needs, Fair Scheduler also supports preemptive scheduling. If a resource pool fails to allocate a fair share of resources for a long time, the scheduler will kill tasks in the resource pool that allocates too many resources. Free up resources and allocate them to this resource pool for corresponding task scheduling. In addition, it also provides a load balancing mechanism based on the number of tasks, so as to distribute system tasks to each node as evenly as possible.
— Google Kubernetes —
Kubernetes is an open source project of Google, which is used to manage Docker clusters. It inherits the advantages of Borg and realizes the arrangement, deployment, operation and management of container applications. The figure below shows the overall architecture of Kubernetes. Kubernetes provides resource pool management, which can abstract resources such as CPU, GPU, memory, network, and hard disk in the entire cluster into a resource pool, and can flexibly schedule according to the real-time resource conditions in the resource pool according to the resource requirements of the application; Kubernetes Contains a unified scheduling framework that can manage up to thousands of servers and tens of thousands of containers, and provides a plug-in interface for third parties to customize and expand the new scheduling system; in addition, Kubernetes supports dynamic adjustments through ConfigMap and other methods Application configuration, so as to have the basic ability of dynamic deployment. Based on these basic technologies, we will develop a scheduling system that supports complex application platforms.
For a detailed introduction to Kubernetes, you can view previous articles: The Past and Present of Docker and Kubernetes (Part 2)
—Summary— _ _
This article introduces two distributed resource management technologies YARN and Kubernetes. The open source community began in 2018, and many projects such as Spark, Flink, and Tensorflow began to shift from YARN to Kubernetes-based management and scheduling. In the long run, as a resource management system for Hadoop clusters, YARN has effectively realized its technical value, but limited by its architecture design, it is difficult to evolve into a general data center scheduling system. Transwarp Technology has switched its internal big data platform from YARN to Kubernetes in 2017. The next article will introduce the Transwarp big data technology system in terms of storage, computing, and resource scheduling.