Anyone who has knowledge of Hadoop knows that Hadoop has experienced version number confusion and architecture adjustments for a long time. YARN is the resource management and task scheduling framework proposed by Hadoop 2.0 (or early 0.23.x). It solves many pain points in the era of Hadoop 1.0 (or 0.21.x, 0.22.x).
With the development, YARN is not only a resource scheduling framework for Hadoop, but also a general resource scheduling manager, which can manage various computing frameworks through YARN, such as Strom, Spark, etc.
The basic idea of YARN is to divide the functions of resource management and job scheduling/monitoring into independent daemon processes. They are a global ResourceManager (RM) and ApplicationMaster (AM) for each application. The application can be a directed acyclic graph (DAG) of a job job or a group of job jobs.
ResourceManager is responsible for the resource allocation of all applications in the system. NodeManager is responsible for container agent and resource monitoring (cpu, memory, disk, network) in each machine, and reports these conditions to ResourceManager or Scheduler.
The ApplicationMaster of each application is a framework-specific library that negotiates resources from ResourceManager and performs monitoring tasks together with NodeManager.
From the structural point of view, YARN is a master/slave architecture. One ResourceManager and multiple NodeManagers together form a data computing framework.
From the above structure diagram, the main components of YARN include ResourceManager, NodeManager, ApplicationMaster and Container.
1. ResourceManager（RM）
ResourceManager is responsible for resource management and allocation of the entire cluster, including processing client requests, starting and monitoring ApplicationMaster, monitoring NodeManager, and resource allocation and scheduling.
ResourceManager is composed of two main components: Scheduler and ApplicationsManager (ASM).
Scheduler: Scheduler allocates resources to running applications based on constraints such as capacity and queues (how many resources are allocated to each queue, how many jobs can be executed at most, etc.). The Scheduler is a pure scheduler and is not responsible for monitoring functions or tracking application status. In addition, if the task fails due to an application error or hardware failure, it does not guarantee to restart the task. These are all handed over to ApplicationMaster to complete.
The resource allocation unit is represented by an abstract concept "Container". The container is a dynamic resource allocation unit that encapsulates resources such as memory, cpu, disk, network, etc., so as to limit the amount of resources used by each task.
Scheduler is a pluggable component that can be redefined according to your needs. YARN provides a variety of schedulers that have been implemented, such as CapacityScheduler and FairScheduler .
ApplicationsManager: ApplicationsManager is responsible for receiving submitted jobs, negotiating with the first container to execute the ApplicationMaster corresponding to the application, and restarting the ApplicationMaster when the container fails. The ApplicationMaster corresponding to each application is responsible for negotiating the resource container from the Scheduler, tracking the status of the application, and monitoring the execution progress.
2. NodeManager（NM）
NodeManager is the resource and task manager on each node in the YARN cluster, responsible for the operation of the current node program, resource management and monitoring.
The NodeManager regularly sends information such as the resource usage of the node and the running status of the container to the ResourceManager. At the same time, NodeManager needs to execute the commands of ResourceManager and ApplicationMaster.
3. ApplicationMaster（AM）
Every application that YARN runs will have an ApplicationMaster. Responsible for coordinating resources from ResourceManager, and monitoring container and resource usage (including memory, CPU, etc.) through NodeManager.
4. YARN workflow
- The client submits applications to ResourceManager, including ApplicationMaster, commands to start ApplicationMaster, user programs, etc.;
- ResourceManager allocates the first Container for the application and communicates with the corresponding NodeManager, asking it to start the ApplicationMaster of the application in this Container;
- ApplicationMaster registers itself with ResourceManager, and keeps heartbeat with ResourceManager after successful startup;
- ApplicationMaster applies for resources from ResourceManager;
- After the resource application is successful, the ApplicationMaster will initialize it and then communicate with the NodeManager to request the NodeManager to start the Container. Then ApplicationMaster keeps heartbeat with NodeManager to monitor and manage tasks running on NodeManager;
- During the operation of the Container, report its own progress and status information to the ApplicationMaster so that the ApplicationMaster can grasp the task running status, so that it can be restarted if the task fails;
- After the application runs, the ApplicationMaster logs itself out of the ResourceManager and allows the Container to which it belongs to recycle.
Can_kao
1. Apache Hadoop YARN
2. INTRODUCING APACHE HADOOP YAR
Personal Homepage: http://www.howardliu.cn
Personal blog post: YARN architecture
CSDN homepage: http://blog.csdn.net/liuxinghao
CSDN blog post: YARN architecture