Introduction to YARN of Hadoop2

1. Introduction

Hadoop 2.x is mainly composed of HDFS, YARN and MapReduce. Its architecture is shown in Figure 1.1.

Figure 1.1 Hadoop three components architecture diagram

YARN is an independent distributed resource management and scheduling system that can provide unified resource management and scheduling for upper-layer applications.
Among them, ResourceManager is responsible for the monitoring, allocation and management of all resources; ApplicationMaster is responsible for each specific job manager; NodeManager is a node manager for each machine, and Container is a resource (node, memory, CPU) abstracted by RM The packaging, the relationship diagram is shown in Figure 1.2.

Figure 1.2 Relationship diagram of each component

Second, ResourceManager

ResourceManager (RM) replaces the cluster manager, mainly composed of ApplicationManager and Scheduler ,

1. ApplicationManager
   ApplicationManager is mainly responsible for:
   (1), receiving job submissions ( Job-submissions );
   (2), negotiating the first container to execute ApplicationMaster Application ;
   (3), providing restart service for the failed ApplicationMaster Container .
   Attachment: ApplicationManager mainly maintains the collection of submitted applications. After the task is submitted, he first checks the resources required by the application and rejects applications that he does not have sufficient resources to run. In addition, ApplicationManager also maintains a first-in first-out The cache is used to store completed applications. When the cache is full and there are new tasks to be stored in the cache, the oldest completed application will be moved out.
2. Scheduler
   Scheduler is mainly responsible for allocating appropriate resources (familiar constraints of capacities, queues etc) to each running application (Application) , is not responsible for application monitoring and status tracking, and does not guarantee the restart of failed tasks
   
   . Figure 2.1 Inside the ResourceManager Structure diagram (from StackOverflow)

Three, NodeManager

NodeManager (NM) replaces TaskTracker in Hadoop 1.x. NM belongs to the Slave process . Similar to TaskTracker, it is equivalent to the packaging of the machine frame . Its main work is as follows:

1. NM handles task requests assigned by RM ;
2. NM receives and processes various requests such as Container start and stop from ApplicationMaster ;
3. The Container (the container that executes the application) that is responsible for starting the application , monitors the resource usage , and reports to the RM through the heartbeat; in
   general, NM is a single node for resource management and task management.

四、ApplicationMaster

ApplicaitonMaster is the master of the application, and each application corresponds to an AM ( each job has an AM , which runs on a machine other than RM as an ordinary Container). When a user submits an application, an AM is lightweight. A process instance of the type will start, and AM coordinates the execution of all tasks within the application.

1. AM is responsible for all work within a Job life cycle, similar to the old JobTracker ;
2. Negotiate resources with RM and negotiate appropriate container with scheduler;
3. Work collaboratively with NM, negotiate the appropriate Container with the Scheduler, and monitor the Container.

五、Application Manager

The application manager is responsible for managing all applications in the entire system , including application submission , coordinating resources with the scheduler to start the AM , monitoring the AM operating status and restarting the AM when it fails, and tracking the progress and status of the assigned Container .

Six, Container

Container is an abstract package of task running environment, which mainly describes the running resources (nodes, memory and CPU), start commands and running environment of the task. In essence, it is only the right to use the specified resources on NM. To start the Container.

Seven, YARN operation process

1. step1——clients submit a task (Job-submissions) to Resource Manager;
2. step2——The Application Manager in Resource Manager processes the client ’s task submission request, and then Resource Manager negotiates with the Scheduler for the first container resource to run the Application Master instance;
3. step3——The Application Master requests more Container resources from the Scheduler in the Resource Manager according to actual needs;
4. step4——The Application Master performs distributed computing through the obtained Container resources.

8. Yarn's fault tolerance

1. Resource Manager
   RM can implement HA high-availability cluster through Zookeeper. The master node provides services, and the slave nodes synchronize the master information in real time. When the master hangs, the slave node will take over the role of the master node.
2. 
   A heartbeat check is performed between NodeManager NM and RM. Once it hangs, RM will immediately notify AM, and AM will perform related processing.
3. ApplicationMaster is
   generally restored by ResourceManager, and it will be restored to the state of the failed task at the same time.

Nine, Yarn scheduler

1. FIFO Scheduler
   processes in the order submitted. Advantages: the simplest, disadvantages: large applications occupy all cluster resources and are not suitable for shared collections.
   
   Figure 9.1 FIFO scheduler

2. Capacity Scheduler
   has a dedicated queue to run small tasks, which will occupy certain cluster resources in advance. Disadvantages: the execution time of large tasks lags behind FIFO.
   
   Figure 9.2 Capacity scheduler

3. The
   resources occupied by Fair Scheduler users are shared equally, and the resources occupied by users submitted jobs share the resources obtained by users. Benefits: no need for exclusive use, dynamic adjustment, and fair sharing.
   
   Figure 9.3 fairs scheduler

10. Conclusion

Relevant knowledge points and pictures come from the Internet, if there is any infringement, please inform by private letter.
In addition, please correct me!