This article mainly introduces Yarn, another core component of Hadoop. Yarn is a resource scheduling platform responsible for providing server computing resources for computing programs. It is equivalent to a distributed operating system platform, while computing programs such as MapReduce are equivalent to running in operations. Applications on the system. Follow the column "Broken Cocoon and Become a Butterfly-Hadoop" to view related series of articles~
table of Contents
1. The basic architecture of Yarn
1.1.2 Application Manager (Applications Manager, ASM)
Third, the process of job submission to Yarn
1. The basic architecture of Yarn
YARN is mainly composed of components such as ResourceManager, NodeManager, ApplicationMaster, and Container. YARN is still a Master/Slave structure as a whole. In the entire resource management framework, ResourceManager is Master, NodeManager is Slave, and ResourceManager is responsible for unified management and scheduling of resources on each NodeManager. When a user submits an application, it needs to provide an ApplicationMaster to track and manage the application. It is responsible for applying for resources from the ResourceManager and requires NodeManger to start tasks that can occupy certain resources. Since different ApplicationMasters are distributed on different nodes, they will not affect each other. As shown below:
1.1 ResourceManager
RM is a global resource manager, responsible for the resource management and distribution of the entire system, and there is only one in the entire cluster. It is mainly composed of two components: Scheduler (Scheduler) and Application Manager (Applications Manager, ASM).
1.1.1 Scheduler (Scheduler)
The scheduler allocates the resources in the system to each running application according to constraints such as capacity and queues (for example, each queue allocates a certain amount of resources and executes a certain number of jobs at most). It should be noted that the scheduler is a "pure scheduler", it is no longer engaged in any work related to specific applications, such as not responsible for monitoring or tracking the execution status of the application, etc., nor is it responsible for restarting due to application execution failure Or failed tasks caused by hardware failures, these are all completed by the ApplicationMaster related to the application. The scheduler only allocates resources according to the resource requirements of each application, and the resource allocation unit is represented by an abstract concept "Resource Container" (Container). Container is a dynamic resource allocation unit that combines memory, CPU, and disk. , Network and other resources are encapsulated together to limit the amount of resources used by each task. In addition, the scheduler is a pluggable component. Users can design a new scheduler according to their needs. YARN provides a variety of directly usable schedulers, such as Fair Scheduler and Capacity Scheduler.
1.1.2 Application Manager (Applications Manager, ASM)
The application manager is responsible for managing all applications in the entire system, including application submission, negotiating resources with the scheduler to start the ApplicationMaster, monitoring the running status of the ApplicationMaster and restarting it when it fails, etc.
1.2 NodeManager
There are multiple NodeManagers in the entire cluster, responsible for single node resource management and use. The detailed functions are as follows: (1) Resource management and task management on a single node. (2) Processing commands from ResourceManager. (3) Processing commands from ApplicationMaster.
1.3 AplicationMaster
Each application submitted by the user contains an ApplicationMaster. The main functions include: (1) Negotiate with the ResourceManager scheduler to obtain resources (represented by Container); (2) Further allocate the obtained tasks to internal tasks; (3) ) Communicate with NodeManager to start/stop tasks; (4) Monitor the running status of all tasks, and reapply for resources for the task to restart the task when the task fails.
1.4 Container
Container is a resource abstraction in YARN. It encapsulates multi-dimensional resources on a node, such as memory, CPU, disk, network, etc. When ApplicationMaster applies for resources from ResourceManager, the resource returned by ResourceManager for ApplicationMaster is represented by Container . YARN will assign a Container to each task, and the task can only use the resources described in the Container. It should be noted that the Container is different from the slot in MRv1. It is a dynamic resource division unit, which is dynamically generated according to the needs of the application. Currently, YARN only supports two kinds of resources: CPU and memory, and uses the lightweight resource isolation mechanism Cgroups for resource isolation.
2. Yarn working mechanism
(1) The MR program is submitted to the node where the client is located. (2) YarnRunner applies for an Application from ResourceManager. (3) RM returns the resource path of the application to YarnRunner. (4) The program submits the resources required for operation to HDFS. (5) After the program resources are submitted, apply for running mrAppMaster. (6) RM initializes the user's request into a Task. (7) One of the NodeManagers receives the Task task. (8) The NodeManager creates a Container and generates MRAppmaster. (9) Container copies resources from HDFS to local. (10) MRAppmaster applies to RM for running MapTask resources. (11) RM assigns the task of running MapTask to the other two NodeManagers, and the other two NodeManagers receive tasks and create containers respectively. (12) MR sends the program startup script to the two NodeManagers that have received the task, and the two NodeManagers start MapTask respectively, and MapTask sorts the data partitions. (13) After MrAppMaster waits for all MapTasks to run, apply for a container to RM to run ReduceTask. (14) ReduceTask obtains the data of the corresponding partition from MapTask. (15) After the program runs, MR will apply to RM to cancel itself.
Third, the process of job submission to Yarn
1. Job submission
(1) The client calls the job.waitForCompletion method to submit a MapReduce job to the entire cluster.
(2) Client applies for a job id from RM.
(3) RM returns the submission path and job id of the job resource to the Client.
(4) The client submits the jar package, slice information and configuration files to the specified resource submission path.
(5) After the client submits the resources, it applies to the RM to run MrAppMaster.
2. Job initialization
(6) When RM receives the client's request, it adds the job to the capacity scheduler.
(7) An idle NM receives the job.
(8) The NM creates a Container and generates MRAppmaster.
(9) Download the resources submitted by the Client to the local.
3. Task allocation
(10) MrAppMaster applies to RM to run multiple MapTask task resources.
(11) RM assigns the task of running MapTask to the other two NodeManagers, and the other two NodeManagers receive tasks and create containers respectively.
4. Task running
(12) MR sends the program startup script to the two NodeManagers that have received the task, and the two NodeManagers start MapTask respectively, and MapTask sorts the data partitions.
(13) After MrAppMaster waits for all MapTasks to run, apply for a container to RM to run ReduceTask.
(14) ReduceTask obtains the data of the corresponding partition from MapTask.
(15) After the program runs, MR will apply to RM to cancel itself.
5. Progress and status update
The task in YARN returns its progress and status (including counter) to the application manager, and the client requests progress updates from the application manager every second (set by mapreduce.client.progressmonitor.pollinterval) and displays it to user.
6. Job completion
In addition to requesting the job progress from the application manager, the client will check whether the job is completed by calling waitForCompletion() every 5 seconds. The time interval can be set by mapreduce.client.completion.pollinterval. After the job is completed, the application manager and Container will clean up the working status. The job information will be stored by the job history server for later user verification.
Four, resource scheduler
There are three main types of Hadoop job schedulers: FIFO (first in first out scheduler), Capacity Scheduler (capacity scheduler) and Fair Scheduler (fair scheduler). The default resource scheduler of Hadoop 2.7.2 is Capacity Scheduler.
We can take a look at the yarn-default.xml file. This file exists in the source code. For details, please refer to "VII. Hadoop Source Code Compilation" .
4.1 FIFO
Sort by arrival time, first come first served.
4.2 Capacity Scheduler
1. Support multiple queues, each queue can be configured with a certain amount of resources, each queue adopts FIFO scheduling strategy. 2. In order to prevent jobs of the same user from monopolizing resources in the queue, the scheduler limits the amount of resources occupied by jobs submitted by the same user. 3. First, calculate the ratio between the number of tasks running in each queue and the computing resources that should be allocated, and select a queue with the smallest ratio. 4. Secondly, the tasks in the queue are sorted in accordance with the order of job priority and submission time, while taking into account user resource limitations and memory limitations. 5. Multiple queues are executed simultaneously in the order of tasks, and multiple queues run in parallel.
4.3 Fair Scheduler
Supports multiple queues and multiple users, the amount of resources in each queue can be configured, and jobs in the same queue share all resources in the queue fairly. For example, there are three queues. Jobs in each queue allocate resources according to priority. The higher the priority, the more resources are allocated, but each job will be allocated resources to ensure fairness. In the case of limited resources, there is a gap between the computing resources that each job obtains under ideal conditions and the computing resources actually obtained. This gap is called a gap. In the same queue, the greater the resource shortage of the job, the more resources are obtained first and executed first. The assignments are executed sequentially according to the vacancy.
This is the end of this article. If you have any problems during this process, please leave a message and let me see what problems you have encountered~