I. Introduction
Yarn plays the role of resource management and task scheduling in the Hadoop ecosystem. Before discussing its constructor, let's take a brief look at Yarn's architecture.
The above figure is the basic architecture of Yarn, in which ResourceManager is the core component of the entire architecture, which is responsible for the management of resources such as memory and CPU in the entire cluster; ApplicationMaster is responsible for the task scheduling of the application throughout the life cycle; NodeManager is responsible for the resources on this node. Supply and isolation; Container can be abstracted as a container for running tasks. The scheduler discussed in this article is scheduled in the ResourceManager. Next, let's study the three schedulers including the FIFO scheduler, the Capacity scheduler, and the Fair scheduler.
2. FIFO scheduler
The above figure is a schematic diagram of the execution process of the FIFO scheduler. The FIFO scheduler is also known as the First In First Out (First In First Out) scheduler. The FIFO scheduler is a scheduling strategy first applied by Hadoop. It can be simply understood as a Java queue. Its meaning is that only one job can be running in the cluster at the same time. All applications are executed in the order in which they were submitted. Only after the execution of the previous job is completed, the subsequent jobs will be executed in the order of the queue. The FIFO scheduler runs jobs exclusively on cluster resources. The advantage of this is that a job can make full use of all cluster resources, but for MR jobs with short running time, high importance, or interactive query types, they have to wait in the queue. The previous job can only be executed after the completion of the job, which also leads to the fact that if there is a very large job running, the subsequent job will be blocked. Therefore, although a single FIFO scheduling implementation is simple, it cannot meet the requirements for many practical scenarios. This also prompted the emergence of the Capacity scheduler and the Fair scheduler.
3. Capacity Scheduler
The figure above is a schematic diagram of the execution process of the Capacity scheduler. The Capacity scheduler is also known as the container scheduler. It can be understood as a resource queue one by one. This resource queue is allocated by the user himself. For example, because the work needs to divide the entire cluster into two queues, A and B, the A queue can continue to be divided, for example, the A queue is divided into two sub-queues, 1 and 2. Then the allocation of the queue can refer to the following tree structure:
---A[60%]
|---A.1[40%]
|---A.2[60%]
---B[40%]
The above tree structure can be understood as queue A occupies 60% of the entire resource, and queue B occupies 40% of the entire resource. Queue A is divided into two sub-queues, A.1 occupies 40%, and A.2 occupies 60%, which means that A.1 and A.2 occupy 40% and 60% of the resources of queue A at this time. Although the resources of the cluster have been specifically allocated at this time, it does not mean that A can only use 60% of the allocated resources after submitting the task, while 40% of the resources of the B queue are idle. As long as the resources in other queues are in an idle state, the queues with tasks submitted can use the resources allocated by the idle queues, and the amount used is determined by the allocation. The configuration of parameters will be mentioned later.
The Capacity scheduler has the following features:
1. Hierarchical queue design, which ensures that child queues can use all the resources set by the parent queue. In this way, it is easier to rationally allocate and limit the use of resources through hierarchical management.
2. Capacity guarantee, a resource ratio will be set on the queue, which can ensure that each queue will not occupy the resources of the entire cluster.
3. Security, and strict access control for each queue. Users can only submit tasks to their own queues, and cannot modify or access tasks in other queues.
4. Flexible allocation, idle resources can be allocated to any queue. When there is contention among multiple queues, they will be balanced proportionally.
5. Multi-tenant leasing, through the capacity limit of the queue, multiple users can share the same cluster, and colleagues ensure that each queue is allocated its own capacity to improve utilization.
6. Operational, Yarn supports dynamic modification and adjustment of the allocation of capacity, permissions, etc., which can be modified directly at runtime. It also provides an administrator interface to display the current queue status. Administrators can add a queue at runtime; but cannot delete a queue. The administrator can also suspend a queue at runtime, which ensures that the cluster will not receive other tasks during the execution of the current queue. If a queue is set to stopped, then no tasks can be submitted to it or its subqueues.
7. Resource-based scheduling, coordinating applications with different resource requirements, such as memory, CPU, disk, etc.
Configuration of related parameters:
(1) capacity: The resource capacity (percentage) of the queue. When the system is very busy, the capacity of each queue should be guaranteed, and if there are fewer applications per queue, the remaining resources can be shared with other queues. Note that the sum of the capacities of all queues should be less than 100.
(2) maximum-capacity: the upper limit (percentage) of the resource usage of the queue. Due to resource sharing, a queue may use more resources than its capacity, and the maximum used resources can be limited by this parameter. (This is also the maximum percentage of resources that can be occupied by a queue with tasks running as mentioned earlier)
(3) user-limit-factor: The maximum amount of resources (percentage) that each user can use. For example, if the value is 30, at any time, the amount of resources used by each user cannot exceed 30% of the queue capacity.
(4) maximum-applications: The upper limit of the number of applications that are waiting and running at the same time in the cluster or queue. This is a strong limit. Once the number of applications in the cluster exceeds the upper limit, subsequent submitted applications will be rejected. The default value is 10000. The maximum number of all queues can be set by the parameter yarn.scheduler.capacity.maximum-applications (can be regarded as the default value), and a single queue can be set by the parameter yarn.scheduler.capacity.<queue-path>.maximum-applications suits itself value of .
(5) maximum-am-resource-percent: The upper limit of the resource ratio used to run the application ApplicationMaster in the cluster. This parameter is usually used to limit the number of active applications. The parameter type is floating point, the default is 0.1, which means 10%. The upper limit of the ApplicationMaster resource ratio of all queues can be set by the parameter yarn.scheduler.capacity. maximum-am-resource-percent (which can be regarded as the default value), and a single queue can be set by the parameter yarn.scheduler.capacity.<queue-path>. maximum-am-resource-percent is set to a value that suits you.
(6) state: The queue state can be STOPPED or RUNNING. If a queue is in the STOPPED state, the user cannot submit applications to the queue or its sub-queues. Similarly, if the ROOT queue is in the STOPPED state, the user cannot Submit the application to the cluster, but the running application can still finish normally so that the queue can exit gracefully.
(7) acl_submit_applications: Limit which Linux users/user groups can submit applications to a given queue. It should be noted that this property is inherited, that is, if a user can submit applications to a queue, it can submit applications to all of its subqueues. When configuring this property, separate users or user groups with ",", and separate users and user groups with spaces, such as "user1, user2 group1, group2".
(8) acl_administer_queue: Specify an administrator for the queue, the administrator can control all applications of the queue, such as killing any application. Again, this property is inherited, if a user can submit applications to a queue, it can submit applications to all of its subqueues.
4. Fair scheduler
The above figure is a schematic diagram of the execution process of the Fair scheduler in a queue. Fair scheduler is also known as fair scheduler. The Fair scheduler is a queue resource allocation method. In the entire timeline, all jobs obtain resources on an average. By default, the Fair scheduler just schedules and allocates memory resources fairly. When only one task is running in the cluster, then this task consumes the resources of the entire cluster. When other tasks are submitted, those freed resources will be allocated to new jobs, so each task will eventually acquire almost the same amount of resources.
The fair scheduler can also work across multiple queues, as shown in the figure above, for example, there are two users A and B, each of which has a queue. When A starts a job and B does not submit tasks, A will get all the cluster resources; when B starts a job, A's tasks will continue to run, but queue A will slowly release some of its resources, and after a while, two tasks Each will get half of the cluster resources. If B starts the second job at this time and other tasks are still running, then it will share the resources of queue B with the first job in queue B, that is, the two jobs of queue B will use it respectively One-fourth of the cluster's resources, while the job in queue A will still use half of the cluster's resources, the result is that the cluster's resources are eventually shared equally between the two users.
Configuration of related parameters:
(1) yarn.scheduler.fair.allocation.file: The location of the "allocation" file, which is a configuration file used to describe queues and their properties. This file must be a strictly formatted xml file. If it is a relative path, it will look for this file on the classpath (under the conf directory). The default value is "fair-scheduler.xml".
(2) yarn.scheduler.fair.user-as-default-queue: Whether to use the username related to allocation as the default queue name, when the queue name is not specified. If set to false (and no queue name is specified) or not set, all jobs will share the "default" queue. The default value is true.
（3）yarn.scheduler.fair.preemption：是否使用“preemption”(优先权，抢占)，默认为fasle，在此版本中此功能为测试性的。
（4）yarn.scheduler.fair.assignmultiple：是在允许在一个心跳中，发送多个container分配信息。默认值为false。
（5）yarn.scheduler.fair.max.assign：如果assignmultuple为true，那么在一次心跳中，最多发送分配container的个数。默认为-1，无限制。
（6）yarn.scheduler.fair.locality.threshold.node：一个float值，在0~1之间，表示在等待获取满足node-local条件的containers时，最多放弃不满足node-local的container的机会次数，放弃的nodes个数为集群的大小的比例。默认值为-1.0表示不放弃任何调度的机会。
（7）yarn.scheduler.fair.locality.threashod.rack：同上，满足rack-local。
（8）yarn.scheduler.fair.sizebaseweight：是否根据application的大小(Job的个数)作为权重。默认为false，如果为true，那么复杂的application将获取更多的资源。
五、总结
如果业务逻辑比较简单或者刚接触Hadoop的时候建议使用FIFO调度器；如果需要控制部分应用的优先级同时又想要充分利用集群资源的情况下，建议使用Capacity调度器；如果想要多用户或者多队列公平的共享集群资源，那么就选用Fair调度器。希望大家能够根据业务所需选择合适的调度器。
如有转载请注明作者和出处！
推荐一款开源私有化部署的移动应用数据统计分析系统Cobub Razor
项目地址：https://github.com/cobub/razor