[Big data day16]——yarn [yarn architecture and workflow, yarn scheduler, common parameter settings for yarn]

yarn resource scheduling

1. Introduction of yarn :

​ Yarn is the resource management system module in the Hadoop cluster. The yarn module has been introduced from Hadoop 2.0. Yarn can provide resource management and scheduling for various computing frameworks. It is mainly used to manage resources in the cluster (mainly various servers). Hardware resources, including CPU, memory, disk, network IO, etc.) and scheduling various tasks running on yarn.

 yarn核心出发点是为了分离资源管理与作业监控，实现分离的做法是拥有一个全局的资源管理（ResourceManager，RM），以及每个应用程序对应一个的应用管理器（ApplicationMaster，AM） 

​ To sum up, one sentence is to say: Yarn is mainly for scheduling resources, managing tasks, etc.

The scheduling is divided into two levels:

• First-level dispatch management:

  ​ Computing resource management (CPU, memory, network IO, disk)

• Secondary scheduling management:

  ​ Internal calculation model management of tasks (AppMaster's refined task management)

Yarn's official website document description:

http://hadoop.apache.org/docs/r2.7.5/hadoop-yarn/hadoop-yarn-site/YARN.html

Monitoring and management interface of yarn cluster:

http://node01:8088/cluster

jobHistoryServer view interface:

http://node01:19888/jobhistory

2. Introduction and function of Yarn's main components

YARN总体上是Master/Slave结构, Mainly composed of several components such as ResourceManager, NodeManager, ApplicationMaster and Container.

• ResourceManager (RM)
  is responsible for processing client requests, and unified management and scheduling of resources on each NM. Assign idle Container to ApplicationMaster to run and monitor its running status. It is mainly composed of two components: scheduler and application manager:
  1. Scheduler (Scheduler) : The scheduler allocates the resources in the system to each running application according to constraints such as capacity and queues. The scheduler only allocates resources according to the resource requirements of each application, and the resource allocation unit is Container. Shceduler is not responsible for monitoring or tracking the status of the application. In short, the scheduler allocates the resources encapsulated in the Container for the application according to the resource requirements of the application and the resource situation of the cluster machine.
  2. Applications Manager : The application manager is responsible for managing all applications in the entire system, including application submission, resource negotiation with the scheduler to start the ApplicationMaster, monitoring the running status of the ApplicationMaster and restarting when it fails, etc., tracking points The progress and status of the given Container is also its responsibility.
• NodeManager (NM)
  NodeManager is the resource and task manager on each node. It will periodically report the resource usage on the node and the running status of each Container to the ResourceManager; at the same time, it will receive and process Container start/stop requests from the ApplicationMaster.
• ApplicationMaster (AM) : All
  applications submitted by users include an ApplicationMaster, which is responsible for application monitoring, tracking application execution status, restarting failed tasks, etc. ApplicationMaster is an application framework, which is responsible for coordinating resources with ResourceManager and working with NodeManager to complete Task execution and monitoring.
• Container :
  Container is the resource abstraction in YARN. It encapsulates multi-dimensional resources on a node, such as memory, CPU, disk, network, etc. When ApplicationMaster applies for resources from ResourceManager, ResourceManager returns the resources for ApplicationMaster to use Container Expressed.

3. Yarn architecture and workflow

4.yarn scheduler

We all know that yarn is mainly used for resource scheduling, task allocation and other functions. So in Hadoop, what algorithm is used for task scheduling needs our attention. Hadoop supports several task scheduling methods and different scenarios. Need to use a different task scheduler.

第一种调度器：FIFO Scheduler（队列调度）

Arrange the tasks into a queue in the order of submission. This is a first-in-first-out queue. When resource allocation is performed, resources are allocated to the top task in the queue first, and then the top task is satisfied before the next task. One allocation, and so on.

FIFO Scheduler is the simplest and easiest to understand scheduler, and does not require any configuration, but it is not suitable for shared clusters. Large tasks may occupy all cluster resources, which causes other tasks to be blocked.

第二种调度器：Capacity Scheduler（容量调度器，apache版本默认使用的调度器）

The Capacity scheduler allows multiple organizations to share the entire cluster, and each organization can obtain a portion of the cluster's computing power. By assigning a dedicated queue to each organization, and then assigning certain cluster resources to each queue, the entire cluster can provide services to multiple organizations by setting up multiple queues. In addition, the queue can be divided vertically, so that multiple members within an organization can share the queue resources. Within a queue, resource scheduling uses a first-in-first-out (FIFO) strategy.

第三种调度器：Fair Scheduler（公平调度器，CDH版本的hadoop默认使用的调度器）

The design goal of the Fair scheduler is to allocate fair resources for all applications (the definition of fairness can be set by parameters). Fair scheduling can also work among multiple queues. For example, suppose there are two users A and B, and they each have a queue. When A starts a job and B has no tasks, A will get all the cluster resources; when B starts a job, A's job will continue to run, but after a while, the two tasks will each get half of the cluster resources. If B starts a second job at this time and other jobs are still running, it will share the resources of the B queue with B’s first job, that is, B’s two jobs will be used for a quarter Cluster resources, and A's job is still used for half of the resources of the cluster, the result is that the resources are ultimately shared equally between the two users

Which scheduler to use depends on the yarn-site.xml

yarn.resourcemanager.scheduler.class Configuration of this property

5. About yarn common parameter settings

Set container to allocate minimum memory

yarn.scheduler.minimum-allocation-mb 1024 The minimum memory allocated to the application container

Set the maximum memory allocated by the container

yarn.scheduler.maximum-allocation-mb 8192 The maximum memory allocated to the application container

Set the minimum number of virtual cores for each container

yarn.scheduler.minimum-allocation-vcores 1 The minimum number of virtual cores allocated by default for each container

Set the maximum number of virtual cores for each container

yarn.scheduler.maximum-allocation-vcores 32 The maximum number of virtual cores that each container can allocate

Set the memory size that NodeManager can allocate

yarn.nodemanager.resource.memory-mb 8192 The maximum memory size that nodemanager can allocate, the default is 8192Mb

Define the memory usage size of each machine

yarn.nodemanager.resource.memory-mb 8192

Define the size of the swap space that can be used

Swap space refers to a hard disk used for memory usage. The specified here is 2.1 times that of nodemanager.

yarn.nodemanager.vmem-pmem-ratio 2.1