Yarn (Yet Another Resource Negotiator) is a resource scheduling platform, which is responsible for allocating resources and scheduling for computing programs such as Spark and MapReduce, and does not participate in the internal work of user programs. The same is the Master/Slave architecture.
The following figure MapReduce submitted to Yarn to run as an example, to see which core components Yarn mainly includes and the role of each component:
Global Resource Manager (ResourceManager)
The master node, the global resource manager, is responsible for resource management and allocation of the entire system, and is mainly composed of a scheduler and an application manager.
The scheduler allocates resources in the system to each running application based on constraints such as capacity and queues (such as how many resources are allocated to each queue, and a maximum number of jobs to execute, etc.). The Application Manager (ApplicationsManager) is responsible for managing all applications in the entire system, including application submission, negotiating resources with the scheduler to start, monitoring the Application Master, and restarting it if it fails, etc.
Node Manager (NodeManager)
The slave node, the resource and task manager on each node, needs to report the resource usage on the node and the running status of each Container to the ResourceManager, and at the same time receive and process various requests from the Application Master such as Container start/stop
Application Manager (Application Master)
Each application submitted by the user contains an application manager, which corresponds to running MapReduce as MRAppMaster. The main functions include:
1. Apply for resources to the global resource manager
2. Further allocate the obtained resources to internal tasks
3. Communicate with node explorer to start/stop tasks
4. Monitor the running status of all tasks, and re-apply resources for the task to restart the task when the task fails to run
Container
可以理解为Yarn中为某个节点已经申请到的资源封装的类，资源如内存、CPU等，是根据应用程序需求动态生成的，是Yarn中基本资源划分单位。一个NodeManager节点上同时存在多个Container。提交作业的每个task都运行在Container中
Yarn调度器
根据一些限制条件如每个队列分配多少资源、最多执行多少数量的作业，将系统中资源按照应用程序的资源需求分配给各个应用程序，资源分配单位就是上文提到的container，并且用户可以根据自己的需求设计新的调度器，目前Yarn也提供了多种可直接使用的调度器：
FIFOScheduler（先进先出调度器）：不支持抢占先机。如果有运行特别慢的任务，会影响其他任务
FairScheduler（公平调度器）：每个用户公平共享集群资源，支持抢占先机，如果有的任务长时间占用大量资源，超出其应该分配的资源比例，会终止得到过多资源的任务
CapacityScheduler（容量调度器）：有层次结构的队列，每个队列分配一定的容量（比如将小job和大job分配到不同的队列），单个队列内部支持FIFO
笔者强调：
1.Yarn并不清楚用户提交程序的运行机制，只提供运算资源的调度（用户程序向yarn申请资源，yarn负责分配资源）
2.Yarn中的主管角色是ResourceManager，具体提供运算资源的角色是NodeManager
3.Yarn与运行的用户程序完全解耦，意味着Yarn上可以运行各种类型的分布式运算程序，如Spark、MapReduce、Storm、Tez等，前提是这些技术框架中有符合Yarn规范的资源请求机制即可
4.因为Yarn不参与用户程序的执行等，使得Yarn成为一个通用的资源调度平台。企业中以前存在的各种计算引擎集群都可以整合在一个资源管理平台上，提高资源利用率
5. The scheduler does not participate in any work related to the specific application, such as monitoring or tracking the execution status of the application, etc., nor is it responsible for restarting failed tasks caused by application execution failure or hardware failure, these are all handed over to the application. The program-related Application Master is completed.
Related Articles:
A few questions you should know about HDFS
Pay attention to WeChat public account: big data learning and sharing , get more technical dry goods