Table of contents
1. YARN is a general resource management system and scheduling platform
3. Program submission YARN interaction process
4. YARN resource scheduler Scheduler
Capacity Scheduler Feature Benefits
How to understand fair sharing
Fair Scheduler features and advantages
Apache Hadoop YARN (Yet Another Resource Negotiator Another Resource Coordinator) is a new Hadoop resource manager
YARN is a general resource management system and scheduling platform that can provide unified resource management and scheduling for upper-layer applications
Its introduction has brought great benefits to clusters in terms of utilization, unified resource management, and data sharing.
1. YARN is a general resource management system and scheduling platform
Resource management system: The hardware resources of the cluster have nothing to do with the running of the program, such as memory, CPU, etc.
Scheduling platform: How to allocate computing resources for multiple programs at the same time, scheduling rules (algorithms)
General: Not only supports MapReduce programs, but also theoretically supports various computing programs. YARN doesn't care what you do, only cares about the resources you need, and gives them to you in some cases, and returns them to me when they are used up.
Hadoop YARN can be understood as a distributed operating system platform, while computing programs such as MapReduce are equivalent to applications running on the operating system, and YARN provides these programs with the resources required for computing (memory, CPU, etc.) .
Hadoop has today's status, thanks to YARN. Because of YARN, more computing frameworks can be connected to HDFS, not just MapReduce. It is precisely because of YARN's tolerance that other computing frameworks can focus on computing performance. improvement.
HDFS may not be the best big data storage system, but it is the most widely used big data storage system, and YARN has contributed a lot
###################################################
2. YARN architecture
ResourceManager(RM)
The main role of the YARN cluster, which determines the final authority of resource allocation among all applications in the system, that is, the final arbiter
Receive the user's job submission and assign it through NM to manage the computing resources of each machine
NodeManager（NM）
The slave role in YARN, one on each machine, is responsible for managing the computing resources on this machine
According to the RM command, start the Container container, monitor the resource usage of the container, and report the resource usage of the RM master role
ApplicationMaster（AM）
Every application submitted by a user contains an AM
The boss in the application program is responsible for resource application at each stage of the program and supervises the implementation of the program.
###################################################
3. Program submission YARN interaction process
When a user submits an application to YARN, YARN will run the application in two stages
The first stage is that the client applies for resources and starts the ApplicationMaster of this program
In the second stage, ApplicationMaster applies for resources for the program according to the specific internal conditions of the program, and monitors its entire operation process until the operation is completed.
Specific process:
Step 1: The user submits the application program to the ResourceManager in YARN through the client (for example, the hadoop jar submits the MR program)
Step 2: ResourceManager allocates the first Container (container) for the application, and communicates with the corresponding NodeManager, asking it to start the ApplicationMaster of this application in this Container
Step 3: After the ApplicationMaster is started successfully, first register the ResourceManager and maintain communication, so that the user can directly view the running status of the application through the ResourceManager (a few percent processed)
Step 4: ApplicationMaster applies for resources for each Task item ResourceManager in this program, and monitors its running status
Step 5: Once the ApplicationMaster has applied for resources, it communicates with the corresponding NodeManager and asks it to start the task.
Step 6: After NodeManager has set up the running environment for the task, write the task start command into a script, and start the task through the script.
Step 7: Each task reports its status and progress to ApplicationMaster through an RPC protocol, so that ApplicationMaster can keep track of the running status of each task, so that the task can be restarted when the task fails. During the running of the application, the user can query the current running status of the application to the ApplicationMaster through RPC at any time
Step 8: After the application finishes running, the ApplicationMaster logs out to the ResourceManager and closes itself
###################################################
4. YARN resource scheduler Scheduler
Ideally, requests made by applications would be immediately approved by YARN. But in practice, resources are limited, and on a busy cluster,
Applications will often need to wait for some of their requests to be fulfilled. The job of the YARN scheduler is to allocate resources to applications according to some defined policies
In YARN, the Scheduler is responsible for allocating resources to applications, which is one of the core components of ResourceManager. Scheduler is completely dedicated
Used for scheduling jobs, it cannot track the state of the application.
Generally speaking, scheduling is a difficult problem, and there is no one "best" strategy. For this reason, YARN provides a variety of schedulers and configurable strategies to choose from
FIFO Scheduler (first in first out scheduler), Capacity Scheduler (capacity scheduler), Fair Scheduler (fair scheduler).
Apache version YARN uses Capacity Scheduler by default.
If you need to use other schedulers, you can configure them in yarn.resourcemanager.scheduler.class in yarn-site.xml.
FIFO Scheduler is the original scheduler implementation of JobTracker in Hadoop1.x, and this scheduler is preserved in YARN
FIFO Scheduler is a first-in-first-out idea, that is, the application submitted first runs first. Scheduling work regardless of priority and scope, suitable for low load
small clusters. It is less efficient and can cause problems when using large shared clusters.
FIFO Scheduler has a global control queue queue, the default queue name is default, the scheduler will get all the current cluster
Resource information acts on this global queue.
Advantages:
No configuration required, first come first served, easy to execute
Disadvantages:
The priority of tasks will not become higher, so high priority jobs need to wait, not suitable for shared clusters
Capacity Scheduler capacity scheduling is the default scheduling strategy of Apache Hadoop3.x. This strategy allows multiple organizations to share the entire cluster resource, each
Organizations can get a portion of the computing power of the cluster. By assigning dedicated queues to each organization and then assigning certain cluster resources to each queue,
In this way, the entire cluster can provide services to multiple organizations by setting up multiple queues
Capacity can be understood as individual resource queues, which are allocated by users themselves. The interior of the queue can be divided vertically, so that a
Multiple members within the organization can share this queue resource. Within a queue, resource scheduling adopts a first-in-first-out (FIFO) strategy.
The Capacity Scheduler scheduler divides resources in units of queues. To put it simply and popularly, each queue has independent resources, and the structure of the queue
and resources are configurable.
###################################################
Capacity Scheduler Feature Benefits
Hierarchical Queues
Hierarchical management makes it easier and more reasonable to allocate and limit the use of resources.
Capacity Guarantees
A resource ratio can be set on each queue to ensure that each queue does not occupy the resources of the entire cluster.
Security
Each queue has strict access control. Users can only submit tasks to their own queues, and cannot modify or access tasks in other queues.
Elasticity
Free resources can be assigned to any queue.
When multiple queues compete, they will be balanced according to the weight ratio.
Fair Scheduler is called fair scheduling, which provides another way for YARN applications to fairly share resources in large clusters. make all apps in ping
On average, an equal share of resources can be obtained over time.
The design goal of Fair Scheduler is to allocate fair resources for all applications (the definition of fairness is set by parameters).
Fair scheduling can work across multiple queues, allowing resource sharing and preemption
How to understand fair sharing
There are two users A and B, each with its own queue.
A starts a job, and since there is no demand from B, it allocates all available resources of the cluster.
Then B starts a job while A's job is still running. After a while, A and B each use half of the resources.
Now, if B user starts a second job while other jobs are still running, it will share its resources with another job of B, so each job of B will
has a quarter of the resources, and A's continuation will have half of the resources. The result is that resources are shared fairly among users
Fair Scheduler features and advantages
Hierarchical Queues: Queues can be arranged in a hierarchy to divide resources, and weights can be configured to share the cluster in specific proportions
User- or group-based queue mapping: Queues can be assigned based on the user name or group that submitted the task. If the task specifies a queue, in that queue
submit task
Resource preemption: Depending on the configuration of the application, preemption and allocation of resources can be friendly or mandatory. Resource preemption is not enabled by default
Guaranteed minimum quota: You can set the minimum queue resource, allowing the guaranteed minimum share to be allocated to the queue to ensure that users can start tasks. when the queue cannot
When the minimum resource is met, it can be preempted from other queues. When the queue resources are not used up, they can be used by other queues. This is useful for ensuring that certain users, groups, or
Production apps always get enough resources
Allow resource sharing: that is, when an application is running, if other queues have no tasks to execute, other queues can be used. When other queues have applications that need resources
Then release the occupied queue. All applications allocate resources from resource queues
By default, there is no limit to the number of concurrently running applications per queue and user. Can be configured to limit the number of applications that queues and users can execute in parallel. limit parallelism
Executing the number of applications will not cause task submission failure, and the excess applications will wait in the queue.