Yarn introduced three major components and operating procedures

Yarn three components

ResourceManager

1. ResourceManager responsible for resource management and allocation of the entire cluster, is a global resource management systems.
2. NodeManager in a heartbeat way to report resource usage to ResourceManager (mainly CPU and memory usage). RM only accept returns resource information NM, for the specific resources to deal with the NM to deal with their own.
3. YARN Scheduler assign resources based on application request, application job is not responsible for monitoring, tracking, operational status feedback, to start work.

 

NodeManager

1. NodeManager resource and task manager on each node, which is managing the machine agent in charge of the node program run, and manage and monitor the node resources. YARN each cluster node runs a NodeManager.
2. NodeManager regular reports node resources (CPU, memory) and the operating state of usage of the Container ResourceManager. When the automatic connection ResourceManager down NodeManager RM standby node.
3. It receives and processes from NodeManager Container ApplicationMaster start, stop, and other requests.

 

ApplicationMaster

1. Each application submitted by a user contains a ApplicationMaster, it can run on a machine other than the ResourceManager.
2. RM responsible for negotiating with the dispatcher to get the resources (expressed Container).
3. You will be assigned to the task of further internal tasks (secondary distribution of resources).
4. NM communicate with the start / stop task.
5. Monitoring all the tasks running, and when a job fails to re-apply for funding for the mission to restart the task.
6. Comes with two current YARN ApplicationMaster implemented, it is an example of a procedure used to demonstrate the AM DistributedShell write process, it can apply to a number of parallel running a Container Shell Shell scripts or command; and the other is running MapReduce application AM- MRAppMaster.

Note: RM is only responsible for monitoring AM, and start it in the AM fail time. RM is not responsible for the fault-tolerant AM internal task, the task is done by the fault-tolerant AM.

 

 

Yarn of running processes

 

1. submitted to the client application RM, which includes the information necessary to start ApplicationMaster application, e.g. ApplicationMaster program, ApplicationMaster start command, the user program and the like.
2. ResourceManager start a container used to run ApplicationMaster.
3. Start of ApplicationMaster register itself with the ResourceManager, keeping heartbeat and RM after a successful start.
4. ApplicationMaster sends a request to the ResourceManager, apply a corresponding number of container.
5. Successful applicants container, is initialized by the ApplicationMaster. After starting the initialization of container information, the NodeManager communication with the corresponding AM, NM required starting container.
6. NM Start Start container.
7. During the container operation, ApplicationMaster of the container to be monitored. container report their progress and status information corresponding to AM via RPC protocol.
8. After the application is running, ApplicationMaster to deregister its ResourceManager, and allows the container belonging to it be withdrawn.

 

Detailed Process