Hadoop series-job submission process on Yarn

Submission process

1. Assignment submission

• ClientCall the job.waitForCompletionmethod to submit MapReduce jobs to the entire cluster.
• ClientTo ResourceManagera job application id.
• RMClientThe path and job submitted to the returned resource id.
• ClientSubmit the jar package, slice information, and xml configuration file to the specified path.
• ClientAfter the submission is completed, MRapply for operation mrApplicationMaster.

2. Job initialization

• RMAfter receiving Clientthe request, convert the job into a Task and submit it to the task queue.
• A certain person is free to NodeManagerreceive the job.
• NMCreate Containerand apply for operation mrApp.
• NMDownload the Clientsubmitted resources.

3. Task Assignment

• mrAppRMRun multiple MapTasktasks to request .
• RMAssign the mrAppsubmitted tasks to multiple NMand NMstart them separately MapTask.
• MapTaskAfter running, run mrAppto the RMrequest ReduceTask.
• RduceTaskFrom the MapTaskacquired data the respective partition.
• After the program is complete, MRthe RMcancellation of their own.