In the working mechanism of MapReduce1, the roles mainly include client, Jobtracker, Tasktracker
Jobtracker is mainly responsible for coordinating the running of jobs; while Tasktracker is responsible for running tasks after job division.
MapReduce process:
1. First, the client requests a new job from Jobtracker, and Jobtracker checks whether the output path of the job exists. Throws an exception if it exists. If it does not exist, Jobtracker will return the submission path and jobID of job-related resources to the client.
2. The next step is that the client submits the resources (jar files, configuration files) required by the job to the shared file system. And tell Jobtracker that the job has been copied to the shared file system, ready to execute.
3. Jobtracker puts the submitted Job into the internal task queue, which is scheduled by the job scheduler and initialized (including creating a container that represents the running job to encapsulate tasks and record information)
4. After that, the job scheduler of jobtracker obtains the input slices calculated by the client from the shared file system to create a task run list
5. Tasktracker maintains communication with Jobtracker through heartbeat, reports its own status, and whether it is ready to run a task. If ready, Jobtracker obtains a task from Jobtracker through a certain scheduling algorithm and assigns it to Tasktracker.
6. Tasktracker obtains task-related resources in the shared file system, implements jar localization, and creates a corresponding folder and a taskrunner to run the task.
7.taskrunnr will start a new JVM and run tasks in the newly started JVM.
Progress and status updates: There is a separate thread that reports the current task status to tasktracker. At the same time, Tasktracker sends status to Jobtracker through heartbeat every 5s. The JobTracker merges these updates and sends them to the client.