MapReduce source code analysis process
Mapper
First mapper mapping is complete, the word is mapped into a form (word,. 1) is.
MapReduce process, Map MapTask stage is also called, in MapTask will be called mapper () method overridden by our users run () method,
Distributed computing application into at least two phases often requires: Map and Reduce phase stage.
The first stage, i.e., stage maptask Map concurrent instances, running completely independently in parallel, independent of each other, each of the plurality of files to be processed, such as Map into three parts, respectively, on each of the data nodes in the cluster, Map stage by maptask process to handle incoming file already exists, the data read line by line, word by internal air Geqie branches, after the segmentation is complete, the statistics word out to hashmap store, where the word is key, as the word of 1 value. Allocated to itself until after all the read pieces of data, this is divided into two hashmap hashmap (partitions sorted) in accordance with the range of the first letter, two hashmap are: HashMap (ap) and HashMap (qz).
The second stage of the reduce task unrelated concurrent instances, but they depend on the output data of all concurrent instances maptask on a stage.
reduce task statistics were beginning ap words and words that begin with qz, and then output to a file.
Note: MapReduce programming model can contain only one map phase and a reduce phase, if the user's business logic is very complex, it can only multiple MapReduce programs, serial execution.
So how maptask assignments?
How reducetask task allocation?
How convergence between maptask and reducetask?
If maptask fails, how to deal with?
If maptask own responsibilities and output data partition, a lot of trouble
coordinating MrAPPMaster process responsible for the entire program schedule and status.
Three processes corresponding to the three classes:
three processes:
. 1) MrAppMaster: the process responsible for the entire program schedule and coordinate the state
2) MapTask: responsible for processing the entire data map phase flow
3) ReduceTask: reduce stage is responsible for the overall flow of data processing
corresponding to each category:
. 1) Driver stage
entire procedure requires a Drvier be submitted, is a description of various information necessary job object
2) Mapper Stage
(. 1) Mapper user-defined to inherit its parent class
(2) input data Mapper is in the form KV pair (type KV customizable)
(. 3) Mapper the logic within map ( ) process
output data (4) Mapper is in the form KV pair (type KV customizable)
(. 5) Map () method (MapTask process) for each <K, V> called once
3) Reducer stage
(1 ) the Reducer user-defined to inherit its parent class
(2) Reducer input data type of the corresponding output data type Mapper is also KV
(. 3) the Reducer of the logic within reduce () method
(4) Reducetask process per the same set of k <k, v> group is called once reduce () method
Determining mechanism and parallelism of the data slice MapTask
Stage 1. Map parallelism determined by a number of slices Job client when submitting the Job
2. Split each slice MapTask assigned a parallel processing example
3. By default, the tile size = BlockSize
not by way data slice 4 set overall, but Zhuge slice separately for each file.
job submission process analysis
1 to submit the task ----> 2 to check the status ( if (state == JobState.DEFINE) {submit();}) ----> 3.0 the Submit (): 3.1 ensure job status ( ensureState(JobState.DEFINE)) ----> 3.2 using the new API ( setUseNewAPI()) ----> 3.3 Connecting clusters ( connect()) ----> 3.4 acquire job submission according to get to the cluster ( final JobSubmitter submitter = getJobSubmitter(cluster.getFileSystem(), cluster.getClient())) ----> 3.5 Submitter submit task ( return submitter.submitJobInternal(Job.this, cluster)) -----> 3.5.1 to check whether to set the output path and output path exists ( checkSpecs(jobs)) ----> 3.5.2 registration JobId ( JobID jobId = submitClient.getNewJobID();) ----> directory to copy a file, the file is before us (setJarByClass (xxx.class)) set a good jar package ----> client is through after this method call InputFormat to slice ----> give us a slice of the input file, execute conf.setInt(MRJobConfig.NUM_MAPS, maps);slice information is written to the directory (submitJobDir) in ----> the job configuration information conf also submit to submitDir --- -> task formally submitted.
First, we call:
boolean b = job.waitForCompletion(true);
The main method:
public boolean waitForCompletion(boolean verbose
) throws IOException, InterruptedException,
ClassNotFoundException {
if (state == JobState.DEFINE) {
submit();
}
if (verbose) {
monitorAndPrintJob();
} else {
// get the completion poll interval from the client.
int completionPollIntervalMillis =
Job.getCompletionPollInterval(cluster.getConf());
while (!isComplete()) {
try {
Thread.sleep(completionPollIntervalMillis);
} catch (InterruptedException ie) {
}
}
}
return isSuccessful();
}
Then call submit () method:
/**
* Submit the job to the cluster and return immediately.
* @throws IOException
*/
public void submit()
throws IOException, InterruptedException, ClassNotFoundException {
ensureState(JobState.DEFINE);
setUseNewAPI();
connect();
final JobSubmitter submitter =
getJobSubmitter(cluster.getFileSystem(), cluster.getClient());
status = ugi.doAs(new PrivilegedExceptionAction<JobStatus>() {
public JobStatus run() throws IOException, InterruptedException,
ClassNotFoundException {
return submitter.submitJobInternal(Job.this, cluster);
}
});
state = JobState.RUNNING;
LOG.info("The url to track the job: " + getTrackingURL());
}In the submit () method, the ensureState(JobState.DEFINE)method confirms again whether the job status is DEFINE, if it is, it is set to use the new API (hadoop2.x version upgrade many new API, but when the old version of the call MapReduce programs, where automatically transferred to the new API), and then call the connect()method for establishing and yarn cluster connections.
connect()The method of the content is as follows:
private synchronized void connect()
throws IOException, InterruptedException, ClassNotFoundException {
if (cluster == null) {
cluster =
ugi.doAs(new PrivilegedExceptionAction<Cluster>() {
public Cluster run()
throws IOException, InterruptedException,
ClassNotFoundException {
return new Cluster(getConfiguration());
}
});
}
}
In the connect()method, first determine the cluster, if the cluster is null, it returns a new cluster, return new Cluster(getConfiguration());if the task is to configure local mode is a LocalMaster, if a yarn cluster is YarnMaster.
After connecting to the cluster, and then you can submit our task to be executed, and then execution
final JobSubmitter submitter = getJobSubmitter(cluster.getFileSystem(), cluster.getClient());
To get a submitter by clients and protocols cluster returned, and then execute the statement
return submitter.submitJobInternal(Job.this, cluster);
using the submitter to submit our real job tasks, submitJobInternal(Job.this, cluster)the method of this method is the real job submission tasks, the content of this method, see the article last.
Then checkSpecs(jobs)the method checks whether the output path is provided, and an output path exists, then execute the statement:
Path jobStagingArea = JobSubmissionFiles.getStagingDir(cluster, conf);Get our stagingDir cluster according to our set, i.e., temporary files generated during data MapReduce folder.
And then perform JobID jobId = submitClient.getNewJobID();our job to register, and then get JobId, by job.setJobID(jobId);setting our registered good jobId, and then perform copyAndConfigureFiles(job, submitJobDir);the directory to copy a file, the file is before us (setJarByClass (xxx.class)) set a good jar package, and then perform int maps = writeSplits(job, submitJobDir);after the client is to be cut, sliced give our input file calls InputFormat by this method, execution conf.setInt(MRJobConfig.NUM_MAPS, maps);slice information is written to the directory (submitJobDir) in. The so-called slice information is marked which machine which slices of data processing, information equivalent to an index, with the index information, MapReduce is APPMaster when the mission started several maptask can know and know which part of each processing machine data, so this information is to be submitted to the HDFS temporary directory.
And then perform writeConf(conf, submitJobFile);the configuration information conf our job is also to submit to submitDir, after the execution, the temporary directory is generated job.xml (job store configuration information, including the various clusters and manually configure the default configuration).
In fact, so far, this series of work is in preparation for the work of the cluster, create a temporary directory cluster, it is possible for all of the nodes in the cluster to access data, first stored in a temporary directory in the cluster jar package, and then placed a slice of information, and finally placed in a configuration file, so you can maptask to three temporary files read information stored in the folder, and then carry out their respective mandates.
The program then submit it on line 240 real job, and then began to run the task.
So far the program is finished.
to sum up
Submitted by the task process: First, check the status of the task, after checking the output directory, no problem, then start connecting the cluster, because the task is performed handed over to others in the cluster, so other people need to know this task is necessary information, job submission time necessary to submit the necessary information on these tasks to a temporary directory accessible to everyone (on HDFS), the necessary information includes: jar package, slice information, and configuration information (job.xml).
Annex: submitJobInternal (Job.this, cluster) of the original code method:
/**
* Internal method for submitting jobs to the system.
*
* <p>The job submission process involves:
* <ol>
* <li>
* Checking the input and output specifications of the job.
* </li>
* <li>
* Computing the {@link InputSplit}s for the job.
* </li>
* <li>
* Setup the requisite accounting information for the
* {@link DistributedCache} of the job, if necessary.
* </li>
* <li>
* Copying the job's jar and configuration to the map-reduce system
* directory on the distributed file-system.
* </li>
* <li>
* Submitting the job to the <code>JobTracker</code> and optionally
* monitoring it's status.
* </li>
* </ol></p>
* @param job the configuration to submit
* @param cluster the handle to the Cluster
* @throws ClassNotFoundException
* @throws InterruptedException
* @throws IOException
*/
JobStatus submitJobInternal(Job job, Cluster cluster)
throws ClassNotFoundException, InterruptedException, IOException {
//validate the jobs output specs
checkSpecs(job);
Configuration conf = job.getConfiguration();
addMRFrameworkToDistributedCache(conf);
Path jobStagingArea = JobSubmissionFiles.getStagingDir(cluster, conf);
//configure the command line options correctly on the submitting dfs
InetAddress ip = InetAddress.getLocalHost();
if (ip != null) {
submitHostAddress = ip.getHostAddress();
submitHostName = ip.getHostName();
conf.set(MRJobConfig.JOB_SUBMITHOST,submitHostName);
conf.set(MRJobConfig.JOB_SUBMITHOSTADDR,submitHostAddress);
}
JobID jobId = submitClient.getNewJobID();
job.setJobID(jobId);
Path submitJobDir = new Path(jobStagingArea, jobId.toString());
JobStatus status = null;
try {
conf.set(MRJobConfig.USER_NAME,
UserGroupInformation.getCurrentUser().getShortUserName());
conf.set("hadoop.http.filter.initializers",
"org.apache.hadoop.yarn.server.webproxy.amfilter.AmFilterInitializer");
conf.set(MRJobConfig.MAPREDUCE_JOB_DIR, submitJobDir.toString());
LOG.debug("Configuring job " + jobId + " with " + submitJobDir
+ " as the submit dir");
// get delegation token for the dir
TokenCache.obtainTokensForNamenodes(job.getCredentials(),
new Path[] { submitJobDir }, conf);
populateTokenCache(conf, job.getCredentials());
// generate a secret to authenticate shuffle transfers
if (TokenCache.getShuffleSecretKey(job.getCredentials()) == null) {
KeyGenerator keyGen;
try {
keyGen = KeyGenerator.getInstance(SHUFFLE_KEYGEN_ALGORITHM);
keyGen.init(SHUFFLE_KEY_LENGTH);
} catch (NoSuchAlgorithmException e) {
throw new IOException("Error generating shuffle secret key", e);
}
SecretKey shuffleKey = keyGen.generateKey();
TokenCache.setShuffleSecretKey(shuffleKey.getEncoded(),
job.getCredentials());
}
if (CryptoUtils.isEncryptedSpillEnabled(conf)) {
conf.setInt(MRJobConfig.MR_AM_MAX_ATTEMPTS, 1);
LOG.warn("Max job attempts set to 1 since encrypted intermediate" +
"data spill is enabled");
}
copyAndConfigureFiles(job, submitJobDir);
Path submitJobFile = JobSubmissionFiles.getJobConfPath(submitJobDir);
// Create the splits for the job
LOG.debug("Creating splits at " + jtFs.makeQualified(submitJobDir));
int maps = writeSplits(job, submitJobDir);
conf.setInt(MRJobConfig.NUM_MAPS, maps);
LOG.info("number of splits:" + maps);
// write "queue admins of the queue to which job is being submitted"
// to job file.
String queue = conf.get(MRJobConfig.QUEUE_NAME,
JobConf.DEFAULT_QUEUE_NAME);
AccessControlList acl = submitClient.getQueueAdmins(queue);
conf.set(toFullPropertyName(queue,
QueueACL.ADMINISTER_JOBS.getAclName()), acl.getAclString());
// removing jobtoken referrals before copying the jobconf to HDFS
// as the tasks don't need this setting, actually they may break
// because of it if present as the referral will point to a
// different job.
TokenCache.cleanUpTokenReferral(conf);
if (conf.getBoolean(
MRJobConfig.JOB_TOKEN_TRACKING_IDS_ENABLED,
MRJobConfig.DEFAULT_JOB_TOKEN_TRACKING_IDS_ENABLED)) {
// Add HDFS tracking ids
ArrayList<String> trackingIds = new ArrayList<String>();
for (Token<? extends TokenIdentifier> t :
job.getCredentials().getAllTokens()) {
trackingIds.add(t.decodeIdentifier().getTrackingId());
}
conf.setStrings(MRJobConfig.JOB_TOKEN_TRACKING_IDS,
trackingIds.toArray(new String[trackingIds.size()]));
}
// Set reservation info if it exists
ReservationId reservationId = job.getReservationId();
if (reservationId != null) {
conf.set(MRJobConfig.RESERVATION_ID, reservationId.toString());
}
// Write job file to submit dir
writeConf(conf, submitJobFile);
//
// Now, actually submit the job (using the submit name)
//
printTokens(jobId, job.getCredentials());
status = submitClient.submitJob(
jobId, submitJobDir.toString(), job.getCredentials());
if (status != null) {
return status;
} else {
throw new IOException("Could not launch job");
}
} finally {
if (status == null) {
LOG.info("Cleaning up the staging area " + submitJobDir);
if (jtFs != null && submitJobDir != null)
jtFs.delete(submitJobDir, true);
}
}
}