1, MapTask operating mechanism Detailed Map task parallelism and the
entire Map stage of the process substantially as shown in FIG.
Brief overview: inputFile is logically split by a split cut into a plurality of files, read the contents of a map Record by row for (the user's own implementation) processing, the data is handed over to the end of the collector OutputCollector map processing, key results of its partition (using hash partitioning default), and then write buffer, each map task has a memory buffer stores the output map, and when the buffer is almost full when the need to buffer the data to a temporary file way to store to disk, when all temporary files after the end of the entire map task on the disk map task produced do this merger to generate the final official output file, and then wait reduce task to pull data.
Detailed steps:
1. First, the InputFormat component reads data (by default the TextInputFormat) are logically sliced planning to give the splits, the number of split number will correspond to the input start MapTask directory files getSplits method. correspondence between the split and the block is the default one.
2, the input file after cutting into the splits, the object is read by RecordReader (default LineRecordReader), to \ n as the separator, a line of data is read, return <key, value>. Key said the first character of each line offset value, value that represents the lines of text content.
3, the read returns split <key, value>, users enter their inheritance Mapper classes, the user performs a function of rewriting the map. RecordReader read a line called once here.
4, after completion of the logic map, the map will result for each collect data collected by context.write. In the collect, the process will first be partitioned, default HashPartitioner.
MapReduce provides Partitioner interfaces, its role is to decide which of the current output data which reduce task should be handed over to the final processing based on the number or value and reduce the key. The default number of key hash to reduce task then modulo. The default modulo way just to reduce the average processing power, if the user's own demand for Partitioner, can be customized and set to the job.
5, the next data is written to memory, this memory is called ring buffer zone, the buffer zone is to map the results to collect bulk, reduce the influence of disk IO. Our key / value pairs and Partition result will be written to the buffer. Before writing course, key values and the value will be serialized into a byte array.
In fact, the ring buffer is an array, the array storage key, value sequence data and metadata information key, value, including Partition, the starting position of the key, the initial value of the position and length of value. Ring structure is an abstraction.
Buffer is limited in size, the default is 100MB. When the output map task a lot, it may explode memory, so it is necessary under certain conditions, the temporary data buffer is written to disk, and then re-use this buffer. This data is written to the disk from memory process called Spill, Chinese can be translated as overflow write. The overflow writing is done by a separate thread, does not affect the write buffer to map the results of threads. It should not prevent the output map of overflow when writing thread starts, so there is a whole buffer overflow written proportion spill.percent. The default ratio is 0.8, that is, when the buffer data has reached the threshold value (buffer size * spill percent = 100MB * 0.8 = 80MB), write overflow thread starts, this lock 80MB memory, perform overflow writing process. Map task outputs also the rest of the previous 20MB memory write, independently of each other.
6, after the overflow when writing thread is started, you need to key in this space 80MB do sort (Sort). Sort MapReduce model is the default behavior of the sort here is the sequence of bytes to do the sort.
If the job is set too Combiner, then now is the time to use Combiner. Will have the same key of the key / value pairs add value, reduce the amount of data written to disk overflow. Combiner optimizes the intermediate results of MapReduce, so it will be used multiple times throughout the model.
What a scene that Combiner to use it? From this analysis, Combiner Reducer output is the input of, Combiner not change the final result of the calculation. Combiner input key that should be used only to Reduce / value of the output key / value exactly the same type, and does not affect the final result of the scene. Such accumulation, the maximum value and the like. Combiner got to use caution, if good use, the efficiency of its job to help, on the contrary will affect the final result of reduce.
7, merger overflow write files: write each overflow will generate a temporary file on disk (written before the judge whether there combiner), if the output map is really great, there are many such write overflow occurs, the disk there will be a corresponding number of temporary files exist. When processing starts after the entire data of the temporary file on disk by merging merge, because ultimately only one file, written to disk, and provides an index file for the file to record data corresponding to each reduce the offset.
At this point the entire map end stage.
mapTask some basic configuration settings (mapred-site.xml among Society):
Set a: setting the size of the circular buffer memory values (default settings are as follows)
mapreduce.task.io.sort.mb 100
disposed II: setting a write overflow percentage (The default settings are as follows)
mapreduce.map.sort.spill.percent 0.80
Setting 3: setting overflow write data directory (the default setting)
mapreduce.cluster.local.dir hadoop.tmp.dir $ {} / mapred / local
settings Four: Setting a maximum number of merger underflow write file (default settings are as follows)
mapreduce.task.io.sort.factor 10
2, and a working mechanism ReduceTask of parallelism reduceTask
Reduce roughly classified into copy, sort, reduce the three stages, the first two stages focus. copy eventFetcher stage contains a map to get a list of completed, to copy data from the Fetcher thread in this process will start two merge threads, respectively inMemoryMerger and onDiskMerger, respectively merge the data in memory to disk and the disk the data merge. After the completion of the data to be copy, copy phase is complete, start the sort phase, the main phase of the implementation of finalMerge sort operation, pure sort stage, after the completion of stage is reduce, reduce call a user-defined function for processing.
Detailed steps:
. 1, the Copy Phase, simply pull data. Reduce process started some data copy thread (Fetcher), by way of HTTP requests maptask obtain files they own.
2, Merge stage. The merge operation merge here map end, but the array is stored in a different value to the map-side copy. Copy the data will first come into a memory buffer, the buffer size is more flexible than the end of the map here. merge three forms: memory to memory; memory to disk; disk to disk. Dir default form is not enabled. When the amount of data in the memory reaches a certain threshold, the start of the merge memory to disk. And map similar end, which is the overflow of the process of writing, this process if you have set Combiner, also enabled, and then generate a large number of overflow write files on the disk. The second way to merge has been in operation until the end of the map data when there is no end, and then start the third disk-to-disk mode merge to generate the final document.
3, merge sort. After the data is combined into a dispersion of large data, the data will then sort the merged.
4, the sorted key value call to the reduce method, equal key value pairs reduce method is called once, each call will produce zero or more key-value pairs, and finally the output of these keys written to the HDFS file.
