Outline
hadoop is mainly used to deal with storage and computing, in front of huge amounts of data hdfs file system, the focus is on storing huge amounts of data. With so much data, how should we in which these data to extract the information we need it? This time there was a very important member of the MapReduce hadoop in.
MapReduce is a programming model for large data sets (greater than 1TB) parallel computing. Note that this parallel computing, it is no longer the traditional sense of the serial on our calculations, it can be carried out in parallel computing in a different process. The concept "Map (Mapping)" and "Reduce (reduction)," is their main thoughts were borrowed from functional programming languages, as well as borrowed from the vector programming language features. It is very easy for programmers in the case will not be distributed and parallel programming will own programs running on a distributed system. The current software implementation is to specify a Map (mapping) function, a key-value pair is used to map into a new set of key-value pairs specified concurrent Reduce (reduction) function to ensure that all of the key-value mappings each group share the same key.
Here's MapReduce implementation of internal procedures and principles to do the next introduction.
MapReduce jobs run the process
1, start a job on the client.
2, a request to the Job ID NodeManager.
3, the copy job operation required resource file to the HDFS, packaged MapReduce comprises JAR files, configuration files, and enter a client computing resultant split information. These files are stored in the file NodeManager specially created for the job folder. Folder named Job ID of the job. JAR files by default will be 10 copies (mapred.submit.replication attribute control); dividing the input information to tell how many map tasks NodeManager should start and other information for this job.
4, after NodeManager receives a job, it is placed in a job queue, waiting for job scheduler schedule it (this is not like the computer in the process of scheduling, huh, huh), when the job scheduler according to their own scheduling algorithm when the schedule to the job, the task will create a map based on the input information for each division is divided and assigned to the ResourceManager map task execution. For map and reduce tasks, ResourceManager map a fixed number of slots and grooves reduce the number and size of the host memory core. It should be emphasized that: map task is not casually assigned to a ResourceManager is, there is a concept called: Localized Data (Data-Local). Means: assigning tasks to map the ResourceManager data blocks comprising processing the map, while the program package to the ResourceManager JAR copy run up, called "move operation, data is not moved." The data does not consider the allocation reduce localization task.
5, ResourceManager NodeManager from time to time will send a heartbeat, tell NodeManager it is still running, while the heart is also carrying a lot of information, such as the current progress of the task is completed map information. When the last task NodeManager receive job completion information, it gave the job is set to "success." When JobClient query state, it has been learned that the task is completed, a message is displayed to the user.
These are the client NodeManager, ResourceManager to analyze the level of MapReduce works, let's again more carefully, hierarchical map task and reduce task analysis to analyze it.
Large columns Hadoop Learning (Five) -MapReduce architecture principles ap, Reduce task Shuffle and sort of process "class =" headerlink "title = " Map, Reduce task Shuffle and sort of process "> Map, Reduce and sort tasks Shuffle the process of
Analysis Process
1, each input slice will map a task to process, by default, a size of a block of HDFS (default 64M) a fragment, of course, we can set the size of the block. Results will map output temporarily in a ring buffer memory (default size of the buffer 100M, io.sort.mb controlled by the property), when the buffer is about to overflow (default buffer size is 80% ,), creates a spill file attributes io.sort.spill.percent controlled by the local file system, the data buffer is written to this file.
2, before writing to disk, according to the number of first thread reduce task divides the data into the same number of partitions, i.e. data corresponding to a task reduce a partition. This is done to avoid some tasks allocated to reduce large amounts of data, while others reduce task was assigned little data or no data assigned to the embarrassing situation. In fact, the data partition is the process of hash. Then to sort the data in each partition, if this time is set Combiner, the result of the sort carried out Combia operation, the aim is to make as little data is written to disk.
3, when the map task output last record, there may be a lot of spill file, then you need to merge these files. The combined process will continue to be sorted and combia operation, has two purposes: 1 to minimize the amount of data written to each disk; 2 minimize the amount of data transmitted over the network to the next stage of replication... Finally merged into a partitioned and sorted files. To reduce the amount of data transmitted over the network, where the data can be compressed, as long as the mapred.compress.map.out it is set to true.
4, a copy of the data partition corresponding to reduce tasks. One might ask: how data partition corresponding reduce its know what it is? In fact, map and its parent NodeManager task has been to keep in touch, and NodeManager has been maintained and ResourceManager heartbeat. So ResourceManager save the macro information for the entire cluster. As long as the task of acquiring map reduce output position corresponding to the ResourceManager on the ok Oh.
Here, map analysis done on end. Shuffle that in the end what is it? Shuffle in Chinese means "shuffle", if we look at this: a map data generated by the hash result of the process of partition was assigned to a different reduce task, in fact, a data shuffling process.
5, Reduce will receive different map data from the mission, and each map data transmitted is ordered. Reduce the amount of data if the receiving side is relatively small, it is stored directly in memory (the buffer size by the mapred.job.shuffle.input.buffer.percent attribute control, expressed as a percentage for this purpose as heap space), if the amount of data more than a certain percentage of the buffer size (determined by mapred.job.shuffle.merge.percent), after the merger of data written to disk overflow.
6, with the increase in overflow write files, background thread will merge them into one larger ordered file, do so in order to save time behind the merger. In fact, whether the map-side end or reduce, MapReduce is repeatedly perform the sort, merge operations, and now finally understand why some people would say: hadoop is the sort of soul.
7, the merger process will produce many intermediate files (written to the disk), but the data will be written to disk MapReduce as little as possible, and the final result of the merger have not written to disk, but entered directly into the reduce function.