Before I saw the hive statement in the implementation of mapreduce job log print when the map is to perform 100%, then the implementation reduce. Today found in the implementation of the hive sql map has not been performed to complete, reduce the task began. As shown below:
Beginning do not know why, and later through the degree of your mother, and found that there is a parameter mapreduce.job.reduce.slowstart.completedmaps in mapred-site.xml configuration file, this parameter can be controlled when the map task execution to which proportion can begin application resources reduce task.
default allocation:
<property>
<name>
mapreduce.job.reduce.slowstart.completedmaps
</name>
<value>
0.05
</value>
<description>
Fraction of the number of maps in the job which should be complete before reduces are scheduled for the job.
</description>
</property>The default configuration is 0.05, then apply map task to start the execution resources to reduce to 5% of the time, reduce operation started, can reduce start copying data and map the results do reduce shuffle operation.
To 100% of the map are performed before the operation began execution reduce, because this parameter is set to 1 mapreduce.job.reduce.slowstart.completedmaps.
How much is appropriate to configure it?
mapreduce.job.reduce.slowstart.completedmaps If this parameter is set too low, it will reduce the premature application of resources, a waste of resources; if this parameter is set too high, such as 1, then only when the map is completed after all only to reduce application resources, reduce operation began, in fact, serial execution, can not be used in parallel full use of resources.
If the number of map more generally recommended before beginning to reduce application resource.
Parameters source code analysis
mapreduce.job.reduce.slowstart.completedmaps parameters source code analysis
