Reprinted: https://blog.csdn.net/wisgood/article/details/42125367
Every time we execute hive's hql, the shell will prompt a paragraph:
- ...
- Number of reduce tasks not specified. Estimated from input data size: 500
- In order to change the average load for a reducer (in bytes):
- set hive.exec.reducers.bytes.per.reducer=<number>
- In order to limit the maximum number of reducers:
- set hive.exec.reducers.max=<number>
- In order to set a constant number of reducers:
- set mapred.reduce.tasks=<number>
- ...
This is a frequent means of tuning, mainly determined by the following three attributes
The hive.exec.reducers.bytes.per.reducer parameter controls how many reducers a job will process, based on the total size of the input files . Default 1GB.
This controls how many reducers a map-reduce job should have, depending on the total size of input files to the job. Default is 1GBThe hive.exec.reducers.max parameter controls the maximum number of reducers. If input / bytes per reduce > max, the number of reducers specified by this parameter will be started. This does not affect the setting of the mapre.reduce.tasks parameter. The default max is 999.
This controls the maximum number of reducers a map-reduce job can have. If input_file_size divided by "hive.exec.bytes.per.reducer" is greater than this value, the map-reduce job will have this value as the number reducers. Note this does not affect the number of reducers directly specified by the user through "mapred.reduce.tasks" and query hints
If the parameter mapred.reduce.tasks is specified, hive will not use its estimation function to automatically calculate the number of reducers, but use this parameter to start the reducer. Default is -1.
This overrides the hadoop configuration to make sure we enable the estimation of the number of reducers by the size of the input files. If this value is non-negative, then hive will pass this number directly to map-reduce jobs instead of doing the estimation.
The number of reduce settings actually has a great impact on execution efficiency:
1. If the reduce is too small: If the amount of data is large, it will cause the reduce to be abnormally slow, so that the task cannot be completed, and there may be OOM.
2. If there are too many reducers: too many small files are generated, the cost of merging is too high, and the memory usage of the namenode will also increase.
If we don't specify mapred.reduce.tasks, hive will automatically calculate how many reducers are needed.
Calculated formula: Number of reducers = InputFileSize / bytes per reducer
These several rough formulas, the detailed formulas are in:
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Let's take a look first:
1. The method of calculating the size of the input file: it is actually very simple, traverse each path to get the length, and accumulate it.
- + * Calculate the total size of input files.
- + * @param job the hadoop job conf.
- + * @return the total size in bytes.
- + * @throws IOException
- + */
- + public static long getTotalInputFileSize(JobConf job, mapredWork work) throws IOException {
- + long r = 0;
- + FileSystem fs = FileSystem.get(job);
- + // For each input path, calculate the total size.
- + for (String path: work.getPathToAliases().keySet()) {
- + ContentSummary cs = fs.getContentSummary(new Path(path));
- + r += cs.getLength();
- + }
- + return r;
- + }
2、估算reducer的个数，及计算公式：
注意最重要的一句话： int reducers = (int)((totalInputFileSize + bytesPerReducer - 1) / bytesPerReducer);
- + /**
- + * Estimate the number of reducers needed for this job, based on job input,
- + * and configuration parameters.
- + * @return the number of reducers.
- + */
- + public int estimateNumberOfReducers(HiveConf hive, JobConf job, mapredWork work) throws IOException {
- + long bytesPerReducer = hive.getLongVar(HiveConf.ConfVars.BYTESPERREDUCER);
- + int maxReducers = hive.getIntVar(HiveConf.ConfVars.MAXREDUCERS);
- + long totalInputFileSize = getTotalInputFileSize(job, work);
- +
- + LOG.info("BytesPerReducer=" + bytesPerReducer + " maxReducers=" + maxReducers
- + + " totalInputFileSize=" + totalInputFileSize);
- + int reducers = (int)((totalInputFileSize + bytesPerReducer - 1) / bytesPerReducer);
- + reducers = Math.max(1, reducers);
- + reducers = Math.min(maxReducers, reducers);
- + return reducers;
- + }
3、真正的计算流程代码：
- + /**
- + * Set the number of reducers for the mapred work.
- + */
- + protected void setNumberOfReducers() throws IOException {
- + // this is a temporary hack to fix things that are not fixed in the compiler
- + Integer numReducersFromWork = work.getNumReduceTasks();
- +
- + if (numReducersFromWork != null && numReducersFromWork >= 0) {
- + LOG.info("Number of reduce tasks determined at compile: " + work.getNumReduceTasks());
- + } else if(work.getReducer() == null) {
- + LOG.info("Number of reduce tasks not specified. Defaulting to 0 since there's no reduce operator");
- + work.setNumReduceTasks(Integer.valueOf(0));
- + } else {
- + int reducers = estimateNumberOfReducers(conf, job, work);
- + work.setNumReduceTasks(reducers);
- + LOG.info("Number of reduce tasks not specified. Estimated from input data size: " + reducers);
- }
- }
这就是reduce个数计算的原理。
By the way :
今天中午在群里看到一位朋友问到：
当前hive的reduce个数的设定是依据map阶段输入的数据量大小来除以每一个reduce能够处理的数据量来决定有多少个的，但是考虑到经过map阶段处理的数据很可能可输入数据相差很大，这样子的话，当初设定的reduce个数感觉不太合理。。。请问hive当前能支持依据map阶段输出数据量的大小决定reduce个数么？（但是，reduce任务的开启是在有某些map任务完成就会开始的，所以要等到所有map全部执行完成再统计数据量来决定reduce个数感觉也不太合理） 有没有什么好方法？谢谢
这个问题的大意是，reducer个数是根据输入文件的大小来估算出来的，但是实际情况下，Map的输出文件才是真正要到reduce上计算的数据量，如何依据Map的阶段输出数据流觉得reduce的个数，才是实际的问题。
我给出的思路是：
1、hack下源码，计算下每个map输出的大小×map个数不就估算出map总共输出的数据量了吗？不用等它结束，因为每个map的处理量是一定的。
2、你把源码的 总输入量 / 每个reduce处理量 改成 总输出量 / 每个reduce处理量不就行了？（总输出=每个Map输出文件的大小×map个数）
Ps:最后朋友提到：
建议不错，虽然有一定误差。 谢谢。 不过，如果ｆｉｌｔｅｒ ｐｕｓｈ ｄｏｗｎ的话，每一个ｍａｐ的输出大小差别可能比较大。。。而且ｆｉｌｔｅｒ ｐｕｓｈ ｄｏｗｎ 现在应该是ｈｉｖｅ默认支持的了
The general idea is that there will still be some errors, and the predicate pushdown may affect the output size of the Map.