hive on tez resource control

sql

insert overwrite table dwintdata.dw_f_da_enterprise2
select *
from dwintdata.dw_f_da_enterprise;

hdfs file size display

Note that there are 17 files here, totaling 321M, and are finally divided into 21 tasks.

Why are there 21 tasks? Isn’t it 128M or 64M? Or is it less than 128? Is there one map for each file?

tez ui log

Look carefully at the logs of each task

map0 data_source=CSIG/HIVE_UNION_SUBDIR_1/000008_0:0+16903572

map1 data_source=CSIG/HIVE_UNION_SUBDIR_1/000000_0:0+16960450

map2 data_source=CSIG/HIVE_UNION_SUBDIR_1/000003_0:0+16808165

map3 data_source=CSIG/HIVE_UNION_SUBDIR_1/000001_0:0+17007259

map4 data_source=CSIG/HIVE_UNION_SUBDIR_1/000006_0:0+16877230

map5 data_source=CSIG/HIVE_UNION_SUBDIR_1/000004_0:0+16941186

map6 data_source=hehe/HIVE_UNION_SUBDIR_2/000004_0:0+16777216

map7 data_source=hehe/HIVE_UNION_SUBDIR_2/000002_0:0+16777216

map8 data_source=CSIG/HIVE_UNION_SUBDIR_1/000002_0:0+16946639

map9 data_source=CSIG/HIVE_UNION_SUBDIR_1/000009_0:0+16855768

map10 data_source=hehe/HIVE_UNION_SUBDIR_2/000001_0:0+16777216

map11 data_source=CSIG/HIVE_UNION_SUBDIR_1/000005_0:0+16872517

map12 data_source=hehe/HIVE_UNION_SUBDIR_2/000000_0:0+16777216

map13 data_source=hehe/HIVE_UNION_SUBDIR_2/000006_0:0+16777216

map14 data_source=hehe/HIVE_UNION_SUBDIR_2/000000_0:16777216+729642 Pay attention here

data_source=hehe/HIVE_UNION_SUBDIR_2/000001_0:16777216+7188613

map15 data_source=CSIG/HIVE_UNION_SUBDIR_1/000007_0:0+16761291

map16 data_source=hehe/HIVE_UNION_SUBDIR_2/000005_0:0+16777216

map17 data_source=hehe/HIVE_UNION_SUBDIR_2/000003_0:0+16777216

map18 data_source=hehe/HIVE_UNION_SUBDR_2/000002_0:16777216+7404916

data_source=hehe/HIVE_UNION_SUBDIR_2/000005_0:16777216+7341669

map19 data_source=hehe/HIVE_UNION_SUBDIR_2/000003_0:16777216+7378488

data_source=hehe/HIVE_UNION_SUBDIR_2/000006_0:16777216+7268763

map20 data_source=hehe/HIVE_UNION_SUBDIR_2/000004_0:16777216+7070700

data_source=hehe/000001_0:0+12488

16777216 What is this number? Everyone should be sensitive to 1024*1024*16=16777216=16M. Does it mean that map can only read 16M?

The total size of my 18 files is 321M. Because some of them are less than 16M, they are finally divided into 21 maps. So why is it a 16M map?

tez.grouping.min-size=16777216

tez.grouping.max-size=134217728 --128M

tez.grouping.split-waves=1.7

Borrowing from an article I’ve seen before

Hive parameter control principle and tuning experience based on Tez engine map and reduce numbers_tez.grouping.max-size_abcdggggggg’s blog-CSDN blog

It seems to make sense here, but it doesn't seem to make sense. The minimum is 16M and the maximum is 128M. What if I have a 64M file?

Divide it into 1 map or 64/16=4 maps or 64+64 2 maps

Test of mapper number

Test parameter set tez.grouping.min-size

Test 1

set tez.grouping.min-size=16777216;

There are 21 maps, 2 reducers, 21 files, and it takes 6 seconds.

Test 2

set tez.grouping.min-size=67108864; --64M

There are 8 maps, 2 reducers, and 12 generated files. It takes 8 seconds.

Test 3

set tez.grouping.min-size=134217728 ;

The number of maps is 5, the number of reducers is 2, the number of generated files is 8, and it takes 11 seconds.

Conclusion analysis

It means that the larger the map is read, the faster the time (not necessarily. If you set the map to 1k...), the fewer files are generated.

Source code analysis

I saw the blogger above wrote about a parameter tez.grouping.split-count, but I couldn’t find it and had to look for the source code. I find

My brother didn't do it well either.

https://github.com/apache/tez/blob/master/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/grouper/TezSplitGrouper.java#L187

set tez.grouping.by-length=true The default is true

set tez.grouping.by-count=false The default is false

set tez.grouping.max-size=1024*1024*1024L --This is defined by java. You can write it yourself.

set tez.grouping.min-size=50*1024*1024

//originalSplits 数据文件分成了多少个切片
//之前算出来的切片数

 public List<GroupedSplitContainer> getGroupedSplits(Configuration conf,
                                                      List<SplitContainer> originalSplits, 
                                                      int desiredNumSplits,
                                                      String wrappedInputFormatName,
                                                      SplitSizeEstimatorWrapper estimator,
                                                      SplitLocationProviderWrapper locationProvider) throws
      IOException, InterruptedException {
    LOG.info("Grouping splits in Tez");
    Objects.requireNonNull(originalSplits, "Splits must be specified");
    //这里获取设置的参数tez.grouping.by-count
    int configNumSplits = conf.getInt(TEZ_GROUPING_SPLIT_COUNT, 0);
    if (configNumSplits > 0) {
      // always use config override if specified
      //desiredNumSplits 是tez算大概要多少  我们设置了始终以我们的为准
      desiredNumSplits = configNumSplits;
      LOG.info("Desired numSplits overridden by config to: " + desiredNumSplits);
    }

    if (estimator == null) {
      estimator = DEFAULT_SPLIT_ESTIMATOR;
    }
    if (locationProvider == null) {
      locationProvider = DEFAULT_SPLIT_LOCATION_PROVIDER;
    }

    List<GroupedSplitContainer> groupedSplits = null;
    String emptyLocation = "EmptyLocation";
    String localhost = "localhost";
    String[] emptyLocations = {emptyLocation};
    groupedSplits = new ArrayList<GroupedSplitContainer>(desiredNumSplits);
    //看所有文件是不是都是本地，个人猜测是数据文件都是有3个节点么，这个任务比如说运行再node11，数据有可能再node12 和node11 node13
    boolean allSplitsHaveLocalhost = true;

    long totalLength = 0;
    Map<String, LocationHolder> distinctLocations = createLocationsMap(conf);
    // go through splits and add them to locations
    for (SplitContainer split : originalSplits) {
      totalLength += estimator.getEstimatedSize(split);
      String[] locations = locationProvider.getPreferredLocations(split);
      if (locations == null || locations.length == 0) {
        locations = emptyLocations;
        allSplitsHaveLocalhost = false;
      }
      //判断是不是本地。
      for (String location : locations ) {
        if (location == null) {
          location = emptyLocation;
          allSplitsHaveLocalhost = false;
        }
        if (!location.equalsIgnoreCase(localhost)) {
          allSplitsHaveLocalhost = false;
        }
        distinctLocations.put(location, null);
      }
    }
    //如果我们配置了group_count 并且文件切片数量>0
    //或者我们没有配置group_count 并且文件数==0 就走if  肯定是上面的情况
    if (! (configNumSplits > 0 ||
        originalSplits.size() == 0)) {
      // numSplits has not been overridden by config
      // numSplits has been set at runtime
      // there are splits generated
      // desired splits is less than number of splits generated
      // Do sanity checks
      //desiredNumSplits已经等于我们配置的数量了，
      int splitCount = desiredNumSplits>0?desiredNumSplits:originalSplits.size();
      //获取文件总大小 320M 337336647/ 3 =112,445,549
      long lengthPerGroup = totalLength/splitCount;
      //获取我们配置的group最大size
      long maxLengthPerGroup = conf.getLong(
          TEZ_GROUPING_SPLIT_MAX_SIZE,
          TEZ_GROUPING_SPLIT_MAX_SIZE_DEFAULT);
      //获取我们配置的group最小size
      long minLengthPerGroup = conf.getLong(
          TEZ_GROUPING_SPLIT_MIN_SIZE,
          TEZ_GROUPING_SPLIT_MIN_SIZE_DEFAULT);
      if (maxLengthPerGroup < minLengthPerGroup ||
          minLengthPerGroup <=0) {
        throw new TezUncheckedException(
            "Invalid max/min group lengths. Required min>0, max>=min. " +
                " max: " + maxLengthPerGroup + " min: " + minLengthPerGroup);
      }
      //如果我们配置的group count 不合理？ 比如100G的文件 你配置了1个count 此时1个group100G 属于 >128M或者这里1G
      if (lengthPerGroup > maxLengthPerGroup) {
        //切片太大了。
        // splits too big to work. Need to override with max size.
        //就按照总大小/max +1来 因为没除尽所以+1 也就是按最大的来 
       
        int newDesiredNumSplits = (int)(totalLength/maxLengthPerGroup) + 1;
        LOG.info("Desired splits: " + desiredNumSplits + " too small. " +
            " Desired splitLength: " + lengthPerGroup +
            " Max splitLength: " + maxLengthPerGroup +
            " New desired splits: " + newDesiredNumSplits +
            " Total length: " + totalLength +
            " Original splits: " + originalSplits.size());

        desiredNumSplits = newDesiredNumSplits;
      } else if (lengthPerGroup < minLengthPerGroup) {
        // splits too small to work. Need to override with size.
        int newDesiredNumSplits = (int)(totalLength/minLengthPerGroup) + 1;
        /**
         * This is a workaround for systems like S3 that pass the same
         * fake hostname for all splits.
         */
        if (!allSplitsHaveLocalhost) {
          desiredNumSplits = newDesiredNumSplits;
        }

        LOG.info("Desired splits: " + desiredNumSplits + " too large. " +
            " Desired splitLength: " + lengthPerGroup +
            " Min splitLength: " + minLengthPerGroup +
            " New desired splits: " + newDesiredNumSplits +
            " Final desired splits: " + desiredNumSplits +
            " All splits have localhost: " + allSplitsHaveLocalhost +
            " Total length: " + totalLength +
            " Original splits: " + originalSplits.size());
      }
    }

    if (desiredNumSplits == 0 ||
        originalSplits.size() == 0 ||
        desiredNumSplits >= originalSplits.size()) {
      // nothing set. so return all the splits as is
      LOG.info("Using original number of splits: " + originalSplits.size() +
          " desired splits: " + desiredNumSplits);
      groupedSplits = new ArrayList<GroupedSplitContainer>(originalSplits.size());
      for (SplitContainer split : originalSplits) {
        GroupedSplitContainer newSplit =
            new GroupedSplitContainer(1, wrappedInputFormatName, cleanupLocations(locationProvider.getPreferredLocations(split)),
                null);
        newSplit.addSplit(split);
        groupedSplits.add(newSplit);
      }
      return groupedSplits;
    }
//总大小处于切片数 by-length
    long lengthPerGroup = totalLength/desiredNumSplits;
//数据所在的节点数 
    int numNodeLocations = distinctLocations.size();
//每个节点含有的切片数 by-node
    int numSplitsPerLocation = originalSplits.size()/numNodeLocations;
//每个group含有的切片数
    int numSplitsInGroup = originalSplits.size()/desiredNumSplits;

    // allocation loop here so that we have a good initial size for the lists
    for (String location : distinctLocations.keySet()) {
      distinctLocations.put(location, new LocationHolder(numSplitsPerLocation+1));
    }

    Set<String> locSet = new HashSet<String>();
//对所有切片开始遍历
    for (SplitContainer split : originalSplits) {
      locSet.clear();
      String[] locations = locationProvider.getPreferredLocations(split);
      if (locations == null || locations.length == 0) {
        locations = emptyLocations;
      }
      for (String location : locations) {
        if (location == null) {
          location = emptyLocation;
        }
        locSet.add(location);
      }
      for (String location : locSet) {
        LocationHolder holder = distinctLocations.get(location);
        holder.splits.add(split);
      }
    }
//按大小划分group 默认true
    boolean groupByLength = conf.getBoolean(
        TEZ_GROUPING_SPLIT_BY_LENGTH,
        TEZ_GROUPING_SPLIT_BY_LENGTH_DEFAULT);
//按指定count划分group
    boolean groupByCount = conf.getBoolean(
        TEZ_GROUPING_SPLIT_BY_COUNT,
        TEZ_GROUPING_SPLIT_BY_COUNT_DEFAULT);
//按照节点划分group
    boolean nodeLocalOnly = conf.getBoolean(
        TEZ_GROUPING_NODE_LOCAL_ONLY,
        TEZ_GROUPING_NODE_LOCAL_ONLY_DEFAULT);
    if (!(groupByLength || groupByCount)) {
      throw new TezUncheckedException(
          "None of the grouping parameters are true: "
              + TEZ_GROUPING_SPLIT_BY_LENGTH + ", "
              + TEZ_GROUPING_SPLIT_BY_COUNT);
    }
//打印日志信息分析
    LOG.info("Desired numSplits: " + desiredNumSplits +
        " lengthPerGroup: " + lengthPerGroup +
        " numLocations: " + numNodeLocations +
        " numSplitsPerLocation: " + numSplitsPerLocation +
        " numSplitsInGroup: " + numSplitsInGroup +
        " totalLength: " + totalLength +
        " numOriginalSplits: " + originalSplits.size() +
        " . Grouping by length: " + groupByLength +
        " count: " + groupByCount +
        " nodeLocalOnly: " + nodeLocalOnly);

    // go through locations and group splits
//处理到第几个切片了
    int splitsProcessed = 0;
    List<SplitContainer> group = new ArrayList<SplitContainer>(numSplitsInGroup);
    Set<String> groupLocationSet = new HashSet<String>(10);
    boolean allowSmallGroups = false;
    boolean doingRackLocal = false;
    int iterations = 0;
//对每一个切片开始遍历
    while (splitsProcessed < originalSplits.size()) {
      iterations++;
      int numFullGroupsCreated = 0;
      for (Map.Entry<String, LocationHolder> entry : distinctLocations.entrySet()) {
        group.clear();
        groupLocationSet.clear();
        String location = entry.getKey();
        LocationHolder holder = entry.getValue();
        SplitContainer splitContainer = holder.getUnprocessedHeadSplit();
        if (splitContainer == null) {
          // all splits on node processed
          continue;
        }
        int oldHeadIndex = holder.headIndex;
        long groupLength = 0;
        int groupNumSplits = 0;
        do {
//这个group
          group.add(splitContainer);

          groupLength += estimator.getEstimatedSize(splitContainer);
          groupNumSplits++;
          holder.incrementHeadIndex();
          splitContainer = holder.getUnprocessedHeadSplit();
        } while(splitContainer != null
            && (!groupByLength ||
            (groupLength + estimator.getEstimatedSize(splitContainer) <= lengthPerGroup))
            && (!groupByCount ||
            (groupNumSplits + 1 <= numSplitsInGroup)));

        if (holder.isEmpty()
            && !allowSmallGroups
            && (!groupByLength || groupLength < lengthPerGroup/2)
            && (!groupByCount || groupNumSplits < numSplitsInGroup/2)) {
          // group too small, reset it
          holder.headIndex = oldHeadIndex;
          continue;
        }

        numFullGroupsCreated++;

        // One split group created
        String[] groupLocation = {location};
        if (location == emptyLocation) {
          groupLocation = null;
        } else if (doingRackLocal) {
          for (SplitContainer splitH : group) {
            String[] locations = locationProvider.getPreferredLocations(splitH);
            if (locations != null) {
              for (String loc : locations) {
                if (loc != null) {
                  groupLocationSet.add(loc);
                }
              }
            }
          }
          groupLocation = groupLocationSet.toArray(groupLocation);
        }
        GroupedSplitContainer groupedSplit =
            new GroupedSplitContainer(group.size(), wrappedInputFormatName,
                groupLocation,
                // pass rack local hint directly to AM
                ((doingRackLocal && location != emptyLocation)?location:null));
        for (SplitContainer groupedSplitContainer : group) {
          groupedSplit.addSplit(groupedSplitContainer);
          Preconditions.checkState(groupedSplitContainer.isProcessed() == false,
              "Duplicates in grouping at location: " + location);
          groupedSplitContainer.setIsProcessed(true);
          splitsProcessed++;
        }
        if (LOG.isDebugEnabled()) {
          LOG.debug("Grouped " + group.size()
              + " length: " + groupedSplit.getLength()
              + " split at: " + location);
        }
        groupedSplits.add(groupedSplit);
      }

      if (!doingRackLocal && numFullGroupsCreated < 1) {
        // no node could create a regular node-local group.

        // Allow small groups if that is configured.
        if (nodeLocalOnly && !allowSmallGroups) {
          LOG.info(
              "Allowing small groups early after attempting to create full groups at iteration: {}, groupsCreatedSoFar={}",
              iterations, groupedSplits.size());
          allowSmallGroups = true;
          continue;
        }

        // else go rack-local
        doingRackLocal = true;
        // re-create locations
        int numRemainingSplits = originalSplits.size() - splitsProcessed;
        Set<SplitContainer> remainingSplits = new HashSet<SplitContainer>(numRemainingSplits);
        // gather remaining splits.
        for (Map.Entry<String, LocationHolder> entry : distinctLocations.entrySet()) {
          LocationHolder locHolder = entry.getValue();
          while (!locHolder.isEmpty()) {
            SplitContainer splitHolder = locHolder.getUnprocessedHeadSplit();
            if (splitHolder != null) {
              remainingSplits.add(splitHolder);
              locHolder.incrementHeadIndex();
            }
          }
        }
        if (remainingSplits.size() != numRemainingSplits) {
          throw new TezUncheckedException("Expected: " + numRemainingSplits
              + " got: " + remainingSplits.size());
        }

        // doing all this now instead of up front because the number of remaining
        // splits is expected to be much smaller
        RackResolver.init(conf);
        Map<String, String> locToRackMap = new HashMap<String, String>(distinctLocations.size());
        Map<String, LocationHolder> rackLocations = createLocationsMap(conf);
        for (String location : distinctLocations.keySet()) {
          String rack = emptyLocation;
          if (location != emptyLocation) {
            rack = RackResolver.resolve(location).getNetworkLocation();
          }
          locToRackMap.put(location, rack);
          if (rackLocations.get(rack) == null) {
            // splits will probably be located in all racks
            rackLocations.put(rack, new LocationHolder(numRemainingSplits));
          }
        }
        distinctLocations.clear();
        HashSet<String> rackSet = new HashSet<String>(rackLocations.size());
        int numRackSplitsToGroup = remainingSplits.size();
        for (SplitContainer split : originalSplits) {
          if (numRackSplitsToGroup == 0) {
            break;
          }
          // Iterate through the original splits in their order and consider them for grouping.
          // This maintains the original ordering in the list and thus subsequent grouping will
          // maintain that order
          if (!remainingSplits.contains(split)) {
            continue;
          }
          numRackSplitsToGroup--;
          rackSet.clear();
          String[] locations = locationProvider.getPreferredLocations(split);
          if (locations == null || locations.length == 0) {
            locations = emptyLocations;
          }
          for (String location : locations ) {
            if (location == null) {
              location = emptyLocation;
            }
            rackSet.add(locToRackMap.get(location));
          }
          for (String rack : rackSet) {
            rackLocations.get(rack).splits.add(split);
          }
        }

        remainingSplits.clear();
        distinctLocations = rackLocations;
        // adjust split length to be smaller because the data is non local
        float rackSplitReduction = conf.getFloat(
            TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION,
            TEZ_GROUPING_RACK_SPLIT_SIZE_REDUCTION_DEFAULT);
        if (rackSplitReduction > 0) {
          long newLengthPerGroup = (long)(lengthPerGroup*rackSplitReduction);
          int newNumSplitsInGroup = (int) (numSplitsInGroup*rackSplitReduction);
          if (newLengthPerGroup > 0) {
            lengthPerGroup = newLengthPerGroup;
          }
          if (newNumSplitsInGroup > 0) {
            numSplitsInGroup = newNumSplitsInGroup;
          }
        }

        LOG.info("Doing rack local after iteration: " + iterations +
            " splitsProcessed: " + splitsProcessed +
            " numFullGroupsInRound: " + numFullGroupsCreated +
            " totalGroups: " + groupedSplits.size() +
            " lengthPerGroup: " + lengthPerGroup +
            " numSplitsInGroup: " + numSplitsInGroup);

        // dont do smallGroups for the first pass
        continue;
      }

      if (!allowSmallGroups && numFullGroupsCreated <= numNodeLocations/10) {
        // a few nodes have a lot of data or data is thinly spread across nodes
        // so allow small groups now
        allowSmallGroups = true;
        LOG.info("Allowing small groups after iteration: " + iterations +
            " splitsProcessed: " + splitsProcessed +
            " numFullGroupsInRound: " + numFullGroupsCreated +
            " totalGroups: " + groupedSplits.size());
      }

      if (LOG.isDebugEnabled()) {
        LOG.debug("Iteration: " + iterations +
            " splitsProcessed: " + splitsProcessed +
            " numFullGroupsInRound: " + numFullGroupsCreated +
            " totalGroups: " + groupedSplits.size());
      }
    }
    LOG.info("Number of splits desired: " + desiredNumSplits +
        " created: " + groupedSplits.size() +
        " splitsProcessed: " + splitsProcessed);
    return groupedSplits;
  }

set tez.grouping.by-length=true The default is true

set tez.grouping.by-count=false The default is false

set tez.grouping.node.local.only defaults to false

set tez.grouping.max-size=1024*1024*1024L=1G --This is defined by java. You can write it yourself.

set tez.grouping.min-size=50*1024*1024=50M

set tez.grouping.split-count=0

Take mine as an example

max=128M min=16M count=10 by-count=true by-length=true split-count=10

The total file size is 320M

I don't care about the previous logic. Anyway, 21 slices are read with min=16M and divided into 21 groups.

If split-count=10 320M/10=32M 32M between min and max, the final number is 10

If split-count=2 320M/2=160M is not between min and max, so 320m/max=3 3+1=4

The back part is too long to read.

Anyway, we need to make another adjustment according to tez.grouping.rack-split-reduction=0.75f. .

In short, this parameter is useful. By-count is somewhat useful.

Test parameter tez.grouping.split-count