HBase BucketAllocatorException Analysis

Recently, the following WARN logs have been observed in the HBase cluster:

2020-04-18 16:17:03,081 WARN [regionserver/xxx-BucketCacheWriter-1] bucket.BucketCache:Failed allocation for 604acc82edd349ca906939af14464bcb_175674734;
org.apache.hadoop.hbase.io.hfile.bucket.BucketAllocatorException: Allocation too big size=1114202; adjust BucketCache sizes hbase.bucketcache.bucket.sizes to accomodate if size seems reasonable and you want it cached.

It probably means: BucketAllocator cannot allocate space for the block because it is too large (size = 1114202). If you want to be cached and think it is reasonable, you can adjust the parameter hbase.bucketcache.bucket.sizes.

By default, the maximum value of HBase BucketCache can cache block is 512KB, that is hbase.bucketcache.bucket.sizes = 5120,9216,17408,33792,41984,50176,58368,66560,99328,132096,197632,263168,394240, 525312, 14 size labels by default. If we want to cache a larger block, we can adjust the parameters to hbase.bucketcache.bucket.sizes = 5120,9216,17408,33792,41984,50176,58368,66560,99328,132096,197632,263168,394240,525312 , 1049600, 2098176, at this time the maximum allowable 2MB block.

Let's take a brief look at the corresponding source code, the related classes are:
/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java
BucketAllocator mainly implements the organization and management of the bucket , Allocate memory space for the block.

/**
   * Allocate a block with specified size. Return the offset
   * @param blockSize size of block
   * @throws BucketAllocatorException
   * @throws CacheFullException
   * @return the offset in the IOEngine
   */
  public synchronized long allocateBlock(int blockSize) throws CacheFullException,
      BucketAllocatorException {
    assert blockSize > 0;
    BucketSizeInfo bsi = roundUpToBucketSizeInfo(blockSize);
    if (bsi == null) {
      throw new BucketAllocatorException("Allocation too big size=" + blockSize +
        "; adjust BucketCache sizes " + BlockCacheFactory.BUCKET_CACHE_BUCKETS_KEY +
        " to accomodate if size seems reasonable and you want it cached.");
    }
    long offset = bsi.allocateBlock();

    // Ask caller to free up space and try again!
    if (offset < 0)
      throw new CacheFullException(blockSize, bsi.sizeIndex());
    usedSize += bucketSizes[bsi.sizeIndex()];
    return offset;
  }

After calling the roundUpToBucketSizeInfo () method, if the returned result is null, a BucketAllocatorException is thrown. Take a look at the roundUpToBucketSizeInfo () method:

/**
 * Round up the given block size to bucket size, and get the corresponding
 * BucketSizeInfo
*/
public BucketSizeInfo roundUpToBucketSizeInfo(int blockSize) {
	for (int i = 0; i < bucketSizes.length; ++i)
	  if (blockSize <= bucketSizes[i])
		return bucketSizeInfos[i];
	return null;
}

This method compares the incoming blockSize with the array bucketSizes starting at index 0. Once it is less than bucketSize [i], it allocates the size of bucketSizeInfos [i] to store the block.

Let's take a look at the initialization process of the array bucketSizes:

private static final int DEFAULT_BUCKET_SIZES[] = { 4 * 1024 + 1024, 8 * 1024 + 1024,
  16 * 1024 + 1024, 32 * 1024 + 1024, 40 * 1024 + 1024, 48 * 1024 + 1024,
  56 * 1024 + 1024, 64 * 1024 + 1024, 96 * 1024 + 1024, 128 * 1024 + 1024,
  192 * 1024 + 1024, 256 * 1024 + 1024, 384 * 1024 + 1024,
  512 * 1024 + 1024 };

private final int[] bucketSizes;
BucketAllocator(long availableSpace, int[] bucketSizes)
  throws BucketAllocatorException {
  this.bucketSizes = bucketSizes == null ? DEFAULT_BUCKET_SIZES : bucketSizes;
  Arrays.sort(this.bucketSizes);
  ...
}

As you can see, if bucketSizes == null, the value of DEFAULT_BUCKET_SIZES is taken by default, and the array is sorted. The ordering of this step laid the foundation for the cyclic comparison of the previous step.

The value of the array DEFAULT_BUCKET_SIZES is the default value of the parameter hbase.bucketcache.bucket.sizes.

Please indicate the source! Welcome to pay attention to my WeChat public account [HBase working notes]