两者有什么区别呢?
两者的区别在于:Minor compact只是进行文件merge操作,而Major compact除了做文件Merge操作,还会将其中的delete项删除。
引用
There are two types of compactions: minor and major. Minor compactions will usually pick up a couple of the smaller adjacent StoreFiles and rewrite them as one. Minors do not drop deletes or expired cells, only major compactions do this. Sometimes a minor compaction will pick up all the StoreFiles in the Store and in this case it actually promotes itself to being a major compaction.
After a major compaction runs there will be a single StoreFile per Store, and this will help performance usually. Caution: major compactions rewrite all of the Stores data and on a loaded system, this may not be tenable; major compactions will usually have to be done manually on large systems.
After a major compaction runs there will be a single StoreFile per Store, and this will help performance usually. Caution: major compactions rewrite all of the Stores data and on a loaded system, this may not be tenable; major compactions will usually have to be done manually on large systems.
知道了两者的区别,那么两者分别在什么条件下执行呢?
1. 当isMajorCompaction判断时,若当前时间减去一个时间,这个时间是能过getNextMajorCompactTime计算的,是一个24小时左右的随机数,这是为了避免在重启动时 major compaction storm。
// TODO: Use better method for determining stamp of last major (HBASE-2990) long lowTimestamp = getLowestTimestamp(filesToCompact); long now = System.currentTimeMillis(); if (lowTimestamp > 0l && lowTimestamp < (now - mcTime)) { // Major compaction time has elapsed. if (filesToCompact.size() == 1) { // Single file StoreFile sf = filesToCompact.get(0); long oldest = (sf.getReader().timeRangeTracker == null) ? Long.MIN_VALUE : now - sf.getReader().timeRangeTracker.minimumTimestamp; if (sf.isMajorCompaction() && (this.ttl == HConstants.FOREVER || oldest < this.ttl)) { if (LOG.isDebugEnabled()) { LOG.debug("Skipping major compaction of " + this.storeNameStr + " because one (major) compacted file only and oldestTime " + oldest + "ms is < ttl=" + this.ttl); } } else if (this.ttl != HConstants.FOREVER && oldest > this.ttl) { LOG.debug("Major compaction triggered on store " + this.storeNameStr + ", because keyvalues outdated; time since last major compaction " + (now - lowTimestamp) + "ms"); result = true; } } else { if (LOG.isDebugEnabled()) { LOG.debug("Major compaction triggered on store " + this.storeNameStr + "; time since last major compaction " + (now - lowTimestamp) + "ms"); } result = true; } } return result; }
long getNextMajorCompactTime() { // default = 24hrs long ret = conf.getLong(HConstants.MAJOR_COMPACTION_PERIOD, 1000*60*60*24); if (family.getValue(HConstants.MAJOR_COMPACTION_PERIOD) != null) { String strCompactionTime = family.getValue(HConstants.MAJOR_COMPACTION_PERIOD); ret = (new Long(strCompactionTime)).longValue(); } if (ret > 0) { // default = 20% = +/- 4.8 hrs double jitterPct = conf.getFloat("hbase.hregion.majorcompaction.jitter", 0.20F); if (jitterPct > 0) { long jitter = Math.round(ret * jitterPct); // deterministic jitter avoids a major compaction storm on restart ImmutableList<StoreFile> snapshot = storefiles; if (snapshot != null && !snapshot.isEmpty()) { String seed = snapshot.get(0).getPath().getName(); double curRand = new Random(seed.hashCode()).nextDouble(); ret += jitter - Math.round(2L * jitter * curRand); } else { ret = 0; // no storefiles == no major compaction } } } return ret; }
2. 在compact时,当选中的文件数小于这个store所有文件数*ratio(hbase.hstore.compaction.ratio)时,进行Minor compact.
当
boolean isMajor = (filesToCompact.getFilesToCompact().size() == this.storefiles.size());时,也就是当选中的文件数等于该store数时,会进行Major compact.