1 异常信息
org.apache.hadoop.hbase.exceptions.FailedSanityCheckException: Requested row out of range for doMiniBatchMutation on HRegion CM_WARE_22,82228221001000100000007175328,1501813702755.c9c1b2ef162c0075420b9ee2df1a3091., startKey='82228221001000100000007175328', getEndKey()='82229221000000000000002251629', row='82229221001000000000001862129'
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:688)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:639)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1931)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2034)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
- 初步分析，由于region分布，存在rowkey空洞，导致rowkey不连续。在页面上查看该表region分布
CM_WARE_22,82227221000000000000002042827,1499516073152.12d95142a820361927b02d8ec8cd466d. shdx095:61520 82227221000000000000002042827 82228221001000100000007175328 0.04316194 31275854
CM_WARE_22,82228221001000100000007175328,1501813702755.c9c1b2ef162c0075420b9ee2df1a3091. shdx082:61520 82228221001000100000007175328 82229221000000000000002251629 1.0 37988291
CM_WARE_22,82230,1499666283926.91f90e2b8335063d6fe9aa0b9822c9a7. shdx087:61520 82230 822312210000000000000019086 0.043608032 20661496
- 根据实际该表的region的分布情况，确实存在空洞。异常信息中，需要的写入的rowkey，应该在空洞的范围区间。
2 检查该表的信息
hbase hbck -details 'CM_WARE_22'
信息截取：
82228221001000100000007175328: [ { meta => CM_WARE_22,82228221001000100000007175328,1501813702755.c9c1b2ef162c0075420b9ee2df1a3091., hdfs => hdfs://shzxhoneycomb/hbasezy/data/default/CM_WARE_22/c9c1b2ef162c0075420b9ee2df1a3091, deployed => shdx082,61520,1516073423971;CM_WARE_22,82228221001000100000007175328,1501813702755.c9c1b2ef162c0075420b9ee2df1a3091., replicaId => 0 }, 82229221000000000000002251629]
82229221000000000000002251629:
82230: [ { meta => CM_WARE_22,82230,1499666283926.91f90e2b8335063d6fe9aa0b9822c9a7., hdfs => hdfs://shzxhoneycomb/hbasezy/data/default/CM_WARE_22/91f90e2b8335063d6fe9aa0b9822c9a7, deployed => shdx087,61520,1516073423769;CM_WARE_22,82230,1499666283926.91f90e2b8335063d6fe9aa0b9822c9a7., replicaId => 0 }, 822312210000000000000019086]
822312210000000000000019086: [ { meta => CM_WARE_22,822312210000000000000019086,1500042454234.9c48825a2fa4ada58b260ebd4eef5e26., hdfs => hdfs://shzxhoneycomb/hbasezy/data/default/CM_WARE_22/9c48825a2fa4ada58b260ebd4eef5e26, deployed => shdx051,61520,1516073621059;CM_WARE_22,822312210000000000000019086,1500042454234.9c48825a2fa4ada58b260ebd4eef5e26., replicaId => 0 }, 82231221001000200000009552631]
82231221001000200000009552631: [ { meta => CM_WARE_22,82231221001000200000009552631,1500042454234.5f90d62d83460bfd73bfa54dba54906f., hdfs => hdfs://shzxhoneycomb/hbasezy/data/default/CM_WARE_22/5f90d62d83460bfd73bfa54dba54906f, deployed => shdx052,61520,1516073423832;CM_WARE_22,82231221001000200000009552631,1500042454234.5f90d62d83460bfd73bfa54dba54906f., replicaId => 0 }, 82232221000000000001038811232]
- 详情展示里，我们可以看到，82229221000000000000002251629--82230段，缺失meta信息
- 也可以通过查询hbase:meta表，获取对应信息
scan 'hbase:meta' , {FILTER=>"PrefixFilter('CM_WARE_22')"}
3 尝试修复rowkey空洞
hbase hbck -repairHoles 'CM_WARE_22'
- 修复后的region分布情况：
CM_WARE_22,82227221000000000000002042827,1499516073152.12d95142a820361927b02d8ec8cd466d. shdx095:61520 82227221000000000000002042827 82228221001000100000007175328 0.04316194 31275854
CM_WARE_22,82228221001000100000007175328,1501813702755.c9c1b2ef162c0075420b9ee2df1a3091. shdx082:61520 82228221001000100000007175328 82229221000000000000002251629 1.0 37988291
CM_WARE_22,82228221001000100000007175328,1515550510214.ff663d437879a2c61ac9217177648059. shdx081:61520 82228221001000100000007175328 82230 0.6747074 0
CM_WARE_22,82230,1499666283926.91f90e2b8335063d6fe9aa0b9822c9a7. shdx087:61520 82230 822312210000000000000019086 0.043608032 20661496
- 执行后，region空洞得到补充。查看region分布，发现有2个region的Start Key是相同的。要解决这个问题，可以尝试将这2个region进行合并。本次处理，未进行合并，写数据时，暂未出现异常