现有集群环境；8台region server、1台master server
早晨重启某一region server后，该server的region全部下线未恢复。
重启前
hbase(main):031:0> status 'simple' 7 live servers ec2d-cntprocdn-03.mypna.com:60020 1556378718902 requestsPerSecond=0, numberOfOnlineRegions=5, usedHeapMB=737, maxHeapMB=4480 ec2d-cntprocdn-05.mypna.com:60020 1556378795513 requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=960, maxHeapMB=4480 ec2d-cntprocdn-04.mypna.com:60020 1556378718227 requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=225, maxHeapMB=4480 ec2d-cntprocdn-06.mypna.com:60020 1556378767123 requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1013, maxHeapMB=4480 ec2d-cntprocdn-01.mypna.com:60020 1556379153372 requestsPerSecond=0, numberOfOnlineRegions=394, usedHeapMB=1136, maxHeapMB=4480 ec2d-cntprocdn-02.mypna.com:60020 1556379080369 requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1751, maxHeapMB=4480 ec2d-cntprocdn-07.mypna.com:60020 1556587728098 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=143, maxHeapMB=4480 0 dead servers Aggregate load: 0, regions: 442
将ec2d-cntprocdn-01节点重启后发现该节点热region server并未如期上线
hbase(main):009:0> status 'simple' 7 live servers ec2d-cntprocdn-03.mypna.com:60020 1556378718902 requestsPerSecond=9, numberOfOnlineRegions=5, usedHeapMB=762, maxHeapMB=4480 ec2d-cntprocdn-05.mypna.com:60020 1556378795513 requestsPerSecond=0, numberOfOnlineRegions=9, usedHeapMB=981, maxHeapMB=4480 ec2d-cntprocdn-04.mypna.com:60020 1556378718227 requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=244, maxHeapMB=4480 ec2d-cntprocdn-06.mypna.com:60020 1556378767123 requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=1031, maxHeapMB=4480 ec2d-cntprocdn-01.mypna.com:60020 1556588642407 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=126, maxHeapMB=4480 ec2d-cntprocdn-02.mypna.com:60020 1556379080369 requestsPerSecond=0, numberOfOnlineRegions=11, usedHeapMB=1771, maxHeapMB=4480 ec2d-cntprocdn-07.mypna.com:60020 1556587728098 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=160, maxHeapMB=4480 0 dead servers Aggregate load: 9, regions: 48
通过 status 'detailed'查看详细region信息
hbase(main):032:0> status 'detailed' version 0.92.1 3 regionsInTransition .META.,,1.1028785192 state=OPENING, ts=1556588320882, server=null vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. state=PENDING_OPEN, ts=1556588026138, server=null vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OPENING, ts=1556587702216, server=null master coprocessors: []
发现元数据表'.META.' region无法正常上线
于是常识重启集群后再次查看状态，发现所有server的region下线
hbase(main):025:0> status 'simple' 7 live servers ec2d-cntprocdn-07.mypna.com:60020 1556589413648 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480 ec2d-cntprocdn-06.mypna.com:60020 1556589532168 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480 ec2d-cntprocdn-05.mypna.com:60020 1556589462880 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480 ec2d-cntprocdn-02.mypna.com:60020 1556589822614 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480 ec2d-cntprocdn-03.mypna.com:60020 1556589365893 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=131, maxHeapMB=4480 ec2d-cntprocdn-01.mypna.com:60020 1556589914551 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=130, maxHeapMB=4480 ec2d-cntprocdn-04.mypna.com:60020 1556589364969 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=138, maxHeapMB=4480 0 dead servers Aggregate load: 0, regions: 0
查看表内容直接报“-ROOT-”表下线
base(main):034:0> scan 'stats_validation' ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) hbase(main):035:0> scan '.META.' ROW COLUMN+CELL ERROR: org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1769) at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
执行hbase hbck，发现也是-ROOT-,,0 offline
$ hbase hbck Exception in thread "main" org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=10, exceptions: Mon Apr 29 21:50:04 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Mon Apr 29 21:50:05 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Mon Apr 29 21:50:06 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Mon Apr 29 21:50:07 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Mon Apr 29 21:50:09 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Mon Apr 29 21:50:11 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Mon Apr 29 21:50:15 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Mon Apr 29 21:50:19 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Mon Apr 29 21:50:27 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) Mon Apr 29 21:50:43 EDT 2019, org.apache.hadoop.hbase.client.ScannerCallable@5d3e754f, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region is not online: -ROOT-,,0 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2833) at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2060) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:200) at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:52) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:130) at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:127) at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:360) at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:127) at org.apache.hadoop.hbase.util.HBaseFsck.getMetaEntries(HBaseFsck.java:1202) at org.apache.hadoop.hbase.util.HBaseFsck.doWork(HBaseFsck.java:169) at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:1727)
其中'.META.'和'-ROOT-'表的原理和关系可以参考
https://songlee24.github.io/2015/08/15/root-and-meta-table-structure/
修复过程
停止hbase集群
$ stop-hbase.sh
执行hbase元数据修复，报错可以先不用管
$ hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair RROR: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry. 19/04/29 22:18:44 ERROR util.HBaseFsck: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry. java.io.FileNotFoundException: File does not exist: /hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea/.regioninfo at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1840) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java:1831) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:578) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:154) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427) at org.apache.hadoop.hbase.util.HBaseFsck.loadMetaEntry(HBaseFsck.java:274) at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:303) at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:423) at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90) 19/04/29 22:18:49 ERROR util.HBaseFsck: Bailed out due to: org.apache.hadoop.hbase.util.HBaseFsck$RegionInfoLoadException: Unable to load region info for table vendor_source_rf_10066! It may be an invalid format or version file. You may want to remove hdfs://ec2d-cntprocnn-01.mypna.com:9000/hbase/vendor_source_rf_10066/a702abacb95649effb1e796b76d963ea region from hdfs and retry. at org.apache.hadoop.hbase.util.HBaseFsck.loadTableInfo(HBaseFsck.java:311) at org.apache.hadoop.hbase.util.HBaseFsck.rebuildMeta(HBaseFsck.java:423) at org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair.main(OfflineMetaRepair.java:90)
启动集群，查看master的日志输出
Mon Apr 29 22:23:18 EDT 2019 Starting master on ec2d-cntprocnn-01 core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 120522 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 120522 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 2019-04-29 22:23:20,723 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati on. 2019-04-29 22:23:20,791 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/master already exists and this is not a retry 2019-04-29 22:23:21,161 WARN org.apache.zookeeper.client.ZooKeeperSaslClient: SecurityException: java.lang.SecurityException: Unable to locate a login configuration occurred when trying to find JAAS configurati on. 2019-04-29 22:23:21,380 ERROR org.mortbay.log: Error starting handlers java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.format(Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple; at org.slf4j.impl.Log4jLoggerAdapter.warn(Log4jLoggerAdapter.java:443) at org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:76) at org.mortbay.log.Log.warn(Log.java:175) at org.mortbay.component.AbstractLifeCycle.setFailed(AbstractLifeCycle.java:183) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:61) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.hadoop.http.HttpServer.start(HttpServer.java:617) at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:760) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:481) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:336) at java.lang.Thread.run(Thread.java:619) 2019-04-29 22:23:27,087 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 1028785192 on ec2d-cntprocdn-02.mypna.com,60020,1556591459397 with ec2d-cntprocdn-02.mypna.com,60020,1556591459397 2019-04-29 22:23:28,118 WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 7eeecbe911ce6add0487d89ba53b7dcc has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state. 2019-04-29 22:23:28,118 WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 3fb0ccc6a9adc3f56e1f526f0d111672 has null regionLocation. But its table vendor_source_rf_10066 isn't in ENABLING state. 2019-04-29 22:23:30,684 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591010682, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:31,442 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591011439, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:31,795 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591011790, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:32,138 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591012131, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:32,447 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591012444, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:32,471 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3be02d44bcba07121d9f29494d66d959 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_poi_10066,52000100000000000001,1544522822830.3be02d44bcba07121d9f29494d66d959. state=OPEN, ts=1556591562176, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:32,754 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591012748, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:33,070 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591013067, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:33,346 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591013340, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:34,575 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region e29ae55c52ac2d6644a117fdb0e9af19 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,04000100000000000001,1544528708735.e29ae55c52ac2d6644a117fdb0e9af19. state=OPEN, ts=1556591014574, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:34,592 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region c0dbd03ee61162038fc46f7c9f964585 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,06003300000715802137,1550466407903.c0dbd03ee61162038fc46f7c9f964585. state=OPEN, ts=1556591564305, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:34,602 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 96f2f63ed9de87542a8f643704bb0934 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,07003300000751330615,1550466821719.96f2f63ed9de87542a8f643704bb0934. state=OPEN, ts=1556591564386, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:34,849 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region aa2892d5c917027afcc68c7261879765 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,08003300001000327984,1544543791636.aa2892d5c917027afcc68c7261879765. state=OPEN, ts=1556591564524, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:34,857 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1e32343c54316020cb96f24bf6ce5cbe from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,09003300000715160412,1544536989039.1e32343c54316020cb96f24bf6ce5cbe. state=OPEN, ts=1556591564612, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:35,662 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591015659, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:36,076 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 5f0e9542c55245ef429557447a03883e from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,18003300001074178697,1544542488834.5f0e9542c55245ef429557447a03883e. state=OPEN, ts=1556591565768, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:36,426 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591016424, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:36,444 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 3105e4d146648aff31262166669cd170 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,22003300001092821096,1550456353523.3105e4d146648aff31262166669cd170. state=OPEN, ts=1556591566201, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:36,617 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region b65d05bdbcf0938e281a0df1667d2ed5 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,24003300000900883934,1550466042819.b65d05bdbcf0938e281a0df1667d2ed5. state=OPEN, ts=1556591566353, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:36,792 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591016789, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:37,605 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591017604, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:37,755 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 230fef99d67d51aa66a0e7234e7b0766 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,37003300001003663497,1550466126727.230fef99d67d51aa66a0e7234e7b0766. state=OPEN, ts=1556591567559, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:38,013 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591018011, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:38,669 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591018667, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:38,787 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591018785, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:39,238 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019236, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:39,341 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019339, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:39,446 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019444, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:39,626 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019625, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:39,727 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591019724, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,095 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020094, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,259 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020257, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,345 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020341, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,418 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020417, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,508 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020502, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,588 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020585, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,674 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but re gion was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020671, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,834 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020832, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,911 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020909, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:40,986 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591020983, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:41,054 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021051, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:41,121 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021119, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:41,254 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 00c4a8cd6ed22995dadc8d6b7e901175 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,72003300001083634933,1544542493454.00c4a8cd6ed22995dadc8d6b7e901175. state=OPEN, ts=1556591571048, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:41,312 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021310, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:41,430 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021427, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:41,483 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021482, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:41,485 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region bd7401a936fb21c910f662b02e254190 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,75003300001027972012,1544537176586.bd7401a936fb21c910f662b02e254190. state=OPEN, ts=1556591571309, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:41,538 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 30711b0a58809200c20aa18d545aa94d from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,76000100000000000001,1544528177250.30711b0a58809200c20aa18d545aa94d. state=OPEN, ts=1556591571346, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:41,592 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021588, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:41,646 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591021645, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:42,015 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 1d4459d84c02de3b0f194f194a183569 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,81003300000720160733,1544537024260.1d4459d84c02de3b0f194f194a183569. state=OPEN, ts=1556591571814, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:42,243 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 46768cc418862ae474f2190bdc7bada5 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,83003300000751674816,1550456452615.46768cc418862ae474f2190bdc7bada5. state=OPEN, ts=1556591572066, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:42,323 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 9ad69996a4668c6cf93592dddc2d2752 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,83003300001023031304,1550456452615.9ad69996a4668c6cf93592dddc2d2752. state=OPEN, ts=1556591572108, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:42,360 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591022358, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:42,363 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 2f1752fa24c76301bca2d63478b65381 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,84003300000715388172,1544531717306.2f1752fa24c76301bca2d63478b65381. state=OPEN, ts=1556591572170, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:42,367 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region dfd32ba7ea53cc76abf277e9eda8547e from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,84000100000000000001,1544528141770.dfd32ba7ea53cc76abf277e9eda8547e. state=OPEN, ts=1556591572183, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:23:42,509 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received FAILED_OPEN for region 7eeecbe911ce6add0487d89ba53b7dcc from server ec2d-cntprocdn-04.mypna.com,60020,1556591001078 but region was in the state vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. state=OFFLINE, ts=1556591022505, server=null and not in PENDING_OPEN or OPENING 2019-04-29 22:23:42,513 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received OPENED for region 965b2b3bb59e0b23cfebabd59c03da12 from server ec2d-cntprocdn-01.mypna.com,60020,1556591551521 but region was in the state vendor_source_rf_10066,86000100000000000001,1544529154135.965b2b3bb59e0b23cfebabd59c03da12. state=OPEN, ts=1556591572328, server=ec2d-cntprocdn-01.mypna.com,60020,1556591551521 and not in expected PENDING_OPEN or OPENING states 2019-04-29 22:30:03,217 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-04.mypna.com,60020,1556591001078 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN. org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1624) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1329) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1304) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1299) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:381) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:351) at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:637) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:518) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:336) at java.lang.Thread.run(Thread.java:619) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72000100000000000001,1544548417172.7eeecbe911ce6add0487d89ba53b7dcc. ,which we are already trying to OPEN. at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:2479) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2437) at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150) at $Proxy11.openRegion(Unknown Source) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:435) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1592) ... 9 more 2019-04-29 22:30:03,232 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: ec2d-cntprocdn-01.mypna.com,60020,1556591551521 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN. org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1624) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1329) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1304) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1299) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughter(ServerShutdownHandler.java:381) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.fixupDaughters(ServerShutdownHandler.java:353) at org.apache.hadoop.hbase.master.HMaster.fixupDaughters(HMaster.java:637) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:518) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:336) at java.lang.Thread.run(Thread.java:619) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:vendor_source_rf_10066,72003300000564249015,1544548417172.3fb0ccc6a9adc3f56e1f526f0d111672. ,which we are already trying to OPEN. at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:2479) at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2437) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150) at $Proxy11.openRegion(Unknown Source) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:435) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1592)
再次查看region信息
hbase(main):023:0* status 'simple' 7 live servers ec2d-cntprocdn-01.mypna.com:60020 1556591551521 requestsPerSecond=0, numberOfOnlineRegions=393, usedHeapMB=1828, maxHeapMB=4480 ec2d-cntprocdn-04.mypna.com:60020 1556591001078 requestsPerSecond=0, numberOfOnlineRegions=3, usedHeapMB=1688, maxHeapMB=4480 ec2d-cntprocdn-03.mypna.com:60020 1556591001666 requestsPerSecond=0, numberOfOnlineRegions=4, usedHeapMB=341, maxHeapMB=4480 ec2d-cntprocdn-06.mypna.com:60020 1556591062296 requestsPerSecond=0, numberOfOnlineRegions=20, usedHeapMB=387, maxHeapMB=4480 ec2d-cntprocdn-05.mypna.com:60020 1556591098649 requestsPerSecond=0, numberOfOnlineRegions=10, usedHeapMB=363, maxHeapMB=4480 ec2d-cntprocdn-02.mypna.com:60020 1556591459397 requestsPerSecond=0, numberOfOnlineRegions=12, usedHeapMB=425, maxHeapMB=4480 ec2d-cntprocdn-07.mypna.com:60020 1556591049767 requestsPerSecond=0, numberOfOnlineRegions=0, usedHeapMB=315, maxHeapMB=4480 0 dead servers Aggregate load: 0, regions: 442