Hadoop集群中遇到了HDFS missing blocks的错误，处理办法如下：
- 切换到hdfs用户
su - hdfs
- 使用hadoop fsck命令检查坏的块
hadoop fsck /
输出示例如下 ，
...
/user/trafodion/.hiveStats/HIVE.DMA.DMA_ENTTYPE_STAT_ORC-1532663517555-NOSTRIPENUMROWSESTIMATE10: CORRUPT blockpool BP-175371422-172.26.130.198-1529452063491 block blk_1074702727
/user/trafodion/.hiveStats/HIVE.DMA.DMA_ENTTYPE_STAT_ORC-1532663517555-NOSTRIPENUMROWSESTIMATE10: MISSING 1 blocks of total size 2459 B.....................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
.......................................................................................Status: CORRUPT
Total size: 1314013464630 B (Total open files size: 567 B)
Total dirs: 2381988
Total files: 291887
Total symlinks: 0 (Files currently being written: 6)
Total blocks (validated): 259824 (avg. block size 5057321 B) (Total open file blocks (not validated): 5)
********************************
UNDER MIN REPL'D BLOCKS: 1 (3.8487592E-4 %)
dfs.namenode.replication.min: 1
CORRUPT FILES: 1
MISSING BLOCKS: 1
MISSING SIZE: 2459 B
CORRUPT BLOCKS: 1
********************************
Minimally replicated blocks: 259823 (99.99962 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 11432 (4.3999014 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.04398
Corrupt blocks: 1
Missing replicas: 68592 (7.980528 %)
Number of data-nodes: 4
Number of racks: 1
FSCK ended at Tue Dec 10 22:07:41 EST 2019 in 27139 milliseconds
The filesystem under path '/' is CORRUPT
- 使用hadoop fsck -delete删除坏的块
hadoop fsck -delete /user/trafodion/.hiveStats/HIVE.DMA.DMA_ENTTYPE_STAT_ORC-1532663517555-NOSTRIPENUMROWSESTIMATE10
示例输出如下，
[hdfs@datanode-1 ~]$ hadoop fsck -delete /user/trafodion/.hiveStats/HIVE.DMA.DMA_ENTTYPE_STAT_ORC-1532663517555-NOSTRIPENUMROWSESTIMATE10
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Connecting to namenode via http://namenode-1.gxgs.gov:50070/fsck?ugi=hdfs&delete=1&path=%2Fuser%2Ftrafodion%2F.hiveStats%2FHIVE.DMA.DMA_ENTTYPE_STAT_ORC-1532663517555-NOSTRIPENUMROWSESTIMATE10
FSCK started by hdfs (auth:SIMPLE) from /172.26.130.194 for path /user/trafodion/.hiveStats/HIVE.DMA.DMA_ENTTYPE_STAT_ORC-1532663517555-NOSTRIPENUMROWSESTIMATE10 at Tue Dec 10 22:10:37 EST 2019
.
/user/trafodion/.hiveStats/HIVE.DMA.DMA_ENTTYPE_STAT_ORC-1532663517555-NOSTRIPENUMROWSESTIMATE10: CORRUPT blockpool BP-175371422-172.26.130.198-1529452063491 block blk_1074702727
/user/trafodion/.hiveStats/HIVE.DMA.DMA_ENTTYPE_STAT_ORC-1532663517555-NOSTRIPENUMROWSESTIMATE10: MISSING 1 blocks of total size 2459 B.Status: CORRUPT
Total size: 2459 B
Total dirs: 0
Total files: 1
Total symlinks: 0
Total blocks (validated): 1 (avg. block size 2459 B)
********************************
UNDER MIN REPL'D BLOCKS: 1 (100.0 %)
dfs.namenode.replication.min: 1
CORRUPT FILES: 1
MISSING BLOCKS: 1
MISSING SIZE: 2459 B
CORRUPT BLOCKS: 1
********************************
Minimally replicated blocks: 0 (0.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 0.0
Corrupt blocks: 1
Missing replicas: 0
Number of data-nodes: 4
Number of racks: 1
FSCK ended at Tue Dec 10 22:10:37 EST 2019 in 56 milliseconds
The filesystem under path '/user/trafodion/.hiveStats/HIVE.DMA.DMA_ENTTYPE_STAT_ORC-1532663517555-NOSTRIPENUMROWSESTIMATE10' is CORRUPT
- 重新执行hadoop fsck /检查状态
.............................................................Status: HEALTHY
Total size: 1313724044130 B (Total open files size: 650 B)
Total dirs: 2381918
Total files: 291461
Total symlinks: 0 (Files currently being written: 16)
Total blocks (validated): 259396 (avg. block size 5064550 B) (Total open file blocks (not validated): 7)
Minimally replicated blocks: 259396 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 11432 (4.407161 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 3
Average block replication: 3.0440638
Corrupt blocks: 0
Missing replicas: 68592 (7.9924493 %)
Number of data-nodes: 4
Number of racks: 1
FSCK ended at Tue Dec 10 22:11:33 EST 2019 in 25360 milliseconds
The filesystem under path '/' is HEALTHY
