Health check of hdfs file system and repair data blocks
Check the health of the hdfs file system
hdfs fsck / : Check whether the data blocks of each file are damaged or lost from the root directory
[hadoop@ruozedata001 sbin]$ hdfs fsck /
Connecting to namenode via http://ruozedata002:50070/fsck?ugi=hadoop&path=%2F
FSCK started by hadoop (auth:SIMPLE) from /192.168.72.201 for path / at Wed Aug 21 00:29:22 CST 2019
Status: HEALTHY
Total size: 0 B
Total dirs: 7
Total files: 0
Total symlinks: 0
Total blocks (validated): 0
Minimally replicated blocks: 0
Over-replicated blocks: 0
Under-replicated blocks: 0
Mis-replicated blocks: 0
Default replication factor: 3
Average block replication: 0.0
Corrupt blocks: 0
Missing replicas: 0
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Wed Aug 21 00:29:22 CST 2019 in 1 milliseconds
The filesystem under path '/' is HEALTHY
Delete damaged files hdfs fsck / -delete (operate carefully)
manually repair damaged files hdfs debug
hdfs debug recoverLease -path 文件位置 -retries 重复次数
hdfs debug recoverLease -path /xxx/yyy/aa.txt -retries 10
automatic repair
hdfs will automatically repair the damaged data block, when the data block is damaged,
Before the DN node executes the directoryscan (datanode checks the inner village and disk data set block) operation, no damage will be found
The directoryscan operation check interval is 6h
dfs.datanode.directoryscan.interval:21600
Before the DN performs a blockreport to NN, the data block will not be restored; that is, the blockreport operation is performed at an interval of 6h
dfs.blockreport.intervalMsec : 21600
Finally, when NN receives the blockreport, the recovery operation will be performed
Production tends to use manual repair methods to repair damaged data blocks.