Hadoop: hdfs 块异常检查

在hdfs上创建了一个文件

[hadoop@hdp02 ~]$ echo "hadoop file" > hadoop01.txt
[hadoop@hdp02 ~]$ hadoop fs -mkdir /hadoop
[hadoop@hdp02 ~]$ hadoop fs -put hadoop01.txt /hadoop/
[hadoop@hdp02 ~]$ hadoop fs -ls -h /hadoop
Found 1 items
-rw-r--r--   3 hadoop supergroup         19 2020-04-05 21:02 /hadoop/hadoop01.txt

 使用fsck命令进行hdfs健康检查,发现了文件异常,配置副本为3,只有2副本

#hdfs文件健康检查
[hadoop@hdp02 ~]$ hdfs fsck /
Connecting to namenode via http://hdp01:50070/fsck?ugi=hadoop&path=%2F
FSCK started by hadoop (auth:SIMPLE) from /192.168.60.101 for path / at Sun Apr 05 22:53:38 CST 2020
.
/hadoop/hadoop01.txt:  Under replicated BP-417206835-192.168.60.101-1586085282109:blk_1073741825_1001. Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
Status: HEALTHY
 Total size:    12 B
 Total dirs:    2
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 12 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       1 (100.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     2.0
 Corrupt blocks:                0
 Missing replicas:              1 (33.333332 %)
 Number of data-nodes:          2
 Number of racks:               1
FSCK ended at Sun Apr 05 22:53:38 CST 2020 in 1 milliseconds


The filesystem under path '/' is indeed HEALTHY

异常信息显示,一个hdfs 文件块丢失

执行hdfs文件手动恢复命令

hdfs debug recoverLease -path /hadoop/hadoop01.txt -retries 10

然后再查看健康状况,执行hdfs fsck,发现信息还是和上面一样。问题出在哪?

后来想想,我的集群只配置了一台namenode和2台datanode,文件副本只会存储在datanode节点上。所以就算我的hdfs-site.xml文件配置了副本数为3,实际也最多只能存在2个副本!

没办法,需要增加一个datanode节点。编辑 $HADOOP_HOME/etc/hadoop/slaves文件,增加hdp01为新的datanode节点,然后将该文件同步到其他节点。

增加了datanode节点,集群性质发生了变化,需要重新初始化namenode,格式化需要进行一系列步骤,否则格式化无效!

如何格式初始化namenode,参见我的 hdfs datanode进程没法启动

完成格式化后,创建新的文件

再次执行hdfs fsck / 查看hdfs 文件系统状况

[hadoop@hdp02 hadoop]$ hadoop fs -put README.txt /hadoop/
[hadoop@hdp02 hadoop]$ hdfs fsck /
Connecting to namenode via http://hdp01:50070/fsck?ugi=hadoop&path=%2F
FSCK started by hadoop (auth:SIMPLE) from /192.168.60.102 for path / at Mon Apr 06 00:01:24 CST 2020
.Status: HEALTHY
 Total size:    1366 B
 Total dirs:    2
 Total files:   1
 Total symlinks:                0
 Total blocks (validated):      1 (avg. block size 1366 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Mon Apr 06 00:01:24 CST 2020 in 5 milliseconds


The filesystem under path '/' is HEALTHY

猜你喜欢

转载自blog.csdn.net/dinghua_xuexi/article/details/105335967
今日推荐