Generally, the hadoop cluster will be in safe mode immediately after restarting. The master node will automatically cancel the safe mode after checking that the data of each slave node is normal. You can use the command "hadoop fsck path" to check whether the data is lost, for example:
hadoop fskc / Check the health status of all files in the root directory, whether there are damaged data blocks
If your cluster is normal before restarting and no data is lost, and after the restarting, you find that there are a lot of damaged blocks in the directory. In this case, a node is usually not started successfully, or it is built using cloudera manager. The hadoop cluster configuration document has been modified, and the blogger has encountered the following situations:
After the hadoop restart, it has been in safe mode. After checking, all nodes are started normally. Using the hadoop fskc / command, it is found that there are a large number of damaged blocks. The damaged blocks account for 30% of the total blocks in the cluster. After checking the cluster, it is found that all nodes The mounted data directory is set to the default directory, for example:
How fast is the disk of host A, which is mounted in multiple directories /data1/dfs/dn, /data2/dfs/dn/data3/dfs/dn, these three disks are used to save cluster data
Host B has only one disk, which is mounted at /data1/dfs/dn
After the cluster is restarted, the data directories of all nodes are reset to /data1/dfs/dn. In this case, the host A/data2/dfs/dn/, /data3/dfs/dn/ directory data is not read by the cluster, so Data loss will occur. After re-modifying the configuration file and restarting the cluster, the data is read normally!
Therefore, all big data engineers try to be cautious when restarting the cluster, and beware of data loss caused by wrong operations!
Pure hand knocking, thank you for your likes and comments!