Problem Description
Fsimage found on the active namenode is April 4, and fsimage on standby namenode is April 21. Description not synchronized in time fsimage, worried subsequent handover may be a problem.
Locate
Depending on the configuration, will be synchronized FsImage at least once every hour, to see two namenode log records related:
Standby namenode:
2019-04-21 10:04:29,853 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Uploaded image with txid 1260456917 to namenode at node05:50070 in 27.421 seconds
Active namanode:
2019-04-21 10:04:28,728 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000001260456917 size 1109646913 bytes.
Log no errors, your synchronization mechanism fsimage work properly.
View configuration hdfs-site.xml configured dfs.namenode.name.dir found two configuration directory: / disk1 / hadoop / namenode_dir, / disk2 / hadoop / namenode_dir
Check these two directories were found disk1 directory is not synchronized, and the directory of the disk2 is normal synchronized.
Continue to pursue active namenode log and found that many operations against disk2, but for a disk1 are not. Theoretically, the operation of two directories exactly is normal.
Upon inquiry related to IT staff, April 4 disk1 disk filled the case happened!
Active namenode found directory is not available, you should print the error log, but the time is long past, when the log has been washed away.
Follow-up
namenode after the discovery directory is unavailable, the back will not try to read and write the directory is available for the recovery.
Attention to the role dfs.namenode.name.dir.restore parameters to be studied.