NameNode && Secondary NameNode mechanism

NameNode && Secondary NameNode mechanism

1) Workflow

 

2)  fsimage和edits

HDFS is NameNode brain, it maintains a directory tree of the entire file system, as well as a directory tree of all files and directories, these two kinds of information in a file stored in the file system: one is the mirror image namespace (also called file system image , File System Image, fsimage), ie a complete snapshot of HDFS metadata, each time NameNode start, load the latest default namespace mirror, the other is a mirror image of the space command to edit the log (edit log).

fsimage file is actually a permanent checkpoint file system metadata, but not every write operation will update this file, because FSImage is a large file, if the write operation is performed frequently, the system will run extremely slowly. NameNode only solution is to change the contents of write-ahead log, edit the log that is written to mirror the namespace. Over time, journal editors will become bigger and bigger, so the event of failure, will be very much calls time to roll back, so just like a traditional relational database, the need for consolidation fsimage and edit logs regularly. If you do merge operation by the NameNode, then NameNode when a cluster service may not provide sufficient resources, in order to solve this problem, Secondary NameNode answer for us.

3) The first stage: NameNode start

• (1) first start NameNode format, create fsimage and edits files. If this is not the first time you start, log and edit the image directly load files into memory.
• (2) the client requests the metadata of additions and deletions.
• (3) NameNode record operating logs, update rolling log.
• (4) NameNode deletions data change in memory check.

4) The second stage: Secondary NameNode work

• (1) Secondary NameNode asked whether NameNode need checkpoint. NameNode whether directly back to check the results.
• (2) Secondary NameNode request execution of checkpoint.
• (3) Secondary NameNode guide NameNode rollover editor was written edits log.
• (4) Secondary NameNode load fsimage file, edit log replay, will be merged into fsimage, written to disk after a new fsimage file compression.
• (5) Copy fsimage to NameNode.
• (6) NameNode fsimage will be renamed into fsimage.

By default, this process occurs once per hour, or when NameNode edit the log file reaches the default of 64MB will be triggered.