Hadoop（五）—— HDFS NameNode工作机制

NN与2NN工作机制

NameNode(NN)

1、当HDFS启动时，会加载日志(edits)和镜像文件(fsImage)到内存中。

2-4、当元数据的增删改查请求进来时，NameNode会先将操作记录到日志中，之后再执行内存数据的增删改查

SecondNameNode(2NN)

1、SecondNameNode请求NameNode，请求是否需要checkPoint，checkPoint的触发条件：

定时时间到
日志文件满了

2-6、请求checkPoint，会将最近写的edits和fsImage拷贝到SecondNameNode本地，加载到内存中合并，生成fsImage.checkpoint。

7-8 将fsImage.checkpoint拷贝到NameNode，并更名为fsImage

Fs Image与Edits解析

到/usr/local/hadoop/Hadoop_tmp/dfs/name/current查看日志和fsImage

查看fsImage

hdfs oiv -p XML -i fsimage_0000000000000000131 -o fsimage.xml

得到结果

<?xml version="1.0"?>
<fsimage><NameSection>
<genstampV1>1000</genstampV1><genstampV2>1006</genstampV2><genstampV1Limit>0</genstampV1Limit><lastAllocatedBlockId>1073741829</lastAllocatedBlockId><txid>131</txid></NameSection>
<INodeSection><lastInodeId>16424</lastInodeId><inode><id>16385</id><type>DIRECTORY</type><name></name><mtime>1576421043641</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>9223372036854775807</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16386</id><type>DIRECTORY</type><name>fzj</name><mtime>1575714994678</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16387</id><type>DIRECTORY</type><name>jiagoushi</name><mtime>1575714994678</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16392</id><type>DIRECTORY</type><name>tmp</name><mtime>1576021083427</mtime><permission>fangzhijie:supergroup:rwx-wx-wx</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16393</id><type>DIRECTORY</type><name>hive</name><mtime>1576021083443</mtime><permission>fangzhijie:supergroup:rwx-wx-wx</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16394</id><type>DIRECTORY</type><name>fangzhijie</name><mtime>1576022280367</mtime><permission>fangzhijie:supergroup:rwx------</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16411</id><type>DIRECTORY</type><name>user</name><mtime>1576021954868</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16412</id><type>DIRECTORY</type><name>hive</name><mtime>1576021954868</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16413</id><type>DIRECTORY</type><name>warehouse</name><mtime>1576021954868</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16414</id><type>DIRECTORY</type><name>bigdata</name><mtime>1576021954868</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16417</id><type>DIRECTORY</type><name>test</name><mtime>1576326634124</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16418</id><type>DIRECTORY</type><name>hdfs</name><mtime>1576327378643</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16419</id><type>DIRECTORY</type><name>client</name><mtime>1576326634124</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16420</id><type>DIRECTORY</type><name>client1</name><mtime>1576327180252</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16421</id><type>DIRECTORY</type><name>client2</name><mtime>1576327378643</mtime><permission>fangzhijie:supergroup:rwxr-xr-x</permission><nsquota>-1</nsquota><dsquota>-1</dsquota></inode>
<inode><id>16422</id><type>FILE</type><name>hdfs-test123456.xml</name><replication>3</replication><mtime>1576327781065</mtime><atime>1576327780439</atime><perferredBlockSize>134217728</perferredBlockSize><permission>fangzhijie:supergroup:rw-r--r--</permission><blocks><block><id>1073741827</id><genstamp>1004</genstamp><numBytes>878</numBytes></block>
</blocks>
</inode>
<inode><id>16423</id><type>FILE</type><name>hdfs-test123.xml</name><replication>2</replication><mtime>1576418894630</mtime><atime>1576418893994</atime><perferredBlockSize>134217728</perferredBlockSize><permission>fangzhijie:supergroup:rw-r--r--</permission><blocks><block><id>1073741828</id><genstamp>1005</genstamp><numBytes>878</numBytes></block>
</blocks>
</inode>
</INodeSection>
<INodeReferenceSection></INodeReferenceSection><SnapshotSection><snapshotCounter>0</snapshotCounter></SnapshotSection>
<INodeDirectorySection><directory><parent>16385</parent><inode>16386</inode><inode>16423</inode><inode>16422</inode><inode>16417</inode><inode>16392</inode><inode>16411</inode></directory>
<directory><parent>16386</parent><inode>16387</inode></directory>
<directory><parent>16392</parent><inode>16393</inode></directory>
<directory><parent>16393</parent><inode>16394</inode></directory>
<directory><parent>16411</parent><inode>16412</inode></directory>
<directory><parent>16412</parent><inode>16413</inode></directory>
<directory><parent>16413</parent><inode>16414</inode></directory>
<directory><parent>16417</parent><inode>16418</inode></directory>
<directory><parent>16418</parent><inode>16419</inode><inode>16420</inode><inode>16421</inode></directory>
</INodeDirectorySection>
<FileUnderConstructionSection></FileUnderConstructionSection>
<SnapshotDiffSection><diff><inodeid>16385</inodeid></diff></SnapshotDiffSection>
<SecretManagerSection><currentId>0</currentId><tokenSequenceNumber>0</tokenSequenceNumber></SecretManagerSection><CacheManagerSection><nextDirectiveId>1</nextDirectiveId></CacheManagerSection>
</fsimage>

查看日志文件

hdfs oev -p XML -i edits_0000000000000000130-0000000000000000131 -o edits.xml