hdfs
Simply put, it is a distributed file system
First, the role division of hdfs
client:分block
NameNode: is the Master node. Manage data block mapping; handle client read and write requests; configure replication policies; manage HDFS namespaces;
SecondaryNameNode: It is a younger brother, sharing the workload of the big brother namenode; it is the cold backup of NameNode; merges fsimage and fsedits and then sends it to namenode.
DataNode: Slave node, working. Responsible for storing the data block block sent by the client; perform read and write operations of the data block.
fsimage: Metadata image file (the directory tree of the file system.)
edits: operation log of metadata (record of modification operations made to the file system)
fsimage+edits analogy the data table and log table in the database
Namenode memory is stored in fsimage+edits.
The SecondaryNameNode defaults to 1 hour. From the namenode, it obtains the fsimage and edits to merge, and then sends it to the namenode. Reduce the workload of the namenode.
2. Startup process
http://blog.csdn.net/amber_amber/article/details/38268407
1 、 purpose
1 "Construction of Server members (for RCP)
2" FSNameSystem construction
Core data structures include:
l blocksMap: map, saves the mapping relationship related to the data block block, including the mapping between files and blocks, blocks and datanodes, etc.
l curruptReplicasMap: saves all corrupted blocks, all replicas are corrupted.
l datanodeMap: All datanodes that have been connected recently are the complete set.
l recentInvalidateSets: recently invalidated blocks.
l excessReplicateMap: The mapping between machines and excess blocks on the machine, which are waiting to be deleted.
l Heartbeats: The datanode that currently sends heartbeat packets, that is, the datanode of alive.
FSDirectory object: This holds the hierarchy of the file system.
Process: Initialize data structures such as blocksMap, load the fsimage file into the class FSImage, read the edits file, and use fsimage and edits to construct the file system hierarchy.
3" start the background daemon
l PendingReplicaitonBlocks$PendingReplicationMonitor: Monitor the copy process of blocks
l FSNamesystem$HeartbeatMonitor, detection of heartbeat packets
l FSNamesystem$ReplicationMonitor: Monitor the number of replicas. If it is insufficient, copy it.
l LeaseManager$Monitor: Controls the write request. The file cannot be written by multiple people at the same time.
l DecommissionManager$Monitor: handles the decommissioning of data nodes
2. Start datanode
1" Obtain and set various configuration information, organize and encapsulate your own block information
2" Obtain information such as namespaceID, version and StorageID from namenode
3" datanode registers with namenode.
Send the service address, information query port, client access port, datanodeID, etc. to the namenode, as well as block_report block data; FSNameSystem first verifies whether the datanode is allowed to mount (verified by the dfs.hosts list), and then maps the datanode data to meta In the data, here is mainly to obtain the mapping relationship between the block_report block and the datanode, add it to the BlocksMap [hx1], and then add the datanode to the heartsbeats for monitoring.
4" Start the datanode thread. Periodically send block_Report and heartbeats to the namenode, and accept the namenode's instructions.
3. Startup of Secondary namenode
Register to the namenode, and synchronize the current fsimage, edits and other file information from the namenode; start the process for rotation training on the secondary namenode, and periodically perform the checkpoint creation operation.
3. The process of write operation
http://www.cnblogs.com/laov/p/3434917.html
1. The client informs the namenode and obtains the datanodes
2. Write each block and package specifically (internal copy)
3. Write other blocks
4. Notify nn to end