Introduction:

The HDFS ( the Hadoop Distributed File the System ) is a distributed file system in the Hadoop, with high fault tolerance, high throughput characteristics, can be deployed on low-cost hardware.

HDFS design principles

HDFS architecture

HDFS follow the master / slave architecture, a single NameNode (NN) and a plurality of DataNode (DN) consisting of:

The NameNode : responsible for the implementation of relatedoperations, such as opening, closing, renaming files and directories. It is also responsible for storing the metadata cluster, the recording position information of each file data block.文件系统命名空间
DataNode : responsible for providing read and write requests from the file system client creates an execution block, delete and other operations.

File system namespace

HDFS's hierarchy similar to most file systems (such as Linux), support the creation of directories and files, move, delete and rename files, support for user configuration and access rights, but does not support hard links and soft connection. Responsible for maintaining the file system name space to record any changes to the namespace or its properties.文件系统命名空间NameNode

Data Replication

Since Hadoop was designed to run on low-cost machines, which means that the hardware is not reliable, in order to ensure fault tolerance, HDFS provides data replication mechanism. Each of HDFS file is stored as a series of blocks , each block is composed of a plurality of copies to ensure fault tolerance, the block size and replication factor can configure itself (by default, the block size is 128M, the default is the replication factor 3).

The principle of data replication

Large HDFS instances on multiple servers are often distributed in a plurality of racks, between two servers communicate through different switch chassis. In most cases, the same network bandwidth between the server rack greater than the bandwidth between different servers in a rack. Therefore, the use of HDFS rack awareness replica placement strategy for the common case, when the replication factor is 3, HDFS placement strategy is:

Located in the writer when on, it will give priority to a copy of the written document placed in this , otherwise placed randomly on. After placing another copy on any node on another remote rack, and place the last copy on another node machine frame. This strategy can reduce write traffic between racks, thereby improving write performance.datanodedatanodedatanode

If the replication factor is greater than 3, determined randomly place the fourth and subsequent copies, while maintaining the number of copies of each rack is below the upper limit, the upper limit is usually , not allowed to be noted that the same having the same multiple copies of blocks.（复制系数 - 1）/机架数量 + 2dataNode

A copy of the selection

To minimize bandwidth consumption and read latency, the HDFS when performing a read request, the priority is read from the latest copy of the reader. If there is a copy of the node in the same reading frame, the copy is preferentially selected. If HDFS cluster spans multiple data centers, the preference on the local copy of the data center.

Stability architecture

Heartbeat mechanism and recopied

Each DataNode periodically sends a heartbeat message to NameNode, if more than the specified time has not received a heartbeat message, the DataNode marked dead. NameNode does not forward any new request to the IO marked DataNode death, we will not use data on these DataNode. Since the data is no longer available, may cause some blocks replication factor less than the predetermined value, the NameNode tracks these blocks, and recopied when necessary.

Data integrity

Storage device failure due to such data block is stored in DataNode damage will occur. To avoid reading data that has been damaged and cause errors, the HDFS provides data integrity checking mechanisms to ensure data integrity, as follows:

When creating an HDFS file on the client, it will be calculated for each block of the file and stored in a separate hidden files in the same HDFS namespace. When the client retrieves the content file, it verifies that the received data from each of the DataNode stored in the associated file checksum match. If the match fails, the certification data has been damaged, then the client can choose to get the other available block copy from another DataNode.校验和校验和校验和

Disk failure metadata

FsImageAnd is at the heart of HDFS data, accidental data loss can cause the whole HDFS service is unavailable. To avoid this problem, you can configure NameNode to support and multiple copies synchronized, so that or any change will cause each copy and synchronize updates.EditLogFsImageEditLogFsImageEditLogFsImageEditLog

Support Snapshot

Support snapshot copies of data stored at a particular time, when the accident damaged data, the data can be restored to a healthy state by rolling back the operation.

HDFS features

High fault tolerance

Because HDFS scheme using multiple copies of data, so some hardware damage will not result in the loss of all data.

High throughput

Key design HDFS data access support high-throughput, low-latency data access instead.

Large file support

HDFS suitable for storing large files, the size of the document should be the level of GB to TB.

Simple consistency model

HDFS is more suitable for Write Once Read Many (write-once-read-many) access model. Support will append to the end of the file, but does not support random access to data, you can not add data from a file anywhere.

Cross-platform portability

HDFS has good cross-platform portability, which allows other large data frame are calculated as the preferred embodiment persistent data storage.

Annex: storage principle diagram HDFS

Description: The following images referenced from blog: translation of classic comics to explain the principles of HDFS

Hadoop Distributed File System --- HDFS focus