Article directory
introduction
HDFS (Hadoop Distributed File System) is a file system that realizes distributed storage of files on multiple hosts through the network. Adopt "client/server" (Client/Server) mode.
1. Basic features
1.1 High fault tolerance
HDFS designs corresponding mechanisms for fast and automatic error detection and recovery. If one node fails, backups on other nodes are automatically replenished.
1.2 Large data capacity
Support hundreds of nodes to meet big data needs.
1.3 Scalability
The horizontal scalability is strong, and data nodes can be added or deleted as needed.
1.4 High Throughput
High data transfer rate, supporting high concurrent big data applications.
1.5 Nearest calculation
Customers request to complete computing tasks directly on data nodes as much as possible, so as to reduce the data transmission burden and increase throughput.
2. Architecture
2.1 NameNode
NameNode maintains the file directory tree of the entire file system, including file/directory metadata and a list of data blocks corresponding to each file, and is responsible for receiving user operation requests. Its main tasks are as follows:
(1) Manage the namespace (Namespace);
(2) Control the client to read/write files;
(3) Perform common file system operations.
Two core data structures, FsImage and EditLog, are saved. FsImage is used to maintain the metadata of the file system tree and all files and directories in the file tree ; the operation log file EditLog records all operations such as creation, deletion, and renaming of files. When the NameNode starts, it will load the content of the FsImage file into the memory, and then perform various operations in the EditLog to keep the metadata in the memory up-to-date.
2.2 DataNode
DataNode stores HDFS data in the form of files in a separate file in the local file system, not in the same directory, and does not know information about HDFS files.
3. Storage mechanism
3.1 Block
Data in HDFS is stored in the form of a file block , which is the smallest unit for each read and write. The default size of Hadoop2.0 is 128MB. You can modify the block size by configuring the parameter dfs.blocksize in hdfs-site.xml , which needs to be 2 to the kth power.
3.2 Copy management strategy
Multiple copies are used to store data redundantly, and multiple copies of a data block are stored on different DataNodes. By default, each block has three copies. The basic principle of placement is to ensure that not all copies are on the same machine . on the Rack . For example:
the first copy is on the node where the client is located, the second copy is on another node on the same rack, and the third copy is placed on a random node on another rack.
4. Data read and write process
4.1 Data reading process
①Open the file . Use the open() method of the DistributedFileSystem object to open a file on HDFS;
② Obtain data block information . Send a request to the NameNode through the RPC (Remote Procedure Call) call to obtain the location information of the file, that is, the data block number and the DataNode address;
③ read the request . The client sends a read() request to read data to FSDataInputStream;
④ read data . DFSInputStream selects the most recent data block to read and returns to the client, and closes the corresponding DataNode connection after the reading is completed;
⑤Read data . DFSInputStream selects the nearest DataNode in turn, reads and returns until the last data block is read (if a DataNode fails, it will automatically select the next DataNode containing this data block to read); ⑥Close
the file . After the client has read all the data blocks, call the close() method of FSDataInputStream to close the file;
4.2 Data writing process
① Create a file request . The client calls the create() method of the FileSystem object to create a file;
② Create file metadata . Use RPC (remote procedure call) to call the name node and create a new file in the namespace of the file system;
③ Write data . Through the FSDataOututStream object, start writing data;
④Write data packets . The data is divided into blocks by DFSOutputStream and written to the data queue. The data queue is read by Data Streamer and notifies the name node to allocate data nodes to store data blocks (three blocks are replicated by default for each block). Allocated data nodes are placed in a data flow pipeline . Data Streamer writes the data block to the first data node in the pipeline, and the first data node sends the data block to the second data node, that is, writing while storing ;
⑤ Receive confirmation packet . DFSOutputStream saves the ack queue for the sent data blocks, and waits for the data nodes in the pipeline to inform that the data has been written successfully;
⑥Close the file . After all the data is written, call close() to write all the data blocks to the data nodes in the pipeline, and wait for the ack queue to return successfully;
⑦The write operation is completed . Notify the name node that the write is complete;
5. Java API programming
//从URL读取HDFS文件
public class HDFSURLReader{
static{
//此处设置文件系统配置为URL流，仅执行一次
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
}
public static void main(String[] args){
InputStream stream = null;
String hdfssurl = "hdfs://192.168.18.130:9000/data/input.txt");
try{
//打开远程HDFS文件系统的文件
stream = new URL(hdfsurl).openStream();
//输出文件内容到标准输出（默认为屏幕）
IOUtils.copyBytes(stream, Ststem.out, 1024, false);
}catch(IOException e){
IOUtils.colseStream(stream);// 关闭文件
}
}
}
6. High reliability mechanism of HDFS
6.1 Heartbeat Mechanism
HDFS detects whether DataNode has an error through heartbeat. Each DataNode node periodically sends a heartbeat signal to the NameNode, and the NameNode confirms whether the DataNode has an error through heartbeat signal detection . I/O requests are sent to them. When the downtime causes the number of replicas to be lower than expected, the NameNode will start the replication operation.
6.2 Secondary NameNode
Secondary NameNode (2NN) is mainly used to merge the EditLog of NameNode into the FsImage file, that is, to regularly update and back up the metadata . Prevent the restart from taking a long time when the EditLog file is too large. The detailed process is as follows:
①Secondary NameNode notifies NameNode to switch EditLog files;
②Secondary NameNode downloads FsImage and EditLog from NameNode through the network;
③Secondary NameNode loads FsImage into memory, and then starts to merge EditLog logs; ④Secondary
NameNode sends back the new FsImage to NameNode;
The new FsImage replaces the old FsImage;
6.3 High availability mechanism of HDFS NameNode HA
HDFS NameNode HA (High Availability) HDFS NameNode high availability mechanism to solve the problem of NameNode single point of failure.
1) ZooKeeper cluster : provide active and standby election support for active and standby switching controllers;
2) active/standby switching controller (Active/Standby ZKFailoverController, ZKFC): detect the health status of NameNode in time, manage sessions, and use ZooKeeper automatically elects and switches between active and standby;
3) Active/Standby NameNode (Active/Standby NameNode): Active NameNode is responsible for all client operations, Standby NameNode acts as a slave and is responsible for maintaining state information for fast switching when needed; 4)
Shared storage System (JournalNode cluster): Synchronize the status of Active/Standby NameNode. Use a group of independent processes called "JournalNode" to communicate, so that the Standby NameNnode reads the EditLog from it and applies it to its own namespace to achieve state synchronization; 5) DataNode: DataNode
will report the location information of the data block to the active and standby NameNode at the same time, But only receive read and write commands from the main NameNode;
Summarize
The above is the content about HDFS, welcome to add in the comment area.
Refer to "Hadoop Big Data Principles and Applications" Xu Luhui