HDFS of the three core components of Hadoop
1. Introduction to HDFS
HDFS (Hadoop Distributed File System) is a distributed file system under Hadoop. HDFS is a highly fault-tolerant system, suitable for deployment on cheap machines . HDFS can provide high-throughput data access, which is very suitable for applications on large-scale data sets. HDFS relaxes some POSIX constraints to achieve the purpose of streaming file system data.
Two, HDFS design principle
2.1, HDFS architecture
HDFS adopts a master/slave architecture and consists of a single NameNode (NN) and multiple DataNodes (DN).
2.2. The roles and functions of HDFS
| Roles | Features |
|---|---|
| Client | 1. When uploading data, divide the data into blocks and merge the data when downloading data. 2. Communicate with the NameNode to obtain the data node when the data is uploaded when the data is uploaded, and obtain the stored data when the data is downloaded. Node 3. Communicate with DataNode to upload and download data 4. Manage and access the entire HDFS |
| NameNode (Metadata Node) | 1. Manage the metadata of the namespace (NameSpace) in HDFS 2. Process the client's read and write requests 3. Responsible for managing the mapping information of the file block |
| DataNode | 1. Data storage node, used to save and retrieve block 2. Perform data read and write operations |
| Secondary NameNode (from metadata node) | 1. Assist NN to share its workload 2. Merge the edit logs of NameNode to the fsimage file, and then copy fsimage to NameNode 3. In an emergency, it can assist in restoring NN |
2.3, HDFS copy mechanism
- What is stored in the copy mechanism?
Block: block
is the basic storage unit HDFS
default size: in Hadoop2.x release version is 64M 128M Hadoop1.x
There is dfs.blocksize in hdfs-site.xml, which stipulates that each block is 128M
<name>dfs.blocksize</name>
<value>134217728</value>
There is dfs.replication in hdfs-site.xml, which specifies the number of copies of each block
<name>dfs.replication</name>
<value>3</value>
The default number of copies is 3
- Copy storage mechanism
As shown in the figure above,
a block is on a local rack node,
a block is on a different node in the same rack, and a block is on a node in
a different rack
- Why do you need a copy mechanism?
When using a cluster, we need a large number of relatively cheap computers, so downtime is an inevitable event. We need to avoid data loss by adopting redundant data storage, and the specific implementation is the copy mechanism.
Three, the advantages and disadvantages of HDFS
advantage:
| Features | reflect |
|---|---|
| High fault tolerance | 1. Data is automatically saved in multiple copies 2. After a copy is lost, it can be automatically restored |
| Suitable for big data processing | Data scale: able to process data files with a scale of GB, TB, or even PB level : able to process files of more than one million scale, a fairly large number of nodes scale: able to handle a scale of 10K nodes |
| Streaming data access | 1. Write once, read multiple times , cannot be modified, only append 2. It can ensure data consistency |
| Can be built on cheap machines | 1. It improves reliability through multiple copies mechanism 2. It provides fault tolerance and recovery mechanism . For example, if a copy is lost, it can be restored by other copies |
Disadvantages:
| Features | reflect |
|---|---|
| Not suitable for low-latency data access | Suitable for scenarios with high throughput rates , that is, to write a large amount of data in a certain period of time. Does not support low latency , such as reading data within milliseconds |
| Not suitable for small file access scenarios | 1. A large number of small files will occupy a large amount of memory of the NameNode to store file, directory and block information . 2. The addressing time of small file storage will exceed the read time, which violates the design goal of HDFS |
| It is not suitable for the scenes of combined sending and writing, random file modification | 1. One file does not allow multiple threads to write at the same time 2. Only supports data appending, does not support random modification of files |
Four, HDFS read and write process
4.1, HDFS write file
- The client sends a write request to NN (NameNode) through the Distributed FileSystem
- NN performs resource verification, verifying whether the source file exists, whether the target path exists, whether the client has write permission, if the source file does not exist/the target path exists/client does not have write permission, the client will throw an exception, and if it succeeds, it will be a file Create a record
- The client cuts the file into blocks and requests NN to obtain the DN (DataNode) list stored in the first block (the number of copies is 3 by default)
- NN returns the list of DNs: dn1, dn2, dn3
- The client requests dn1 to upload data through FSData OutputStream. After receiving the request, dn1 will continue to request dn2, then dn2 requests dn3 to establish a communication channel, then dn3 responds to dn2, dn2 responds to dn1, dn1 responds to client, and the communication channel is established.
- The client starts uploading the first block to dn1, and uploads it in units of packet (64kb) each time. Each time dn1 transmits a packet, a response queue will be established to wait for a response, and then it will be sent to dn2 and dn3 in turn. dn3, dn2, and dn1 respond step by step, and return the result to the client, and then start the transmission of the next packet until the upload of the first block is completed
- Close FSData OutputStream
- The client sends complete to NN
- Reread steps 3-8 until the file upload is complete
4.2, HDFS read files
- The client creates a DFS (Distributed FileSystem) by calling the FileSystem.open() method
- DFS requests NN to obtain an ordered list of some or all blocks through RPC (Remote Procedure Call)
• The list will be sorted in ascending order according to the distance between the block and the client.
• Each block contains the address of the DN. If the client is the DN, it will be read locally
- NN will return an input stream object FSDIS (FSDataInputStream) that supports file positioning
- The client calls the read() method of FSDIS to find the DataNode closest to the client and establish a connection with it
- The client reads the data in the DataNode through the IO stream. When the first block is read, it will close the connection with the DN and continue to connect to the next closest DN until the block of this batch is read.
- Repeat steps 2-5 until all blocks have been read
- Close all streams
Five, HDFS Shell command
Basic format
hdfs dfs -cmd
View available commands
hdfs dfs
Create a directory to store data files
hdfs dfs -mkdir -p /hdfs/shell
hdfs dfs -ls /hdfs/shell
Upload data to HDFS and view
hdfs dfs -put /home/test.txt /hdfs/shell
hdfs dfs -text /hdfs/shell/test.txt
Download the file to the local
hdfs dfs -get /hdfs/shell/test.txt /home/
Count the file size in the directory
hdfs dfs -du /hdfs/shell
Delete and move data files and directories
hdfs dfs -rm /hdfs/shell/test.txt
hdfs dfs -rmr /hdfs
hdfs dfs -rm -r /hdfs
hdfs dfs -cp /hdfs/shell/test.txt /hdfs/
hdfs dfs -mv /hdfs/shell/test.txt /hdfs/
Modify file permissions
Change file owner
hdfs dfs -chmod
hdfs dfs -chown
Six, NameNode and SecondaryNameNode
NN and SNN working mechanism
The first stageNameNode
(1) After starting the NameNode for the first time and formatting, create fsimage and edits files. If it is not the first startup, directly load the edit log and mirror file to the memory
(2) The client requests to add, delete and modify the metadata
(3) The NameNode records the operation log and update the rolling log
(4) The NameNode performs the data in the memory Additions, deletions, modifications
second stageSecondary NameNode
(1) The Secondary NameNode asks whether the NameNode needs a checkpoint. Bring back the
result of NameNode check directly
(2) Secondary NameNode requests to execute checkpoint
(3) NameNode scrolls the edits log being written
(4) Copy the edit log and mirror file before rolling to the Secondary NameNode
(5) Secondary NameNode loads the edit log and Mirror file to memory, and merge
(6) Generate a new mirror file fsimage.chkpoint
(7) Copy fsimage.chkpoint to NameNode
(8) NameNode rename fsimage.chkpoint to fsimage