HDFS (Distributed File Storage System)--Overview
Table of contents
HDFS (Distributed File Storage System)--Overview
I. Overview
- The full name is Hadoop Distributed File System, Hadoop distributed file storage system
- HDFS is designed according to Google's paper: "The Google File System"
- Itself is a distributed, scalable, reliable file system
- HDFS contains three main processes: NameNode, DataNode, and SecondaryNameNode. These three processes are generally distributed on different hosts, so it is generally customary to use the name of the process to call the node
2. Features
advantage:
- Support for very large files. Oversized files here refer to files of hundreds of M, hundreds of GB, or even several TB in size. Generally speaking, Hadoop's file system will store TB-level or PB-level data. Therefore, in enterprise applications, there may be thousands of data nodes
- Detect and quickly respond to hardware failures. In a clustered environment, hardware failure is a common problem. Since there are thousands of servers connected together, this leads to a high failure rate. Therefore failure detection and automatic recovery (heartbeat mechanism) is a design goal of the HDFS file system
- Streaming data access. The data processing scale of HDFS is relatively large, and applications need to access a large amount of data at a time. At the same time, these applications generally perform batch processing instead of user-interactive processing. Applications can access datasets as streams. The main thing is data throughput, not access speed
- Simplified consistency model. When most hdfs operates files, it needs to be written once and read many times. In HDFS, once a file is created, written, and closed, it generally does not need to be modified. Such a simple consistency model is conducive to improving throughput
- High fault tolerance. Data is automatically saved in multiple copies, and automatically restored after the copy is lost
- Can be built on cheap machines. Built on cheap machines, it is easy to increase the storage capacity of the cluster almost linearly by expanding the number of machines
shortcoming:
- No low-latency data access. For example, applications that interact with users require data to be responded within milliseconds or seconds. Since Hadoop is optimized for the throughput of massive data and sacrifices the delay of obtaining data, it is not suitable for low latency to use hadoop
- Not suitable for storing large numbers of small files. HDFS supports very large files by distributing the data on the data nodes, and the metadata of the data is stored on the name nodes. The memory size of the name node determines the number of files that can be stored in the HDFS file system. Although the current system memory is relatively large, a large number of small files will still affect the performance of the name node
- It does not support multi-user writing and modifying files. HDFS files can only be written once, and do not support modification and additional writing (version 2.0 supports appending), nor does it support modification. Only in this way can the data throughput be large
- Super-strong transactions are not supported. There is no strong support for transactions like relational databases, and the loss of a block will not affect all data because the amount of data is too large.