How HDFS reads and writes files

The file reading principle of HDFS mainly includes the following steps:

• First call the open method of the FileSystem object, in fact, an instance of DistributedFileSystem is obtained.
• DistributedFileSystem obtains the locations of the first block of the file through RPC (remote procedure call). The same block will return multiple locations according to the number of repetitions. These locations are sorted according to the hadoop topology, and the ones closest to the client are listed first.
• The first two steps will return an FSDataInputStream object, which will be encapsulated into a DFSInputStream object. DFSInputStream can easily manage datanode and namenode data streams. When the client calls the read method, DFSInputStream will find the datanode closest to the client and connect to the datanode.
• Data flows continuously from the datanode to the client.
• If the data of the first block is read, the datanode connection to the first block will be closed, and then the next block will be read. These operations are transparent to the client, and from the client's point of view is just reading a continuous stream.
• If the first batch of blocks are read, DFSInputStream will go to the namenode to get the location of the next batch of blocks, and then continue to read. If all blocks are read, then all streams will be closed.

The file writing principle of HDFS mainly includes the following steps:

• The client creates a new file by calling the create method of DistributedFileSystem.
• DistributedFileSystem calls NameNode via RPC (Remote Procedure Call) to create a new file with no blocks associated. Before creation, NameNode will do various checks, such as whether the file exists, whether the client has permission to create it, etc. If the verification passes, the NameNode will record the new file, otherwise it will throw an IO exception.
• After the first two steps, the object of FSDataOutputStream will be returned. Similar to reading files, FSDataOutputStream is encapsulated into DFSOutputStream, which can coordinate NameNode and DataNode. The client starts to write data to DFSOutputStream, DFSOutputStream will cut the data into small packets, and then queue them in the data queue.
• DataStreamer will process and accept the data queue. It first asks the NameNode in which DataNodes the new block is most suitable for storage. For example, if the number of repetitions is 3, then it will find the 3 most suitable DataNodes and arrange them into a pipeline. . The DataStreamer queues the packets to the first DataNode of the pipeline, and the first DataNode outputs the packets to the second DataNode, and so on.
• DFSOutputStream also has a queue called ack queue, which is also composed of packets. It waits for the DataNode to receive a response. When all the DataNodes in the pipeline indicate that they have been received, then the akc queue will remove the corresponding packet.
• After the client finishes writing data, it calls the close method to close the write stream.
• DataStreamer flushes the remaining packets into the pipeline, and then waits for the ack information. After receiving the last ack, it informs the DataNode to mark the file as completed.