1. HDFS read file process:
Detailed reading process:
Client calls the FileSystem.open() method:
1 FileSystem communicates with NN through RPC , and NN returns part or all of the block list (DN address containing block copy) of the file.
2 Select the nearest DN of the client to establish a connection, read the block, and return FSDataInputStream .
The client calls the read() method of the input stream:
1 When the end of the block is read, FSDataInputStream closes the connection with the current DN and does not read the next block to find the nearest DN .
2 After reading a block, checksum verification will be performed . If there is an error when reading the DN, the client will notify the NN, and then continue reading from the next DN that has a copy of the block.
3 If the file is not over after the block list is read, FileSystem will continue to obtain the next block list from NN.
close FSDataInputStream
2. HDFS file writing process:
Detailed writing process:
Client calls the create() method of FileSystem:
1 FileSystem sends a request to NN to create a new file in the namespace of NN, but does not associate any blocks .
2 NN checks whether the file already exists and the operation permission . If the check passes, the NN records the new file information and creates a data block on a certain DN.
3 Return to FSDataOutputStream and direct the Client to the data block to perform the write operation.
The client calls the write() method of the output stream:
HDFS places 3 copies of each data block by default. FSDataOutputStream first writes the data to the first node, the first node transmits and writes the data packet to the second node, the second node --> the third node.
The client calls the close() method of the stream:
Flush the data packets in the buffer. After the block completes the number of copies, NN returns a success message.