data flow:
1. Analysis file read:
The FileSystem object is responsible for contacting the namenode, obtaining the block location, and then giving the information to FSDataInputStream to construct an object. In this figure, there is also an intermediate stream missing, that is, the DFSInputStream object, which is constructed by FSDataInputStream for direct The data block is read in the datanode (here, the most recent topologically loaded first), and FSDataInputStream is responsible for closing the data stream.
DFSInputStream will also check whether the block is damaged. If it is damaged, it will remember the damaged block and report it to the namenode, and then read the next copy. It is estimated that the MD5 encryption algorithm is used to obtain the digital fingerprint. This is entirely a personal guess. .
In this way, the namenode can tell the client the datanode where the location of the most recent best block is located, and the namenode only provides the information of the block location, which is all in memory, and getting this information is just one word, fast. Note: datanode will also be a client (^*^)
As shown in the figure:
2. Analysis file write
1. The client first reports to namenode that I want to write this file, namennode said: "Let me see if I don't have this file, um, I don't, let me check if you have the right to write, um, OK", you can go and write. But there is also a DataStream in it to process the data packets sent by the client, and according to the number and size of these data packets, apply to the namenode on which datanode to store, and DFSOutputStream will also supervise (using the confirmation queue) whether all The datanodes are successfully saved to the data.
2. If the datanode fails, go to the downstream node for storage, but add all the data packets back to the data queue to protect the integrity of the data.
3. Replica storage strategy: the default client node saves one, if the client is not in the cluster, randomly select one to save,
Then store two more replicas on another rack. Doing so ensures data redundancy and saves network bandwidth.
