First, the pipeline is closed, and the data packets that have been sent to the pipeline but have not received an acknowledgment are rewritten back to the data queue, so that no matter which node fails, no data loss will occur. This process is to delete the unacknowledged data packets in the confirmation queue and write them back to the data queue.
Then the currently working data node will be given a new version number (using the lease information in the namenode to obtain the latest timestamp version), so that after the faulty node recovers, due to incorrect version information, the faulty DataNode will be deleted after recovery.
In the current normal datanode, select a master DataNode according to the lease information, and communicate with other normal DataNodes to obtain the size of the current data block of each DataNode, select a minimum value, and synchronize each normal DataNode to this size. Then re-establish the pipeline.
Delete the failed node in the pipeline, and write the data to the remaining normal DataNode in the pipeline, that is, the new pipeline. When the file is closed, the namenode creates a new replica on another node when it finds that the number of replicas is insufficient.