HDFS Erasure coding-Introduction and principles of erasure coding

Disadvantages of the three-copy strategy

• In order to provide fault tolerance, HDFS will replicate text blocks on different datanodes based on the replication factor.
• The default replication factor is 3, (note that 3 here refers to 1+2=3, not 3 additional copies), then in addition to the original block, there will be two additional copies. Each copy uses 100% of the storage overhead, thus resulting in 200% of the storage overhead. These copies also consume other resources, such as network bandwidth
• When the replication factor is N, there are N-1 fault tolerances, but the storage efficiency is only 1/N
  

Introduction to Erasure Coding (EC)

• Erasure coding technology, referred to as EC, is a coding fault-tolerant technology. It was first used in the communications industry for data recovery during data transmission. It divides the data into blocks and then calculates the check data to correlate each part of the data. When a part of the data blocks is lost, the lost data blocks can be calculated from the remaining data blocks and parity blocks.
• Erasure coding technology was introduced after Hadoop3.0, which can increase storage utilization by more than 50% and ensure data reliability.
  

Reed-Solomon (RS) code

• The reed-solomon (rs) code is a commonly used erasure code. It has two parameters k and m, denoted as RS (k, m)
• A vector composed of k data blocks is multiplied by a generator matrix (Generator Matrix) GT to obtain a codeword vector, which consists of k data blocks (d0, d1...d3) and m check blocks (c0 , c1) composition
• If a data block is lost, the lost data block can be recovered by multiplying the codeword vector by the inverse GT matrix.
  
  

EC architecture

To support erasure coding, some changes were made to the HDFS architecture

• namenode extension
  • Striped HDFS files are logically composed of block groups, each of which contains a certain number of internal blocks. This allows file management at the block group level instead of the block level
• client extension
  • The arrival of client-side read and write path enhancements that can process multiple internal blocks in a block group in parallel
• datanode extension
  • The datanode runs an additional ECWorker task to perform background recovery of failed erasure encoding blocks. The namenode detects the EC block, and then the namenode selects a datanode for recovery work.
• Erasure Coding Policy
  To accommodate heterogeneous workloads, files and directories in an HDFS cluster are allowed to have different replication and erasure coding policies. Erasure coding strategies encapsulate how files are encoded/decoded. The RS-6-31024k policy is enabled by default. RS is the encoding algorithm Reed-Solomon. 6,3 represents the number of data blocks and parity blocks. 1024k represents the size of the striping unit. The directory also supports the default REPLICSTION
  . plan. It can only be set on a directory to force the directory to adopt a 3x replication scheme rather than the inheritor's entanglement strategy. This strategy can be 3x replication scheme directory interleaved with error correction code directory. REPLICSTION is always in a poor state.
  In addition, users are also supported to define their own EC policies through XML files. There is a sample EC policy XML file named uers_ec_policies.xml.template in the Hadoop conf directory. Users can refer to this file.
• Inter ISA-L
  Intel ISA-L stands for Intel Intelligent Storage Acceleration Library. isa-l is an open source collection of low-level functions optimized for storage applications. It includes fast block Reed-Solomon type erasure code optimized for Intel AVX and AVX2 instructions. HDFS erasure coding can leverage ISA-L to accelerate encoding and encoding calculations