What is Erasure Code
Erasure Code (EC for short), that is, erasure code, is a forward error correction technology (Forward Error Correction, FEC), which is mainly used to avoid packet loss during network transmission. Storage systems use it to improve storage and reliability. Compared with multi-copy replication, erasure coding can obtain higher data reliability with less data redundancy, but the encoding method is more complicated and requires a lot of calculations. Erasure codes can only tolerate data loss, but cannot tolerate data tampering, which is why erasure codes are named.
Definition of EC: Erasure Code is an encoding technology that can add n copies of original data to m copies of data, and can restore the original data through any n copies of data in n+m copies. That is, if any data less than or equal to m copies fails, it can still be restored through the remaining data.
At present, there are three main types of erasure code technology in distributed storage systems, array erasure code (Array Code: RAID5, RAID6, etc.), RS (Reed-Solomon) Reed-Solomon type erasure code and LDPC (LowDensity Parity Check Code) Low Density Parity Check Erasure Code.
RAID is a special case of EC. In traditional RAID, only limited disk failure is supported, RAID5 supports only one disk failure, RAID6 supports two disk failures, and EC supports multiple disk failures.
EC is mainly used in the field of storage and digital encoding. For example, disk array storage (RAID 5, RAID 6), cloud storage (RS), etc.
LDPC codes can also provide a good redundancy mechanism to ensure reliability. Compared with RS coding, LDPC coding efficiency is slightly lower, but the coding and decoding performance is better than RS code and other erasure codes, mainly due to the relatively few and simple XOR operations used in coding and decoding. LDPC codes are currently mainly used in communication, video and audio coding and other fields.
This article mainly explains RS erasure codes.
Reed-Solomon Code
RS code is a coding algorithm based on finite fields. Finite fields are also called Galois Field. It is named after the famous French mathematician Galois (Galois). GF(2^w) is used in RS code , where 2 ^w >= n + m.
The codec definition of RS code is as follows:
❝Coding
: Given n data blocks (Data block) D1, D2...Dn, and a positive integer m, RS generates m code blocks (Code block), C1, C2...Cm according to n data blocks.
❞
RS encoding uses word as the encoding and decoding unit, and splits a large data block into a word with a word length of w (generally 8 or 16 bits), and then encodes and decodes the word. The encoding principle of the data block is the same as that of the word encoding. In the following, a word will be used as an example, and the variables Di and Ci will represent a word.
Treat the input data as a vector D=(D1, D2,..., Dn), and the encoded data as a vector (D1, D2,..., Dn, C1, C2,..., Cm), and the RS code can be regarded as the following figure shows matrix operations.
The far left of the above figure is the encoding matrix (or called generation matrix, distribution matrix, Distribution Matrix), and the encoding matrix needs to satisfy any n*n sub-matrix reversibility.
For the convenience of data storage, the upper part of the encoding matrix is a unit matrix (n rows and n columns), and the lower part is a matrix with m rows and n columns. The lower matrix can choose Vandermonde matrix or Cauchy matrix.
RS code encoding data recovery principle
RS can tolerate up to m data blocks being deleted. The process of data recovery is as follows:
(1) Assuming that D1, D4, and C2 are lost, delete the row corresponding to the lost data block/encoding block from the encoding matrix.
According to the RS coding operation equation shown in FIG. 1, the following B' and the equation can be obtained.
(2) Since B' is reversible, record the inverse matrix of B' as (B'^-1), then B' * (B'^-1) = I unit matrix. Multiply the B' inverse matrix on both sides
(3) to get the following calculation formula of the original data D,
that is to restore the original data D:
4) Re-encode D to get the lost code
Limitations of RS code encoding
Data recovery and update costs are high, so it is often aimed at read-only data or cold data.
RS encoding relies on two log tables with a size of 2^w-1. Usually, only 16-bit or 8-bit word lengths can be used, which cannot fully utilize the computing power of 64-bit servers. Some optimizations may be required for specific implementation.
RS encoding upgrades
the data after RS encoding. If a piece is lost, at least n pieces of data need to be read to restore the lost data. In a production environment, hard disk failures often occur, and restoring data consumes a lot of network IO and CPU.
Therefore, some companies have made some improvements on the basis of EC codes, using LRC or SEC to replace RS codes.
LRC - Locally Repairable Code The local copy storage
LRC code is basically the same as the RS code, while adding an additional copy of the data block. LRC encoding is essentially RS encoding + 2 copy backup.
The LRC encoding steps are as follows:
Use RS encoding for the original data, for example, the encoding is 4:2, and the encoding result is 4 data blocks: D1, D2, D3, D4, and 2 encoding blocks C1, C2;
Make 2 copies of the original data, and generate 2 encoding blocks from the first 2 data blocks and the last 2 data blocks of the 4 data blocks, that is, R1=D1D2, R2=D3D4;
If a data block is lost, for example, D2 is lost, only R1 and D1 are needed to restore D2;
The 2-copy code of step b is added to the LRC coding matrix, which looks like this:
SEC - Sparse Erasure Code
In LRC coding, only 2 copies of the data block are made. When the coded block is lost, n blocks still need to be read data to recalculate the encoding block.
In SEC encoding, a check block is added to both the data block and the code block.
SEC encoding is essentially RS encoding + parity block. The SEC encoding steps are as follows:
Use RS encoding for the original data, for example, the encoding is 4:2, and the encoding result is 4 data blocks: D1, D2, D3, D4, and 2 encoding blocks C1, C2;
Generate the verification block X1 of D1D2, the verification block X2 of D3D4, and the verification block X3 of C1C2;
When one of the data blocks or coding blocks is lost, for example, C2 is lost, C2 can be recovered through C1 and check block X3;
SEC also reduces the network and CPU overhead of restoring data by adding storage blocks.