Author: Wang Ao (OX)
Link: https://www.zhihu.com/question/20131784/answer/90235520
Source: Zhihu
The copyright belongs to the author. For commercial reprints, please contact the author for authorization, and for non-commercial reprints, please indicate the source.
## Preface
RAID explanation I am a little lazy to quote WikipediA, Redundant Array of Independent Disks (RAID, Redundant Array of Independent Disks), formerly known as Redundant Array of Inexpensive Disks, or disk array for short. The basic idea is to combine multiple relatively cheap hard disks into a hard disk array group, so that the performance can reach or even exceed an expensive hard disk with huge capacity. Depending on the version chosen, RAID has one or more of the following benefits over a single hard drive: enhanced data integration, enhanced fault tolerance, and increased throughput or capacity. In addition, a disk array looks like a single hard disk or logical storage unit to a computer. Of course, writing this article is not simply to introduce concepts and usage methods, but more importantly, how to make reasonable RAID configurations and parameter optimizations for different business scenarios. For the addition of SSD solid-state drives, I introduced experimental data from the Xiaomi operation and maintenance team, and at the same time I also believe that distributed storage will gradually mature, and technologies such as OpenStack, VSAN, and Nutanix that represent the concepts of software-defined and hyper-converged overhead have also begun a dark battle.
> The essence of RAID is to balance availability and cost
---
## update history
March 11, 2016 - first draft
Read the original text - RAID Disk Array Configuration and Tuning Summary
**Extended reading**
RAID - https://en.wikipedia.org/wiki/RAID
RAID Technology Development Overview - RAID Technology Development Overview
SSD Array Card Solution Optimization: Consider Using RAID 50 to Replace RAID 10 - SSD Array Card Solution Optimization: Consider Using RAID 50 to Replace RAID 10
---
## RAID Basics
>Thanks to @刘爱贵, for detailed knowledge points, please refer to extended reading
### Fundamental
RAID (Redundant Array of Independent Disks) is a redundant array of independent disks, usually referred to as a disk array. Simply put, RAID is a disk subsystem composed of multiple independent high-performance disk drives, thereby providing higher storage performance and data redundancy technology than a single disk. RAID is a multi-disk management technology that provides high-performance storage with moderate cost and high data reliability to host environments. SNIA defines RAID as: a disk array in which part of the physical storage space is used to record redundant information of user data stored in the remaining space. When one of the disks or access paths fails, the redundant information can be used to reconstruct user data. Disk striping, although inconsistent with the definition of RAID, is often referred to as RAID (ie RAID0).
The original intention of RAID is to provide high-end storage capabilities and redundant data security for large servers. In the whole system, RAID is regarded as a storage space composed of two or more disks, which improves the I/O performance of the storage system by concurrently reading and writing data on multiple disks. Most RAID levels have complete data verification and corrective measures, thereby improving the fault tolerance of the system, and even mirroring, which greatly enhances the reliability of the system. Redundant also comes from this.
Here to mention JBOD (Just a Bunch of Disks). Originally JBOD was used to denote a collection of disks with no control software to provide coordinated control, and this is the main factor that differentiates RAID from JBOD. Currently JBOD often refers to a disk enclosure, regardless of whether it provides RAID functionality or not.
Two key goals of RAID are to improve data reliability and I/O performance. In a disk array, data is spread across multiple disks, but to a computer system, it looks like a single disk. Redundancy is achieved by simultaneously writing the same data to multiple disks (typically mirrored), or by writing the calculated parity data to the array, ensuring that no data will be lost in the event of a single disk failure. Some RAID levels allow more disks to fail simultaneously, such as RAID6, where two disks fail at the same time. Under such a redundancy mechanism, the failed disk can be replaced with a new disk, and the RAID will automatically rebuild the lost data based on the data in the remaining disks and the parity data to ensure data consistency and integrity. Data is scattered on multiple different disks in RAID, and concurrent data read and write is much better than a single disk, so higher aggregate I/O bandwidth can be obtained. Of course, disk arrays reduce the total available storage space for all disks, sacrificing space for higher reliability and performance. For example, RAID1 storage space utilization is only 50%, RAID5 will lose the storage capacity of one of the disks, and the space utilization is (n-1)/n.
Disk arrays can guarantee uninterrupted continuous operation of the system in the event of partial disk failure (single or multiple blocks, depending on the implementation). During the process of rebuilding the failed disk data to the new disk, the system can continue to operate normally, but the performance will be degraded to a certain extent. Some disk arrays must be shut down to add or remove disks, while others support Hot Swapping, allowing disk drives to be replaced without downtime. This high-end disk array is mainly used for application systems that require high probability, the system cannot be shut down or the downtime is as little as possible. Generally speaking, RAID cannot be used as a replacement for data backup. It cannot do anything about data loss caused by non-disk failure, such as virus, human damage, accidental deletion, etc. The data loss at this time is relative to the operating system, file system, volume manager or application system. For the RAID system itself, the data is intact and no loss has occurred. Therefore, data protection measures such as data backup and disaster recovery are very necessary. They complement each other with RAID to protect the security of data at different levels and prevent data loss.
There are three key concepts and technologies in RAID: Mirroring, Data Stripping and Data parity. Mirroring, which replicates data to multiple disks, improves reliability on the one hand, and improves read performance by concurrently reading data from two or more replicas. Obviously, the write performance of the mirror is slightly lower, and it takes more time to ensure that the data is correctly written to multiple disks. Data striping stores data shards on multiple different disks, and multiple data shards together form a complete copy of data, which is different from multiple copies of mirroring, which is usually used for performance considerations. Data stripes have higher concurrency granularity. When accessing data, data on different disks can be read and written at the same time, thereby obtaining a considerable improvement in I/O performance. Data verification uses redundant data to detect and repair data errors. Redundant data is usually calculated and obtained by algorithms such as Hamming code and XOR operation. With the check function, the reliability, robustness and fault tolerance of the disk array can be greatly improved. However, data verification needs to read data from multiple places and perform calculations and comparisons, which will affect system performance. Different levels of RAID use one or more of the three technologies to achieve different levels of data reliability, availability, and I/O performance. As for what kind of RAID to design (even a new level or type) or which mode of RAID to use, it is necessary to make a reasonable choice under the premise of in-depth understanding of system requirements, and comprehensively evaluate reliability, performance and cost to make a compromise choice.
The RAID idea has been widely accepted by the industry since it was proposed, and the storage industry has invested a lot of time and financial resources to research and develop related products. Moreover, with the continuous development of processor, memory, computer interface and other technologies, RAID continues to develop and innovate, and has been widely used in the field of computer storage, gradually extending from high-end systems to ordinary low-end systems. The popularity of RAID technology stems from its significant features and advantages, which can basically meet most data storage needs. In general, the main advantages of RAID are as follows:
(1) Large capacity
This is an obvious advantage of RAID, which expands the capacity of the disk, and the RAID system composed of multiple disks has massive storage space. Now the capacity of a single disk can reach more than 1TB, so that the storage capacity of RAID can reach PB level, and most storage needs can be met. Generally speaking, the available capacity of RAID is less than the total capacity of all member disks. Different levels of RAID algorithms require a certain redundancy overhead, and the specific capacity overhead is related to the algorithm used. If the RAID algorithm and capacity are known, the usable capacity of the RAID can be calculated. Typically, RAID capacity utilization is between 50% and 90%.
(2) The high performance of high-performance
RAID benefits from data striping technology. The I/O performance of a single disk is limited by computer technologies such as interface and bandwidth, and the performance is often limited, which can easily become the bottleneck of system performance. With data striping, RAID spreads data I/O across member disks, resulting in exponentially increased aggregate I/O performance over a single disk.
(3) Reliability
Availability and reliability are another important feature of RAID. In theory, a RAID system consisting of multiple disks should be less reliable than a single disk. There is an implicit assumption here: a single disk failure will render the entire RAID unavailable. RAID uses data redundancy techniques such as mirroring and data parity to break this assumption. Mirroring is the most primitive redundancy technology, which completely replicates the data on one set of disk drives to another set of disk drives to ensure that there is always a copy of the data available. Compared with the 50% redundancy overhead of mirroring, data verification is much smaller, and it uses the verification redundancy information to check and correct the data. RAID redundancy technology greatly improves data availability and reliability, ensuring that when several disks fail, data will not be lost and the continuous operation of the system will not be affected.
(4) Manageability
In fact, RAID is a virtualization technology that virtualizes multiple physical disk drives into a large-capacity logical drive. To an external host system, RAID is a single, fast and reliable large-capacity disk drive. In this way, users can organize and store application system data on this virtual drive. From the perspective of user application, the storage system is easy to use and management is also very convenient. Since a lot of storage management work is done inside RAID, administrators only need to manage a single virtual drive, which can save a lot of management work. RAID can dynamically add or remove disk drives, and can automatically perform data checksums and data reconstructions, which can greatly simplify management.
### Key Technologies
> Mirror
Mirroring is a redundancy technology that provides protection for disks against data loss due to disk failure. For RAID, mirroring typically produces two identical copies of data in the array at the same time, spread over two different sets of disk drives. Mirroring provides complete data redundancy. When one data copy fails and becomes unavailable, the external system can still access the other copy normally without affecting the operation and performance of the application system. Moreover, the mirroring does not require additional calculation and verification, and the fault repair is very fast, and it can be directly copied. Mirroring technology can concurrently read data from multiple copies and provide higher read I/O performance, but cannot write data in parallel, and writing multiple copies will result in a certain I/O performance degradation.
Mirroring technology provides very high data security, but it is also very expensive, requiring at least double the storage space. The high cost limits the wide application of mirroring, which is mainly used for critical data protection, where data loss will cause huge losses. In addition, the mirror can obtain a snapshot of the data at a specific point in time through "splitting", so that a data backup technology with almost zero backup window can be realized.
> Data stripes
The performance bottleneck of disk storage is head seek positioning, which is a slow mechanical motion that cannot match a high-speed CPU. Furthermore, there is a physical limit to the performance of a single disk drive, and I/O performance is very limited. RAID consists of multiple disks, and data striping technology distributes and stores data on multiple disks in blocks, so that data can be processed concurrently. In this way, data writing and reading can be performed simultaneously on multiple disks, resulting in very high aggregate I/O concurrently, which effectively improves the overall I/O performance and has good linear scalability. This is especially true for large volumes of data, where without chunking, the data can only be stored sequentially on the disks of the disk array and read sequentially when needed. With the striping technology, the performance improvement of several times and sequential access can be obtained.
The choice of block size for striping technology is critical. The stripe granularity can be one byte to several KB in size. The smaller the block, the stronger the parallel processing capability and the higher the data access speed, but at the same time, the randomness of block access and block addressing time will be increased. In practical applications, an appropriate block size should be selected according to data characteristics and requirements, and a balance should be made between data access randomness and concurrent processing capability to strive for the highest possible overall performance.
Data striping is based on improving I/O performance, that is to say, it only focuses on performance without any improvement in data reliability and availability. In fact, the corruption of any one of the data stripes will make the entire data unavailable, and the use of data striping technology increases the conceptual rate of data loss.
> Data verification
Mirroring has high security and high read performance, but the redundancy overhead is too expensive. Data striping greatly improves performance through concurrency, but data security and reliability are not considered. Data verification is a redundancy technology, which uses verification data to provide data security, can detect data errors, and perform data reconstruction under the premise of capability. Compared with mirroring, data verification greatly reduces the redundancy overhead and obtains excellent data integrity and reliability at a lower cost. Data striping technology provides high performance, data parity provides data security, and different levels of RAID often combine these two technologies at the same time.
When data parity is used, RAID should perform parity calculation while writing data, and store the obtained parity data in RAID member disks. The parity data can be centrally stored on a certain disk or scattered in multiple different disks, and even the parity data can be divided into blocks, and the implementation of different RAID levels varies. When a part of the data is in error, the remaining data and the check data can be reverse-checked to reconstruct the lost data. The advantage of parity technology compared to mirroring technology is that it saves a lot of overhead, but because a large number of parity operations are performed each time data read and write, the computer's computing speed is very high, and a hardware RAID controller must be used. Verification techniques are much more complex and slower than mirroring techniques in terms of data reconstruction and recovery.
Hamming and XOR are the two most commonly used data verification algorithms. Hamming check code was proposed by Richard Hamming, which not only detects errors, but also gives the location of errors and corrects them automatically. The basic idea of Hamming check is: divide the valid information into several groups according to a certain rule, perform parity test for each group and arrange a check bit, so as to provide multi-bit error detection information to locate and correct the error point. It can be seen that Hamming check is essentially a multiple parity check. The XOR check is generated by the XOR logic operation, and an XOR operation is performed on a valid information and a given initial value to obtain the check information. If the valid information is wrong, the correct valid information can be restored by the XOR operation between the check information and the initial value.
## Common RAID Types
> Comparison of 5 common RAID types, the number of n-bit disks, for details, please refer to Extended Reading
RAID Level | RAID0 | RAID1 | RAID5 | RAID6 | RAID10
----|----|----|----|----|----
Alias| Stripe| Mirror| Distribution Parity Parity Stripe | Double Parity Stripe | Mirror Plus Stripe
Fault Tolerance | No | Yes | Yes | Yes |
Yes Redundancy Type | No | Yes |
Yes | Yes | |Yes|
YesRead performance|High|Low|High|High|High
Random write performance|High|Low|Normal|Low|Normal
Continuous write performance|High|Low|Low|Low|
Number of disks required| n≥1| 2n (n≥1) | n≥3 | n≥4 | 2n(n≥2)≥4
Usable Capacity | All | 50% | (n-1)/n | (n-2)/n | 50%
## RAID level
### Standard RAID Level
Organizations such as SNIA and Berkeley have set seven levels of RAID0, RAID1, RAID2, RAID3, RAID4, RAID5, and RAID6 as standard RAID levels, which are also recognized by the industry and academia. Standard levels are the most basic set of RAID configurations that utilize data striping, mirroring, and data parity techniques alone or in combination. Standard RAID can be combined, that is, RAID combination level, to meet the storage application requirements that require higher performance, security, and reliability.
### JBOD
JBOD (Just a Bunch Of Disks) is not a standard RAID level, it is usually used to represent a collection of disks that have no control software to provide coordinated control. JBOD concatenates multiple physical disks to provide one huge logical disk. The data storage mechanism of JBOD is to start from the first disk and store it in order. After the current disk storage space is used up, the data is stored in the subsequent disks in turn. JBOD storage performance is completely equivalent to a single disk, and it does not provide data security protection. It simply provides a mechanism to expand storage space, and the available storage capacity of JBOD is equal to the sum of the storage space of all member disks. Currently JBOD often refers to a disk enclosure, regardless of whether it provides RAID functionality or not.
### RAID0
RAID0 is a simple data striping technology with no data parity. Not actually a true RAID, as it doesn't provide any form of redundancy strategy. RAID0 stripes the disks to form a large-capacity storage space, stores data in all disks, and realizes parallel read access of multiple disks in an independent access mode. Since I/O operations can be performed concurrently, the bus bandwidth is fully utilized. Coupled with the fact that no data verification is required, RAID0 has the highest performance among all RAID levels. Theoretically, a RAID0 composed of n disks, its read and write performance is n times the performance of a single disk, but due to the limitations of various factors such as bus bandwidth, the actual performance improvement is lower than the theoretical value.
RAID0 has the advantages of low cost, high read and write performance, and 100% high storage space utilization, but it does not provide data redundancy protection, and once data is damaged, it cannot be recovered. Therefore, RAID0 is generally suitable for applications that have strict performance requirements but low data security and reliability, such as video, audio storage, and temporary data cache space.
### RAID1
RAID1 is called mirroring, it writes data to the working disk and the mirror disk in complete consistency, and its disk space utilization is 50%. In RAID1, the response time will be affected when data is written, but not when data is read. RAID1 provides the best data protection, once the working disk fails, the system automatically reads data from the mirror disk, which will not affect the user's work.
RAID1 is the exact opposite of RAID0, in order to enhance data security, the data of the two disks are completely mirrored, so as to achieve good security, simple technology and convenient management. RAID1 is fully fault tolerant, but it is expensive to implement. RAID1 is used in applications that require high sequential read and write performance and attach great importance to data protection, such as data protection for mail systems.
### RAID5
RAID5 should be the most common RAID level at present, its principle is similar to RAID4, the difference is that the parity data is distributed on all the disks in the array, and no special parity disk is used. For data and parity data, their writes can occur on completely different disks at the same time. Therefore, RAID5 does not have the performance bottleneck of the parity disk during concurrent write operations in RAID4. In addition, RAID5 also has good scalability. As the number of disks in the array increases, the capacity for parallel operations also increases, supporting more disks than RAID4, resulting in higher capacity and higher performance.
Both data and parity data are stored on RAID5 disks. Data blocks and corresponding parity information are stored on different disks. When a data disk is damaged, the system can data to rebuild corrupted data. As with other RAID levels, RAID5 performance suffers significantly when rebuilding data.
RAID5 takes into account various factors such as storage performance, data security and storage cost. It can be understood as a compromise between RAID0 and RAID1, and is currently the best data protection solution with comprehensive performance. RAID5 can basically meet the needs of most storage applications, and most data centers use it as a protection scheme for application data.
### RAID6
Each of the RAID levels described above can only protect against data loss due to the failure of a single disk. If both disks fail at the same time, data cannot be recovered. RAID6 introduces the concept of double parity, which can protect the array to continue to work when two disks in the array fail at the same time without data loss. RAID6 level is a RAID method designed on the basis of RAID5 to further enhance data protection. It can be regarded as an extended RAID5 level.
RAID6 has to support not only the recovery of data, but also the recovery of parity data, so the implementation cost is high, and the design of the controller is more complicated and expensive than other levels. The most common implementation of the RAID6 idea is to use two independent parity algorithms, which are assumed to be called P and Q, and the parity data can be stored on two different parity disks, or scattered across all member disks. When two disks fail at the same time, the data on both disks can be reconstructed by solving two-variable equations.
RAID6 has fast read performance and higher fault tolerance. However, it is much more expensive than RAID5, has poor write performance, and is very complex to design and implement. Therefore, RAID6 is rarely used in practice, and is mainly used in occasions that require very high levels of data security. It is generally an economical alternative to RAID10 solutions
### RAID Combination Level
Standard RAID levels have advantages and disadvantages. Naturally, we thought of combining multiple RAID levels to achieve complementary advantages and make up for each other's shortcomings, so as to achieve a RAID system with higher performance, data security and other indicators. At present, the RAID combination levels mentioned in the industry and academic research mainly include RAID00, RAID01, RAID10, RAID100, RAID30, RAID50, RAID53, RAID60, but only RAID01 and RAID10 are actually widely used. Of course, the implementation cost of the combined grade is generally very expensive, and it is only used in a few specific occasions.
### RAID10 and RAID01
Some literature treats these two RAID levels as equivalent, and this paper considers them to be different. RAID01 is striping first and then mirroring, essentially mirroring physical disks; while RAID10 is mirroring first and then striping, mirroring virtual disks. Under the same configuration, usually RAID01 has better fault tolerance than RAID10.
RAID01 combines the advantages of RAID0 and RAID1. It uses two disks to create a mirror first, and then does striping within the mirror. The data of RAID01 will be written to two disk arrays at the same time. If one of the arrays is damaged, it can still continue to work, ensuring data security and improving performance. Both RAID01 and RAID10 contain RAID1 mode, so the overall disk utilization is only 50%.
### RAID 50
The combination of RAID 5 and RAID 0, first make RAID 5, then make RAID 0, that is, to form Stripe access to each other for multiple groups of RAID 5. Since RAID 50 is based on RAID 5, and RAID 5 requires at least 3 hard drives, to form RAID 50 with multiple sets of RAID 5, at least 6 hard drives are required. Take the minimum 6-disk configuration of RAID 50 as an example, first divide the 6 hard disks into 2 groups, each group of 3 disks constitutes RAID 5, so that two groups of RAID 5 are obtained, and then the two groups of RAID 5 constitute RAID 0.
RAID 50 can still maintain operation when one hard drive in any one or more groups of RAID 5 at the bottom is damaged, but if 2 or more hard drives in any group of RAID 5 are damaged, the entire RAID 50 group will fail. invalid.
RAID 50 has higher performance than pure RAID 5 because multiple groups of RAID 5 are formed into Stripe in the upper layer, and the capacity utilization rate is lower than that of RAID 5. For example, 9 hard disks are also used, and 3 RAID 5s each form a RAID 50 of RAID 0. Each group of RAID 5 wastes one hard disk, and the utilization rate is (1-3/9), while the RAID 5 is (1-1/ 9).
### RAID 60
Combination of RAID 6 and RAID 0: Make RAID 6 first, and then make RAID 0. In other words, Stripe access to more than two groups of RAID 6. RAID 6 requires at least 4 hard drives, so the minimum requirement for RAID 60 is 8 hard drives.
Since the bottom layer is composed of RAID 6, RAID 60 can allow up to 2 hard drives in any RAID 6 group to be damaged, and the system can still maintain operation; It will fail, of course, the probability of this situation is quite low.
Compared with pure RAID 6, the upper layer of RAID 60 forms Stripe access by combining multiple groups of RAID 6, so the performance is higher. However, the use threshold is high, and the low capacity utilization rate is a big problem.
## About RAID parameter tuning
> Usually, it is recommended to separate the system (RAID1) and data (RAID[5|10]), here is a quote from Mr. Ye Jinrong
1. Use SSD or PCIe SSD devices, at least hundreds of times or even 10,000 times increase in IOPS
2. Purchase an array card equipped with CACHE and BBU modules, which can significantly increase IOPS (mainly refers to mechanical disks, except for SSD or PCIe SSD. At the same time It is necessary to check the health status of CACHE and BBU modules regularly to ensure that data will not be lost in the event of an accident)
3. When there is an array card, set the array write strategy to WB, or even FORCE WB (if there is double power protection, or the data security requirements are not If it is particularly high), it is strictly forbidden to use the WT strategy. And the closed-array read-ahead strategy
4. Use RAID-10 instead of RAID-5 as much as possible (`this sentence is open to question`)
5. If you use a mechanical disk, choose a high-speed one as much as possible, such as 15KRPM instead of 7.2 KRPM disks are not less than a few dollars;
## SSD array card solution optimization
> Thanks to the @xiaomi noops operation and maintenance team, please refer to the extended reading for detailed experimental data
**Performance test conclusion**
The performance test shows that the performance of R50 and R10 of the same capacity is similar: the random read of small block files is better than R50, and the random write of 4K is 28%. R50 is better than R10 after the block is increased. . In terms of sequential read and write, R50 and R10 are very close.
In terms of fault tolerance, R50 is close to R10: The fault tolerance rate of the second disk is very close to R50, with a difference of 30%. The advantage of R10 is mainly to provide the fault tolerance rate of the third or even the fourth disk with a certain probability, but considering that it is not 100% fault-tolerant, from the perspective of fault tolerance, although R50 has some gaps with R10, it has also been reflected. Better fault tolerance, at least better than R5. Moreover, the R50 is flexible, and even 3 groups of R5s can be specified to achieve a maximum fault tolerance of 3 disks;
In terms of cost, the R50 has a great advantage: according to this configuration, the R50 is only 3/4 of the R10.
**Summarize**
RAID 50 provides features close to RAID 10 performance, availability, and cost close to RAID 5, and has a better overall price/performance advantage, so consider using RAID 50 instead of RAID 10.