Data is stored for analysis

The reading and writing of disks has not been able to keep up with the development of the times. In the 1990s, an ordinary hard disk had a space of about 1370MB, and the speed was 4.4MB/S, and it took about 5 minutes to read all the data at this speed. Today, 1TB is very normal, but the speed is only about 100MB/S, so it may take two and a half hours to read 1TB speed. Writing is slower.

A relatively simple optimization method can distribute 1TB of data to different disks, such as 100 disks. Each time the data is read, multiple disks can be performed at the same time, so the time may be reduced by 100 times (the most perfect case). So (assuming the data is partitioned in files), a disk split can be done for each data. This solution needs to ensure the availability of data, and it is necessary to perform redundant backups for each disk to prevent a recovery mechanism (RAID backup mechanism is one) when a disk fails.