Lead: metadata management, storage access performance and efficiency of the mass of small files and other issues are currently academia and industry recognized problem. Fir rock mass object storage MOS technology, including the use of unique features, including the aggregation of small files to help users deal with waste storage resources, inefficient, and many other challenges, the core competitiveness of enterprises achievements and big data era of artificial intelligence.
Mass access to small files stored bring recognized problem
In today's world, the Internet, the rapid development of large data applications, networking, artificial intelligence, cloud computing technology advances, followed by a variety of business and personal applications continually generate one hundred million or even ten billion massive small files. Metadata management, storage access performance and efficiency of these small files and other issues thus become academia and industry recognized problem.
For example, currently the largest electricity supplier website Taobao store product images more than 20 billion, the average size of these files is only about 15KB, the total picture of the famous social networking site Facebook store more than 60 billion; online video play services, each video server is divided into slices of about 1MB fragmented files, image files contain an animated film could more than 5 million, with an average size of 15KB; some reading books online page content will It is scanned into approximately several tens of KB image file ......
Therefore, the application in the shopping website, radio and television, network video scene, a large number of small and medium file storage system, its footprint is generally 1M or less, operating characteristics for the write-once, read many times. Since each small file storage will be executed once all written procedures, frequent operation, so that the distributed file system disk pressure surge and cause problems in the following two aspects:
Waste of resources: when massive data reading small files, often small size of each file are less than the minimum block size in a transfer. That is, no matter how really useful amount of data, the data will get fixed size, which makes it useful for data acquisition accounting for the amount of data is too low, resulting in memory bandwidth waste.
High access latency: When a user reads a large number of small files, first select the metadata server interaction, the delay when accessing too concentrated, too frequent information exchange, the resulting clear. Also, since many systems currently does not provide a high access latency of prefetching, without considering the correlation between the data stored in the file, and thus exacerbated the delays.
Cedar Rock MOS nirvana two major challenges to deal with massive storage of small files
In response to these challenges, fir rock data launched its own unique skills to help users deal with the storage challenges in a typical application scenarios. Which comprises: a global storage resources for striping, to ensure concurrent writes application data read; small files providing aggregate functionality, reduce the number of small files written to disk, the write data relieve stress; improve disk usage, reading when small files where the object is read into the cache read hit rate increase, shortening the read I / O path.
:: technical advantages a small file storage performance polymeric solve a big problem
First, small files fir rock MOS polymerization techniques polymerization fragmented small files into one large file, reading data in advance to read and write data, improve the cache hit rate, so that small files (128KB less than) 100% increase read and write performance, read cluster performance increased by 50%.
The reason is that, compared to small files, file aggregation significantly reduced when writing data write OSD (object-based storage devices) times, reducing the pressure OSD written, thereby greatly reducing the client to the local cache (memory) usage of. When writing a small file order, frequency and OSD data are written can be reduced 99.9% (4KB), 98.4% (64KB), 96.8% (128KB).
Fir MOS rock in a small file is read sequentially, IO can also shorten the path. The reason is that, under polymerization conditions, the small files are available 99.9% (4KB),, 96.8% (128KB) cache hit rate of 98.4% (64KB), thus shortening the IO path, improve file reading performance, with small random read cache hit rate can reach more than 85%.
Secondly, instead of re-write-modify-write, reduce the write latency. Due to the small file Generally a write operation to complete, and in order to reduce the complexity, the MOS for the disk has a small file to the file polymerization not taken modify write mode, but take the "source file is read -> Delete file polymerization source files -> re-write general file "approach to write changes to reduce the write latency. Third, a timer intelligent scanning, defragmentation, garbage removal mechanism, improve utilization of storage space. In the polymerization file, a plurality of source objects share files, deleting the source file and can not really release the space occupied, resulting in waste of storage space; the same time, reducing the source file further results in a reduction of the number of cache hit rate when reading the file impact read performance. For this purpose, the scanning timing of the MOS aggregate file information using configurable: aggregate file usage is lower than p (set value) and the time t (set value) is not within the modified, then the aggregate file into the clean queue; Scan All files have been processed polymerization, if there is a need to clean up the queue aggregate file cleanup, defragmentation is triggered, thereby improving space utilization.
Garbage task is active in the polymerization file data in proportion in the total file size, when below a set threshold, and when no aggregate file modifications within the time t, clean aggregate file system, in order to improve storage space utilization, improve small file read performance.
Technical Advantage two: pre-read by the data read speed increase
Fir MOS rock when reading a file in a directory, the order will be preloaded behind the content of the document file into several cache, to speed up the read speed; In addition, the use of multi-node concurrent read operations to speed up pre- read speed.
Meanwhile a read request, when a file in the directory asynchronous read, the MOS also triggers its subsequent files, and these read requests are asynchronous, this does not affect the read operation returns the user. If the user immediately reads the next file, and this operation is not synchronized, then the local cache can directly read the data, so as to avoid interaction with the OSD, improves performance.
Meanwhile, when the prefetch, if the writing operation of the file, the MOS current will wait to read files into the queue and returns the read, write, etc. after the completion of the read signal the document again, and will be placed in the local cache, thereby improving the user reading speed.
Because of the above unique technology mechanism, fir rock mass object storage product MOS solves small file read and write speed is slow, high access latency, waste of resources and other issues to help radio and television, electricity providers, online video and other users to calmly deal with massive small files storage and access challenges. Currently, fir rock data system has been widely used in finance, telecommunications, government, public security, industry and other large enterprise customers to fully verify its leading technology. Future, fir rock MOS software-defined storage storage technology will continue to lead the era of big data and artificial intelligence for users to increase their core competitiveness of innovative enterprises.