Article Directory
Foreword
Before I explained how we Ozone for compatibility as Hadoop file system using external applications, what we call the Ozone FileSystem. Ozone FileSystem but did not change the underlying storage format in Key Ozone is, in essence, it's just made a path from tree to Ozone's volume, bucket, key transformation (detailed article can be seen here: the inner workings of the Ozone FileSystem achieve analysis ). So if we can in a tree in full compliance with organizational form of the namespace in the traditional sense, while maintaining the underlying file objects stored in the form of Key Ozone over it? In this paper the author of this community to talk about Ozone's a tentative plan, currently in the implementation process.
Namespace Summary Ozone FS
First introduced what is called Ozone FS Namespace, Ozone FS Namespace mean we achieve FileSystem namespace support within the Ozone. In FS Namespace, we have a table of contents and metadata information as well as up and down the organization and management of files between them.
Compared implementation of existing Ozone FileSystem for Ozone FS Namespace, has the following advantages and disadvantages:
Advantage: achieve a Namespace FileSystem form itself, more flexibility, making file operations more efficient. Ozone FileSystem an existing path just outside the packaging conversion process, which is performed at the directory level operation, using the prefix matching processing traversal way to do this is not an efficient approach.
Disadvantages: The Namespace form introduced Ozone can cause performance problems of potential Ozone. Because the changes will be updated FS Namespace Manager Ozone internal metadata for existing treatments.
Design of Ozone FS Namespace
Ozone FS Namespace specific design of the structure we look at below. In this design the community in which the separation of the two tables to store metadata information respectively directories and files, here we simply called Directory Table and File Table.
For example, the path named "/a/b/c/d/1.txt" file, which is stored in the information format Ozone FS Namespace, the four entry items in the Directory Table, 1 Ge entry item in the File Table.
4 Directory Table entries: a, b, c, d .
1 File Table entries: 1.txt.
Light of the separate items of entry is not enough, we also lack organizational relationships of them. Ozone is represented here by an additional saving Prefix attribute information of the parent directory to which it belongs, parent directory information ID.
[parent unique-id]/[directory name] => [directory unique-id]
File information indicates similar as above:
[parent unique-id]/[file name] => [OM KeyInfo]
In /a/b/c/d/1.txt an example, the actual Directory Table, File Table storage format would be as follows:
Directory Table：
| Prefix | ID |
|---|---|
| / | 99 |
| 99/a | 100 |
| 100/b | 101 |
| 101/c | 102 |
File Table：
| Filename | Key Metadata |
|---|---|
| 102/1.txt | KeyInfo |
Prefix stored in the form of the above provide significant cost savings of space, since each directory entry parent directory path portion of the Entry simply use a 64-bit (8-byte) can be represented by a digital ID. For example, in the case where the average of 32 bytes the file name, the total path length of 256 bytes, 100w files consumes to 256GB (= 2 ^ 30 * 256Bytes) space. In the case where the Prefix Mode, we only use 32 bytes of storage, only need to use the space to 32GB, which we just a 32-byte therein for storing 8 bytes to prefix ID .
But the memory will give rise to 2 deficiencies in other areas:
- Each file path inserting operation becomes Directory Table, File Table multiple insertion table, but if the follow-up of the parent directory already exists, the insertion operation of the parent directory may be omitted, close to a single insertion operation.
- Find require progressively traversing the path of the file table information. All because there is no record directory entry in the Table, but cut off the split deposit, so it is necessary to find step by step down to get the final file directory information. Not as good as the traditional single directory entry is presented to the namespace of the lookup performance efficiency.
But the structure of the way the whole, the above mentioned new FS Namespace nothing more than trade off system in space and time, space and must be saved, the efficiency will lower a little bit.
Ozone FS Namespace concurrency control
For Ozone FS Namespace may be on the order of a few very large scale storage of files in the directory in the actual usage scenarios, then there is for concurrency control FS Namespace can not be overlooked up.
The current program is the community without considering the operating conditions recursively delete a directory, concurrency control can be done Directory directory level. Simply put, all my operational control at the directory level Directory is operating, which are basically the minimum level of lock granularity level. But the control for the closing of this there will rename operation between cross-directory, in order to ensure atomicity, also we need to introduce a global lock. In general, is the assumption that there is no global lock control, they conduct their operations in accordance with the updated directory level locks.
Quote
[1]https://issues.apache.org/jira/browse/HDDS-2939 . Ozone FS namespace
