Leveldb source code analysis 1.2-basic overview of the overall architecture

introduction

Leveldb adopts LSM tree (Log Structured-Merge Tree) to achieve, by giving up part of the read performance in exchange for the improvement of write performance. Convert the traditional random write file into a sequential append write file. The entire source code implementation of leveldb has many clever designs. For example, bucket hashing is used to reduce the probability of thread mutual exclusion, and the design of lru_ and in_use_ in the Cache to avoid accidental clearing of data objects that are being used, etc., have been used when I wrote other middleware before.

1 Overall architecture

The core architecture of the entire leveldb is as follows:

it is divided into two layers: the upper layer is the memory architecture, and the lower layer is the file system architecture.
When writing data: first, append the data to the log file; then, add it to the MemTable cache; when the MemTable exceeds a certain threshold, it will be converted to an immutable MemTable, and then written to the sstable file by the background thread ; The
sstable file uses hierarchical storage, which is why it is called leveldb. The manifest file records which files are included in each layer; when there is an immutable MemTable or a certain layer reaches a certain condition (for example, the total file size exceeds a certain value), it will be executed Merge to lower levels, and then a new manifest file will be generated; the current file records the name of the current manifest file.
When reading data: query MemTable first, return if there is, otherwise, perform subsequent query; then query immutable MemTable, return if there is, otherwise, perform subsequent query; if not found after the first two steps, query will be given priority Whether the sstable file of level 0 exists, if there is, return it, otherwise, execute the subsequent query; finally query the level 1 to 6 in sequence, if it exists, return, otherwise return an empty result.

2 Design term

2.1 UserKey与InternalKey

In the implementation of leveldb, there are two concepts of Key: UserKey and InternalKey. UserKey is the key value passed in by the caller of leveldb in put/get/delete operations; since the operations on the same UserKey are ordered, InternalKey is the aggregate object of the internal UserKey plus the operation sequence SequenceNumber and the operation type ValueType. In the sstable file, InternalKey is sorted by UserKey first. If UserKey is equal, it is sorted by SequenceNumber in descending order.

2.2 Slice

Slice, also known as slice, is a wrapper class for fragments of byte arrays. It does not manage the life cycle of the memory occupied by the character array, and is mainly used to prevent the copy of byte data from affecting performance.