On the afternoon of July 28, the second stop of the openGemini offline Meetup concluded successfully at Beijing Institute of Technology.

In this technical salon, the topic brought by the openGemini community is "Application of Data Structures and Algorithms in Databases". The community initiator shared the data structures and algorithms used in the openGemini time series database with the students of Yuwei University of Science and Technology. LSM-Tree is mainly explained from the aspects of meaning, structure, difference from B+tree, and optimization.

Below is the complete content shared in this session, full of useful information, don’t miss it~

Overview of data structures and algorithms used in time series databases

data structure

Array (including dynamic array)
Map (including HashMap, BitMap)
dictionary tree
Inverted index
Big and small heap
linked list
LSM tree

algorithm

word segmentation algorithm
Data compression algorithm
Sorting algorithm (heap sort, insertion, merge, fast)
Search algorithm (bisection, tree traversal)
Anomaly Detection/Prediction Algorithms (AI)
Data encoding and decoding algorithm
Memory reuse
Raft
Link reuse

LSM Tree is the core of modern NoSQL database storage engine

LSM-tree: Log-StructuredMerge-tree, not a tree, it is just a hierarchical structure for organizing data. It mainly solves the problem of how to write data efficiently in big data scenarios.

main idea:

Random writing is converted into sequential writing, making full use of every I/O to achieve efficient sequential writing of data.
Data hierarchical merging avoids the problem of data files being dispersed over time

Image source: https://anyview.fun/2022/09/23/%E4%B8%80%E6%96%87%E4%BA%86%E8%A7%A3lsm-tree/

Why do NoSQL databases collectively abandon B+tree?

Advantages of the B+ tree: Using the principle of locality, loading one node, which is exactly one Page, only requires one I/O read. The B+ tree is shorter and fatter, reducing the number of I/Os, so it is very efficient.

Disadvantages of B+ tree: If the primary key is not sequentially increasing, it will cause a large amount of data migration and space fragmentation each time data is inserted. Even if the primary keys increase in order, the distribution of a large number of write requests is still random, which may cause a large number of random disk reads and writes, resulting in increased disk IO time.

The time consumption of a disk IO is mainly composed of three parts: seek time + rotation delay + data transmission time

Optimization direction: Minimize seek time and rotation delay, that is, reduce random I/O

Summary: In a big data scenario, the amount of data is large and write concurrency is large. If B+ tree is used, the performance is too poor. LSM-tree mainly solves this problem and changes a large number of random writes to sequential writes.

Reference text:《The Design and Implementation of a Log-Structured File System》

How does LSM-tree ensure that data is written in order?

In order to improve IO performance, we need a mechanism that can turn random operations into sequential operations, so there is an LSM tree. LSM caches a large number of writes in the memory. When they accumulate to a certain extent, they are written to the file in batches. This allows multiple pieces of data to be written in one I/O, thereby greatly increasing the write operation, at the expense of sacrifice. Improved some read performance.

Implementation of LSM-tree in openGemini

openGemini caches a large amount of concurrently written time series data in buckets according to timeline keys in memory. Each bucket is called a Record. One Record stores the data of one timeline and is organized by columns. The cache is periodically persisted to disk files.

LSM-tree optimization

LSM-tree achieves better write throughput by reducing random I/O, but the background Compacttion (file merging) still brings large write I/O amplification, affecting write performance, and frequent writes are harmful to Disk life also plays a role. Therefore, there is no shortage of related research on how to reduce LSM write amplification and how to improve LSM query performance.

LSM-tree optimization method:

Reference paper: "LSM-based Storage Techniques: A Survey"

One of LSM-tree optimization methods: Bloom filter

Bloom filter: a long binary vector and a series of random mapping functions. When all the bits pointed to by the hash function are 1, it means that the target data exists, otherwise it does not exist.

Application scenario: When processing massive data, use Bloom filters to achieve deduplication. Compared with the deduplication function of Set collections, Bloom filters can save more than 90% of space.

When the Bloom filter determines that a value does not exist, then this value definitely does not exist

When it determines that a certain value exists, it is actually only possible that the value exists, and the probability of misjudgment is about 1%.

Thanks for reading

openGemini has evolved since 2019 and has accumulated a lot of practical experience in optimizing data structures and algorithms. The community provides technical guidance and related community development training for university developers. Students and developers are welcome to join the community to learn and participate in development. , Improve personal technical abilities.

The third phase of the meetup is in preparation~

Where will our story continue? Let us look forward to it together ✨...

The second stop of the meetup·BIT review is here! new knowledge get√