Xline v0.3.0: A distributed KV store for metadata management

What is Xline? Why do we do Xline?

Xline is a distributed KV storage for metadata management based on the Curp protocol. Most of the existing distributed KV storage adopts the Raft consensus protocol, which requires two RTTs to complete a request. When deployed in a single data center, the latency between nodes is low, so there is no large impact on performance.

However, when deployed across data centers, the latency between nodes may be tens or hundreds of milliseconds, at which point the Raft protocol will become a performance bottleneck. Curp protocol is designed to solve this problem. It can improve performance by reducing one RTT without command conflicts. Therefore, Xline aims to achieve high-performance data access and strong consistency across data center scenarios.

What new features does V0.3.0 have?

The main change of this version is to introduce a persistence layer . Since the change itself is relatively large, a separate version is extracted. The improvements of the new version include the following:

Features:

Implement a persistent storage layer for persistence, including:
Implement a storage engine layer to abstract specific storage engines, such as rocksdb, and enable upper-layer storage functions (#185, #187).
Enable recovery mechanism for Curp and Xline (#194, #184).

Fix bugs:

Fix concurrent cmd order error (#197)

Since the previous storage is done in memory, if the process crashes, data recovery will take a long time. Based on this consideration, Xline now introduces a persistence layer that stores data on disk. Similarly, based on the question "whether the performance test based on memory is convincing" received before, after careful consideration, we decided to make a benchmark on this basis, and the results are expected to be displayed in v0.3.1 .

Welcome to participate in the Xline project

We welcome any positive contributions to Xline. At present, some tasks on GitHub do not require a deep understanding of the Curp protocol or the Xline project, but only need to understand the API and Rust language. Even if you are just getting started and want to use the Rust language in an open source project, the community will provide guidance and help to guide you to better participate in the project. The construction of Xline requires the participation of everyone who is interested and willing to pay, and we look forward to your joining.

Recommendations for previous articles:

Xline v0.2.0: A distributed KV storage for metadata management https://blog.csdn.net/DatenLord/article/details/129090170?spm=1001.2014.3001.5501 DatenLord | Xline Geo-distributed KV Storage https:/ /blog.csdn.net/DatenLord/article/details/126301572?spm=1001.2014.3001.5501

DatenLord focuses on the next generation of cloud computing - the infrastructure technology of "Sky Computing", and is committed to broadening the boundaries of cloud computing. DatenLord, a new generation of open source cross-cloud storage platform created by Datan Technology, breaks through cloud-cloud barriers through deep integration of software and hardware, realizes unlimited cross-cloud storage and cross-cloud connectivity, and establishes a unified storage access mechanism for massive remote and heterogeneous data. Cloud applications provide high-performance secure storage support. To meet the needs of customers in different industries for high-performance access to massive data across clouds and data centers.

Public account : Datan Technology DatenLord

Zhihu account: Datan Technology DatenLord - Zhihu

Station B : https://space.bilibili.com/2017027518