Author: Li Haixiang Tencent TDSQL expert engineer
"Under the distributed background, how to achieve double consistency (transaction consistency, distributed consistency) and improve the processing efficiency of distributed transaction clusters?" Tencent TDSQL database has been committed to basic research and innovation for a long time, and continues to obtain key technologies breakthrough.
On December 21, 2020, at the 11th DTCC (China Database Technology Conference) conference, Li Haixiang, an expert engineer of Tencent TDSQL database, shared the core technology of the database field-the core of distributed transaction processing technology- multi-level consistency technology . While following the ACID characteristics, this technology makes the transaction processing technology comply with the CAP principle, and expands it from the "strict serialization" technology at the theoretical level, and further proposes a variety of complete technologies with strong consistency . At the same time, the technology has achieved high performance at the engineering practice level, which is 4+ times the performance of similar products Spaneer, 2+ times that of CorchroachDB, and 3+ times that of Greenplum in high-concurrency scenarios.
This means that TDSQL has realized the first high-performance and strong consistency engineering system in the database industry, which is of great significance to promote the core technology of distributed database and industrial applications.
0. Background of the problem
Scientific research begins with problems. In the field of distributed transaction processing, there are many important issues, far beyond what 2PC technology can cover. For example, Tencent TDSQL is currently solving a complex and compound problem as follows:
The problem that transactional databases must solve is the problem of data consistency. This problem is a problem of C of the ACID characteristics of transactions. If C is not satisfied, there will be a variety of data abnormalities , such as dirty reads, non-repeatable reads, phantom reads, read partial order, write partial order and other data abnormalities.
The distributed system follows the CAP theory, where C is also "consistency", which is the consistency of data read between operations. If the C of CAP is not satisfied, there will be abnormal consistency problems such as Steal read, Immortal write, and Causal reverse (as shown in Figure 2).
The distributed transactional database faces the above two types of problems at the same time, as shown in Figure 1. In distributed transactional databases, it is difficult to solve these two types of problems separately (for example, many databases like Oracle do not provide serialization and cannot ensure that data exceptions do not occur, and NoSQL systems are lost to ensure A and P in CAP. C) Not to mention that the synergy is solved.
Figure 1 Background and problems of multi-level consistency research
Figure 2 Problems in distributed systems
As a distributed transactional database, TDSQL is faced with a composite and complex technical problem of the above two problems (distributed consistency + transaction consistency). The difficulty lies in how to achieve high accuracy under the background of ensuring correctness. Efficiency, which directly affects the transaction throughput of the distributed transaction database (Transaction throughput).
1. The complexity of academic research and issues
In academia, a solution to the aforementioned problems is called " strict serialization ". This technology can solve the problems caused by transaction consistency and CAP consistency. As shown in Figure 3 (document "Highly available transactions: Virtues and limitations"), the root of the tree is the junction of transaction consistency and distributed consistency, which is strictly serializable. The left subtree (transaction consistency) and the right subtree (distributed consistency) of the tree are two independent systems, and there is no relationship between them.
Figure 3 Academic research on the "junction point of transaction consistency and distributed consistency" (1)
It can also be seen from Figure 3 that there are many types of transaction consistency isolation levels for the left subtree, and many types of distributed consistency for the right subtree. Therefore, it is necessary to systematically study the consistency and distribution of transactions at various levels. The consistency of the system can better analyze and solve problems (instead of only considering the only combination of tree roots). Figure 4 tells us that there are many distributed consistency, and Figure 4 summarizes about 60 types, which brings complexity to the study of this problem.
Figure 4 Distributed consistency diagram
Although the problem of merging distributed consistency has great complexity, such obstacles cannot stop our enthusiasm and pace of in-depth research on the problem. During TDSQL's forward exploration, as shown in Figure 5, we saw the direction and dawn (in the red box). The blue box in Figure 5 is the consistency level related to isolation and distributed consistency in the transaction. The "Isolation (transactional)" column on the horizontal axis corresponds to the isolation level of the transaction, and they intersect The place is surrounded by a red box, but it is "blank." This blank space is not reached by the research institute, and it is this no man's land that gives TDSQL research space and stage.
Figure 5 Academic research on the "junction point of transaction consistency and distributed consistency" (2)
In addition, the industry often mentions "strong consistency", but the concept is confused and unclear. The document "Consistency in Non-Transactional Distributed Storage Systems" points out the following and clarifies the meaning of strong consistency:
In a hybrid consistent system, strong operations are guaranteed to be seen in some sequential orderby all processes(as in sequential consistency), while weak operations aredesigned to be fast, and they eventually become visible by all processes (muchlike in eventual consistency). Weak operations are only guaranteed to beordered according to their interleaving with strong operations: if twooperations belong to the same session and one of them is strong, then theirrelative order of invocation is respected and visible by all processes.
Therefore, whether it can be seen by "all processes" is the criterion for judging strong consistency. In distributed consistency, only linear consistency and sequential consistency satisfy the meaning of strong consistency. The combination of transactional consistency, need to be serialized to meet strong consistency can be called a distributed transaction .
To sum up, the problem that TDSQL is studying is the internal relationship between ACID C, CAP C, and strong consistency . They have two dimensions: one is how to ensure correctness when considering these factors, and the other is It is how to ensure that the implemented system has good performance under the influence of these factors. If there is no good performance, it is difficult to provide even strong consistency, so achieving high performance in the context of strong consistency becomes a challenge.
2. Realization and problems in the industry
In the industry, currently only Google's Spanner system has achieved "strict serialization", which Spanner calls "external consistency" ("Spanner: Googles globally distributed database"). The technology was published in 2012, making the research and practice of the database industry from the NoSQL era into the NewSQL era, which has epoch-making significance. The technical essence of its realization is to combine the linear consistency in the distributed system and the serialization of transaction consistency, thereby realizing "strict serialization".
Figure 6 The industry's practice of "the junction of transaction consistency and distributed consistency"
However, there is a problem in the implementation of Spanner: the correctness can be guaranteed, but the performance is low (as shown in the calculation formula of transaction throughput in Figure 6). Therefore, the system is symbolic. In practice, the performance does not meet the requirements for transaction processing scenarios that need to be at the financial level.
3. Tencent TDSQL's multi-level consistency technology
TDSQL deeply studied the distributed consistency problem and solved some technical problems. For example, as shown in Figure 7, linear consistency has multiple sub-categories. When combining linear consistency and transaction consistency, should there be multiple levels of "strict serialization"? Another example is why sequential consistency can be combined with serialization to ensure strong consistency? These problems need to be proved at the theoretical level before they can be transferred to the engineering system to guide the implementation of the project.
Figure 7 Definition of linear consistency
After in-depth study of ACID's C and CAP's C and strong consistency, the TDSQL team adopted consistency modeling technology from multiple perspectives such as concepts, models, model-based operations, transaction model frameworks, data structures, and efficient algorithm design. , Based on TDSQL and Greenplum to achieve "multi-level consistency" technology (based on Greenplum is to facilitate the comparison with the industry's product implementation to find problems), this technology in terms of correctness, provides a linear serializable that meets strong consistency The definition and proof of the serializable level of integration and sequence have eliminated the aforementioned data anomalies in transaction consistency, such as partial write order, and distributed anomalies in distributed consistency, such as Steal read. In terms of performance, it has excellent performance.
This technology has been compared and tested the performance of Spanner on Google Cloud, and compared and tested the well-known NewSQL system CorchroachDB in the industry, and compared and tested the native Greenplum, and obtained good results, as shown in Figure 8 (Multi-level consistency of TDSQL : SER-L, linear serializable; SER-S, sequential serializable), verified and compared the performance of TDSQL multi-level consistency from multiple angles, and part of the experimental data is given in Figure 8. The experiment in Figure 8 is divided into 4 levels for comparison:
Comparing the performance of Spanner: In the subgraph in the upper left corner of Figure 8, the multi-level consistency of TDSQL is 4+ times that of Spanner with the same cost-effective resources.
Comparing the performance of CorchroachDB with the subgraph in the lower left corner of Figure 8, under the same hardware test environment, the multi-level consistency of TDSQL is 2+ times that of CorchroachDB.
Comparing the performance of native Greenplum: In the right sub-graph of Figure 8, under the hardware test environment, the multi-level consistency of TDSQL is 3+ times the performance of native Greenplum.
In the presence of a certain degree of network delay, the performance of TDSQL's multi-level consistency: the right subgraph of Figure 8, also tested in a distributed environment, when the network has a delay of 1.5ms and 5ms, high concurrency scenarios Under the TDSQL multi-level consistency technology, the performance of Greenplum's RC isolation level is still better than that of the Greenplum RC isolation level without delay, which shows that the isolation level with higher consistency can achieve better performance (the early version of CorchroachDB realized snapshot isolation And serializability, but in the new version, only the stricter serializability level is retained. Can you think about why?).
Figure 8 TDSQL multi-level consistency technology performance verification
4. The significance of TDSQL multi-level consistency work
The research and practical work of TDSQL multi-level consistency has several values:
The first engineering system to achieve multiple strong consistency: In the currently known systems, only Spanner and TDSQL achieve strong consistency; only TDSQL implements multiple levels of strong consistency systems. This serves as an example for the in-depth practice of the strong consistency system. We can know from this that the theory in this direction is feasible in engineering practice .
The first high-performance and strong-consistent engineering system: a system that ensures correctness may not be able to be effectively used in the actual environment, because high performance is the core indicator of a system. And TDSQL under a variety of strong consistency has performance that can meet practical applications. In the absence of any data anomalies and consistency anomalies, users no longer need to worry about unexpected problems with data or operations, but can enjoy an excellent performance experience, which shows that the technology is feasible in engineering practice. .
We look forward to the beginning of the curtain, and the real strong consistency system will spring up...
5. Future work
The good news is that this technology will most likely be open source in the future.
Some of TDSQL's transaction processing technology is already open source. You can find the 3TS open source project on github. This project focuses on transaction processing technology and will deeply involve all aspects related to transaction processing technology. It will be valuable and in-depth Research and practice. Currently, 13 concurrent access control algorithms have been open sourced under the project, and detailed documentation has been provided.
Figure 9 TDSQL's future technical development direction
Tencent TDSQL and the Ministry of Education Key Laboratory of Data Engineering and Knowledge Engineering of Renmin University of China expect to use their own practical actions to allow us to complete the project together to make meaningful contributions to the academic and engineering circles and to promote database technology forward development of. The road is long and long, we are working hard, welcome to join TDSQL and swim in the ocean of advanced technology.
December 23, 19:30
Xiao Meng from the TAPD team
Will be combined with specific scenarios in the development process to share
How to Realize Team Agile R&D Measurement Based on TAPD