Author: Wang Xudong
Databend R&D Engineer
Importance of join order
Join order refers to determining the order in which multiple tables are joined when executing a SQL query. It is an important aspect of database query optimization and has an important impact on query performance and efficiency. Different join orders may have an order of magnitude impact on performance.
The optimizer optimizes the core process of join order
- join plan enumeration
- Estimating the size of the result based on statistical information (cardinality estimation)
- Bring the results in 2 into the cost model to calculate the cost of the enumeration plan (cost model)
In this article, we only care about the first step of join plan enumeration, which is the join reorder algorithm.
join reorder algorithm
Greedy heuristic algorithm
When there are too many tables that need to be joined (usually more than 10), the greedy algorithm is suitable. Its advantage is that it can quickly find a good join order.
Core idea: Expand from one table to N tables, select a table that minimizes the current cost each time, add it to the join tree, and build a left-deep tree.
The greedy algorithm also has many extensions. The main extension points are how to avoid local optimality and generate bushy trees.
Enumeration algorithm (top-down & bottom-up)
Two mainstream
- Top-down enumeration based on rule transformation can be implemented in a memorized manner in conjunction with the Top-down cascasde framework.
- DP-based bottom-up enumeration, a typical representative of the DPhyp algorithm, has the advantage of efficiently generating bushy trees.
Under normal circumstances, the database system will effectively combine greedy and enumeration, so that any number of table joins can obtain an effective join order within a reasonable time.
Databend join reorder status quo
The databend optimizer optimizes based on Rules, and each Rule matches the sub-tree in the plan through pattern. It is mainly divided into two stages, heuristic optimization and cost optimization based on the cascades framework. The two stages share a set of Rules.
After the heuristic phase optimization is completed, the DPhyp algorithm will be executed on the optimized plan to try to obtain the optimal join order. If Dphyp optimization fails, the optimal left-deep tree will be found in CBO. (This is not attempted in CBO. Bushy tree optimization, because if Dphyp has failed to optimize, then try to perform bushy tree optimization in CBO, the search space is likely to explode, such as tpcds 64)
Databend currently does not support the greedy algorithm (the next stage of roadmap will provide relevant support to handle extreme cases). It will first use the dphyp algorithm to get the optimal solution. If dphyp fails (there is a pattern in the query that is not suitable for the dphyper algorithm) , the left-deep tree will be obtained by using Top-down enumeration based on rule transformation in the cascades framework. If the number of tables is too large, such as more than ten, part of the search space will be given up in the dphyp algorithm for tradeoff.
The core definition and algorithm of Dphyp
hypergraph
A hypergraph is a tuple H = (V,E) consisting of a node set V and a hyperedge set E, where:
- V is a non-empty node set.
- E is a hyperedge set. A hyperedge is an unordered pair (u, v) of a non-empty subset (u ⊂ V) and (v ⊂ V) of V, and satisfies the additional condition u∩v = ∅.
With a hypergraph, connections between multiple nodes can be described.
For the above figure, their join condition is R1.a + R2.b + R3.c = R4.d + R5.e + R6.fso hyperedge is{R1, R2, R3} — {R4, R5, R6}
csg-cmp-pair
csg: connected-subgraph (connected subgraph)
cmp: connected-complement (connected complementary pair)
If there is no intersection between two csg and there is a hyperedge connection, one of them is the cmp of the other, and the two constitute a csg-cmp-pair. The core of the algorithm is to enumerate all csg-cmp-pairs through recursion without duplication. pair, find the csg-cmp-pair that contains all points with the smallest cost.
algorithm
Core of the algorithm: The nodes in the hypergraph are ordered. The nodes are iterated from back to front (decreasingly). Each node only considers itself and the nodes after it (with a larger sequence number) to find possible connected subgraphs and their connections. The complementary graph forms a csg-cmp-pair, and its cost is calculated and updated. After iterating to the smallest node, a csg-cmp-pair containing all points will be obtained, and the algorithm ends.
Algorithm process
- EmitCsg: Find the complementary connected subgraph of {v}
- a. If found, EmitCsgCmp
- b. EnumerateCmpRec: Extended complementary connected subgraph
- If the expanded complementary connected subgraph can form a csg-cmp-pair with {v}, then EmitCsgCmp
- Go back to b and continue to expand
- EnumerateCsgRec: extension {v}
- a. Get the expanded {v'}, execute 1 on {v'}
- b. Return to 2 and continue to expand
For core code and data structure definitions, please refer to:
Connect With Us
Databend is an open source, flexible, low-cost, new data warehouse based on object storage that can also perform real-time analysis. We look forward to your attention and exploring cloud native data warehouse solutions together to create a new generation of open source Data Cloud.
The author of the open source framework NanUI switched to selling steel, and the project was suspended. The first free list in the Apple App Store is the pornographic software TypeScript. It has just become popular, why do the big guys start to abandon it? TIOBE October list: Java has the biggest decline, C# is approaching Java Rust 1.73.0 Released A man was encouraged by his AI girlfriend to assassinate the Queen of England and was sentenced to nine years in prison Qt 6.6 officially released Reuters: RISC-V technology becomes the key to the Sino-US technology war New battlefield RISC-V: Not controlled by any single company or country, Lenovo plans to launch Android PC