Z-Order was first proposed in 1966 to map multidimensional data to one dimension. With the development of database technology, this mapping method has been applied to database technology due to its characteristics, especially in the era of big data. Mentioned, it has applications in hudi and iceberg. This article will introduce the use of Z-Order in the database field, analyze its usage scenarios, and finally compare related technologies in multiple database fields to obtain the characteristics of Z-Order.
Part 1 Z-Order in the Database Field
In the database field, the application of Z-Order can be divided into two stages: OLTP and OLAP stage. Due to the different scenarios in these two phases, the storage engine cannot be unified in the data stacking mechanism, which in turn affects the use of Z-Order. But whether it's OLTP or OLAP, the motivation to use Z-Order is the same. This section analyzes the motivation for using Z-Order, and then describes the use of Z-Order at different stages separately.
1.1 The leftmost matching principle
CREATE TABLE members(
id int not null,
age int not null,
viplevel int not null;
……
constraint PK_members primary key(id,viplevel) // *
);
SELECT avg(age) FROM members WHERE id >= 1 and id <= 3; // 1
SELECT avg(age) FROM members WHERE viplevel >= 1 and viplevel <= 3; // 2Code Listing 1 Table structure and query statement
Consider the query statement and the corresponding table structure in Listing 1. The primary key in the members table is a composite primary key, and the database will do a B+ tree index on the primary key. The essence of B+ tree index is to arrange data in natural order to improve the performance of range query . Therefore, when executing the statement //1 in the code listing, since the primary key has been indexed by the B+ tree, the data with ids between [1, 3] has been placed together continuously, so the query can be completed relatively efficiently .
Figure 1 Impact of Compound Indexes
However, when the // statement number 2 is executed, a range query is performed on the viplevel. Since the viplevel is on the right in the index, it must be sorted by id first, and data with the same id will be sorted by viplevel. Therefore, the data with viplevel between [1, 3] is not clustered together.
Figure 1 depicts a possible ordering that can illustrate this situation. In Figure 1, the data of ID ∈ [1, 3] are stacked together consecutively, so the query can be completed quickly with a single match. As the second sort key of the index, viplevel needs to give priority to ensuring the order of the first sort key. Therefore, the overall data is divided into 4 areas, and it is necessary to traverse the index items to obtain all the data.
This phenomenon reflects the leftmost matching principle of database query . When the database matches a B+ tree type index, the index will be hit only when the query condition exactly matches the leftmost column or multiple consecutive columns, otherwise the index cannot be hit. When the leftmost principle is not satisfied, the index has little effect on the improvement of query performance.
1.2 Z-Order motivation
Through its special sorting mechanism, the Z-Order index can effectively avoid the leftmost matching principle brought by the B+ tree. The purpose of the data index is to solve the problem that the index of the B+ tree type fails when the leftmost matching principle is not satisfied.
Traditional B+ trees use natural ordering when sorting data, so when a compound index is encountered, the sort keys are processed in left-to-right order. First complete the sorting of the leftmost sorting key, and then complete the sorting of the subsequent sorting keys on this basis. This leads to the leftmost matching principle.
Figure 2 Schematic diagram of Z-Order
The Z-Order is to sort two or more sort keys at the same time, and arrange the data in the order of the interval. It can be simply understood that when the two attributes of AB are indexed by Z-Order, it is assumed that the two attributes of AB are arranged from small to {a1, a2, a3,...,an} and {b1,b2,b3,...,bn} } The first data is the smallest AB at the same time (a1, b1), the second data is the smallest A and B is the smallest (a2, b1), and the third data is the B with the smallest A in the remaining AB The second smallest is (a1,b2), the fourth data is the smallest of the remaining AB at the same time, which is (a2,b2)... Figure 2 shows an example of this situation, the figures in the table in 2(a) is the ordering number, called Z-ORDER because its order is similar to the letter Z.
Figure 2(2) shows the order of Z-Order in three dimensions. Z-Order can be extended to N-dimensional scenarios to provide indexes for composite primary keys of more dimensions.
1.3 OLTP
Due to its special data aggregation characteristics, Z-Order solves the failure problem of the B+ tree type joint index when it does not satisfy the leftmost principle, so it can be used in joint indexes in traditional databases. But in fact, not many databases actually use this indexing technology, and most transaction database indexes still use B+ tree indexes to solve range query problems.
In the OLTP era, Z-Order has not been applied on a large scale. A very important reason is that in the storage engine in the OLTP era, data is generally written to disk in transaction order. In this case, even if the data is indexed using Z-Order, the clustering effect of the data on the index is guaranteed. But in fact, data storage on disk is still discontinuous. Therefore, the application of Z-Order only reduces the time of traversing the index. After obtaining the index information, it is still necessary to obtain the real data from the discontinuous disk. That is , the acceleration effect brought by Z-Order cannot directly affect the IO time of the disk.
At the same time, since the data is not continuous on the physical disk, for transactional databases, multiple B+ tree indexes can be established for the composite index to solve this problem, and it is not necessary to use a complex Z-Order index.
In addition to the two reasons mentioned above, there is a more core reason: Z-Order does improve the indexing effect of B+ tree objectively, but it also pays some costs. This content will be analyzed in the third part of this article. It is precisely because the price paid by Z-Order is not proportional to the benefits it brings, so it is only a flash in the pan at the stage of transactional database.
1.4 OLAP
In the OLAP era, Z-Order re-entered people's field of vision. In the OLAP stage, since there is no need to provide complex transaction support, the design of the storage engine is very different from the transaction database. One of the important features is that the OLAP storage engine does not need to stack data in the order in which it was inserted. The data can be stacked according to different sorting strategies. This feature makes Z-Order fundamentally solve two small problems in the OLTP stage.
The first problem is that the acceleration effect brought by Z-Order cannot directly affect the IO time of the disk. This problem is essentially due to the index order not equal to the physical order of the data in the storage engine. In the OLAP database, the physical storage order of the OLAP database can be completely arranged according to the index order. Therefore, the acceleration effect obtained by Z-Order can directly affect the IO time of the disk, thereby greatly improving the performance.
The second problem is that a transactional database can create multiple ordered indexes on multiple sort keys to achieve the same speedup as Z-Order. This optimization is not possible in OLAP. Because there can only be one physical order of data on disk. If it is sorted according to A, it can no longer be sorted according to B, and if it is sorted according to B, it can no longer be sorted according to A. Because the physical order of the data is inconsistent with the index order itself, the transaction database can achieve multiple sequential indexes coexisting. It is precisely because the two orders are inconsistent that the acceleration effect of multiple sequential indexes coexisting is consistent. For OLAP databases, keeping the physical order and index order consistent can improve query performance. Under this premise, there can only be one physical order. Therefore, for compound indexes, the use of Z-Order is greatly increased.
To sum up, in the OLAP era, since the underlying storage engine does not need to stack data in transaction order like a transactional database, the physical order and index order can be kept consistent. Therefore, several major problems of Z-Order in the OLTP database era no longer exist. exist. Z-Order began to exert force in the OLAP stage.
Part II Z-Order Efficiency Analysis
The first part introduces the reader to the problem-solving and application of Z-Order at different stages. This chapter will analyze the acceleration efficiency of Z-Order based on the example in Figure 2.
2.1 Query according to A
Figure 2(3) shows an example of querying based on the A field, that is, executing the statement where a >= a1 and a <= a2. Under this query condition, it is only necessary to read the general data in the shaded part at most, and the data outside the shaded part cannot have data that meets the condition, so the data outside the shaded part can be skipped. In other words, after using Z-Order, a range query on A can skip at least 50% of the data.
2.2 Query according to B
Figure 2(4) shows an example of querying based on the B field, that is, the statement where b >= b1 and b <= b2 is executed. As in the case of 2.1, it is only necessary to read the data of the shaded part, and the data of the non-shaded part can be skipped. Also skipped 50% of the data. But due to
2.3 Summary
Through the example in Figure 2, it is not difficult to find that for a two-dimensional query condition, no matter whether the range query is performed on A or B, at least 50% of the data can be filtered out. If it is put into the big data scenario of OLAP, it is a very considerable benefit. Extending to the general situation, the data sorted by Z-Order can guarantee at least a part of the data, and the filtering effect is inversely proportional to the coverage of the query conditions. The smaller the query scope, the more obvious the filtering effect. Unlike a single sequential index, the filtering effect is unpredictable, and the worst case may require traversing the data.
Part III Z-Order Defects
Architecture means that there are gains and losses, and architecture analysis without analyzing the cost is incomplete. This chapter will analyze the flaws of Z-Order.
Table 1 Two different sequential data stacking methods
| serial number | (A,B) natural order | Z-Order order |
| 0 | a1, b1 | a1, b1 |
| 1 | a1,b2 | a1,b2 |
| 2 | a1, b3 | a2,b1 |
| 3 | a1, b4 | a2,b2 |
| 4 | a2,b1 | a1, b3 |
| 5 | a2,b2 | a1, b4 |
| 6 | a3,b3 | a2,b3 |
| 7 | a4, b4 | a2,b4 |
Table 1 shows the data stacking methods in two different orders in the example in Figure 2. The deepened cells in the table represent the data returned when executing where A=a1. It can be clearly seen that the Z-Order order is compared with the natural order of (A, B). In the case of satisfying the leftmost principle, due to the discontinuity of A, one more disk IO needs to be performed, which slows down satisfying the leftmost principle. Query speed in principle scenarios.
It should be noted that the data in Table 1 is only to illustrate the principle. In fact, due to the IO mechanism of the operating system, the gap is almost 0 for such a small amount of data in the example. But in the big data scenarios that real OLAP faces, this phenomenon will be very obvious.
Indeed, after using the Z-Order sequence, only 50% of the data can be skipped regardless of whether A is used as the query condition or B is used as the query condition. However, in the case that the query satisfies the leftmost principle, because Z-Order changes the ordered data in the original natural order into disorder, it reduces the query efficiency under the leftmost principle. This phenomenon is not obvious in the case of small data, but when the amount of data increases, the impact of this phenomenon will become larger and larger.
Part IV Summary and Recommendations
Z-Order essentially sacrifices the query performance of the primary sort key in exchange for the improvement of the query performance of the secondary sort key . Objectively reduces the modeling ability requirements of database engineers (using Z-Order can obtain not bad query performance in any scenario, and does not require database engineers to optimize data tables according to business). At the same time, it is more suitable for some scenarios where data requirements are changeable and it is impossible to determine which column is more commonly used in advance.
At present, hudi and iceberg have some solutions for applying Z-Order. Readers should fully understand your data requirements when applying the Z-Order solution. Z-Order does not necessarily improve query speed in all cases. Therefore, if the reader's data needs are relatively certain, it is recommended that the reader use the most primitive natural order. Objectively speaking, Z-Order is a moderate solution that sacrifices extreme performance. It has no extreme advantages and no obvious disadvantages. This is why hudi and iceberge only provide solutions, but do not use Z-Order as the default index. Readers should fully consider whether you really need such a moderate solution.
Of course, if the reader encounters a lot of data requirements that do not meet the leftmost principle and cannot be fixed, using Z-Order at this time will bring you a surprise. It cannot bring you the ultimate query speed, but it can You bring average not slow speeds. It will not be like ClickHouse, once the leftmost principle is not satisfied, the performance will drop sharply, even the performance gap of tens or hundreds of times before (there is also a solution, but it does not rely on zorder).