Understand the principle and use of Databend Cluster key

Databend Cluster Key means that Databend can be sorted and stored according to the declared key. It is mainly used for users who have a high time response and are willing to perform sorting operations for this cluster key. Databend only supports one Cluster key, and the Cluster key can contain multiple columns and expressions.

basic grammar

-- 语法：
alter table T cluster by(c1, fun(c2));

-- 例如： 
alter table T cluster by(user_id);  -- 指定数据按 user_id 排序存储

-- 日志场景 按 msg_id, 小时 排序存储
alter table T cluster by(msg_id, to_yyyymmddhh(c_timestamp));

-- 强制数据排序
optimize table T compact;
alter table T recluster final;  -- 全局排序, 建议第一次创建 Cluster key 后使用，后期如果遇到性能退化，也可以再次使用

More about Databend Cluster key syntax reference:

https://databend.rs/doc/sql-commands/ddl/clusterkey/

Precautions for use

Currently, Databend uses

copy into
replace into

When writing data in these two ways, compact and recluster operations are automatically performed.

What you need to know about Databend Cluster Key:

Data partitions in Databend are organized according to: block_size_threshold (default: 100M) or row_per_block (default 1 million). If either of the two is reached, a new Block will be generated.
The newly generated Block will be stored in order according to the defined cluster key. When the key's min = max, the block is a constant_block. At the same time, the cluster key does not guarantee global ordering.
There may be overlapping intervals between multiple blocks, such as cluster by (age)

The overlap of different intervals forms different depths, such as the picture above:

select * from T where age >30 and age <35;

For such a query, the depth to be found is 3, which is 3 Blocks.

Therefore, the smaller the average depth of the overlapping block-partitions of the specified columns in the table, the better. As follows:

-- 可以通过 clustering_information('db','tbname') 查看该表的 Cluster 信息
 select * from clustering_information('wubx','sbtest10w')\G;
*************************** 1. row ***************************
        cluster_by_keys: (id)   -- 定义的 Cluster key
      total_block_count: 451    -- 当前有多少的 block
   constant_block_count: 0      -- min/max 相等 block， 也就说 block 中只包括一个（组） cluster_key 的值  
unclustered_block_count: 0      -- 还没 Cluster 的 Block
       average_overlaps: 2.1774   -- 在一个 Range 范围内，有多少个 block有重叠比率
          average_depth: 2.4612   -- cluster key 在分区的重叠分区数的平均深度
  block_depth_histogram: {"00001":32,"00002":217,"00003":164,"00004":38}
1 row in set (0.02 sec)
Read 1 rows, 448.00 B in 0.015 sec., 67.92 rows/sec., 29.71 KiB/sec.

The most important information in the result is "average_depth". The smaller the number, the better the clustering effect of the table. The above picture is: 2.46, which is a relatively good state (less than total_block_count * 0.1). block_depth_histogramTell more details about how many partitions there are at each depth. A table is clustered better if there are more partitions in lower depths. For example, "00004" : 38 means (3,4] has 38 blocks and 4 depths.

Other optimization suggestions

Generally speaking, after declaring Cluster key, interval query and point query will be greatly optimized.
If after declaring the cluster key, you still want to further improve the capabilities of enumeration or interval query, you can adjust the block size.

-- 把 Block 的大小修改为压缩前 50M ，行数不超过 10 万行
alter table T set options(row_per_block=100000,block_size_threshold=52428800);

About options check: https://databend.rs/doc/sql-reference/table-engines/fuse#options

Default data distribution:

Optimize the distribution of data in Blocks

create table sbtest10w like sbtest1;
alter table sbtest10w set options(row_per_block=100000,block_size_threshold=52428800);
insert into sbtest10w select * from sbtest1;

For particularly wide tables, it is recommended to access only the required columns in the query to reduce time overhead.

For complex SQL with a large number of aggregation operations, it is recommended to use a larger Block and row number.