Databend is a modern cloud data warehouse. Designed for flexibility and efficiency to support your large-scale analysis needs. Free and open source. Experience the cloud service now: https://app.databend.cn .
What's On In Databend
Explore Databend's new developments this week and encounter Databend that is closer to your heart.
aggregate index
Databend recently introduced aggregate indexes to improve query performance, especially when involving aggregate queries such as MIN, MAX, and SUM. Aggregated indexes use techniques such as precomputation and independent storage of query results, thereby eliminating the need to scan the entire table and speeding up data retrieval.
In addition, this feature also comes with a refresh mechanism that supports updating and saving the latest query results on demand to maintain data accuracy and reliability. Databend recommends manually refreshing the aggregate index before executing related queries to retrieve the latest data; if using Databend Cloud, automatic refresh of the aggregate index is supported.
-- Create an aggregating index
CREATE AGGREGATING INDEX my_agg_index AS SELECT MIN(a), MAX(c) FROM agg;
-- Refresh the aggregating index
REFRESH AGGREGATING INDEX my_agg_index;
Aggregated indexes require upgrading to Enterprise Edition. For upgrade information, please contact the Databend team .
If you would like to learn more, check out the resources listed below.
Code Corner
Let’s explore code snippets or projects in Databend and the surrounding ecosystem.
MERGE INTO assembly line panorama
Some time ago, Databend implemented MERGE INTOsupport for statements to provide more comprehensive data maintenance capabilities. Friends who are interested in the principles can view MERGE INTOthe pipeline panorama below.
+-------------------+
+-----------------------------+ output_port_row_id | |
+-----------------------+ Matched | +------------------------>-ResizeProcessor(1)+---------------+
| +---+--------------->| MatchedSplitProcessor | | | |
| | | | +----------+ +-------------------+ |
+----------------------+ | +---+ +-----------------------------+ | |
| MergeIntoSource +---------->|MergeIntoSplitProcessor| output_port_updated |
+----------------------+ | +---+ +-----------------------------+ | +-------------------+ |
| | | NotMatched | | | | | |
| +---+--------------->| MergeIntoNotMatchedProcessor+----------+------------->-ResizeProcessor(1)+-----------+ |
+-----------------------+ | | | | | |
+-----------------------------+ +-------------------+ | |
| |
| |
| |
| |
| |
+-------------------------------------------------+ | |
| | | |
| | | |
+--------------------------+ +-------------------------+ | ++---------------------------+ | +--------------------------------------+ | |
+---------+ TransformSerializeSegment<--------+ TransformSerializeBlock <-----+---------+|TransformAddComputedColumns|<---------+-----+TransformResortAddOnWithoutSourceSchema<-+ |
| +--------------------------+ +-------------------------+ | ++---------------------------+ | +--------------------------------------+ |
| | | |
| | | |
| | | |
| | | |
| +---------------+ +------------------------------+ | ++---------------+ | +---------------+ |
+----------+ TransformDummy|<----------------+ AsyncAccumulatingTransformer <-+---------------+|TransformDummy |<---------------+---------------+TransformDummy <------------------+
| +---------------+ +------------------------------+ | ++---------------+ | +---------------+
| | |
| | If it includes 'computed', this section |
| | of code will be executed, otherwise it won't |
| | |
| -+-------------------------------------------------+
|
|
|
| +------------------+ +-----------------------+ +-----------+
+------->|ResizeProcessor(1)+----------->|TableMutationAggregator+------->|CommitSink |
+------------------+ +-----------------------+ +-----------+
If you would like to learn more, check out the resources listed below.
Highlights
Here are some noteworthy events, maybe you can find something of interest.
- Implement automatic aggregation and automatic compact support for MERGE INTO.
- SQLsmith can now cover DELETE, UPDATE, ALTER TABLE and CAST.
- Added semi-structured data processing functions such as
json_eachand .json_array_elements - Newly added
to_week_of_yearanddate_partequal time and date functions, you can read Docs | Date & Time Functions for more details. - Read Sending IoT Stream Data to Databend with LF Edge eKuiper to learn how to integrate Databend with eKuiper to meet the growing needs of IoT data analysis.
What's Up Next
We are always open to cutting-edge technologies and innovative ideas, and welcome you to join the community and inject vitality into Databend.
Enhance role-based access control capabilities
Currently, Databend's access control system consists of two parts: role-based access control (RBAC) and discretionary access control (DAC). However, in order to further improve this system, some follow-up work needs to be completed.
We plan to support permission checks for more uncovered resources in 2023 Q4 and develop corresponding permission definition guidelines.
Issue #13207 | Tracking: RBAC improvement plan in 2023 Q4
If you are interested in this topic, you can try to solve some of the problems or participate in discussions and PR reviews. Alternatively, you can click on https://link.databend.rs/im-feeling-lucky to pick a random question, good luck!
Changelog
Check out the changelog for Databend's daily builds to stay up to date on the latest developments.
Address: https://github.com/datafuselabs/databend/releases
Contributors
A big thank you to the contributors for their great work this week.
Connect With Us
Databend is an open source, flexible, low-cost, new data warehouse based on object storage that can also perform real-time analysis. We look forward to your attention and exploring cloud native data warehouse solutions together to create a new generation of open source Data Cloud.
Lei Jun announced the complete system architecture of Xiaomi's ThePaper OS, saying that the bottom layer has been completely restructured. Yuque announced the cause of the failure and repair process on October 23. Microsoft CEO Nadella: Abandoning Windows Phone and mobile business was a wrong decision. Both Java 11 and Java 17 usage rates exceeded Java 8 Hugging Face was restricted from accessing. The Yuque network outage lasted for about 10 hours and has now returned to normal. Oracle launched Java development extensions for Visual Studio Code . The National Data Administration officially unveiled Musk: Donate 1 billion if Wikipedia is renamed "Weiji Encyclopedia" USDMySQL 8.2.0 GA