Databend is a modern cloud data warehouse. Designed for flexibility and efficiency to support your large-scale analysis needs. Free and open source. Experience the cloud service now: https://app.databend.cn .
What's On In Databend
Explore Databend's new developments this week and encounter Databend that is closer to your heart.
Feature preview: Use SQL to manage background tasks
Previously, Databend introduced a background service (Serverless Background Service) to allow Databend to perform background single jobs or run cron jobs in daemon mode, thereby simplifying the complexity of data maintenance task management.
In order to facilitate the creation, management and maintenance of background tasks, Databend recently supports a series of SQL statements including CREATE TASK, ALTER TASKand .SHOW TASK
For example, the following SQL statement requires Databend to execute a task named MyTask1 at 6 a.m. every day (PST time zone) to insert (1, 2)and (3, 4)into table t.
CREATE TASK IF NOT EXISTS MyTask1 SCHEDULE = USING CRON '0 6 * * *' 'America/Los_Angeles' COMMENT = 'serverless + cron' AS insert into t (c1, c2) values (1, 2), (3, 4)
The background service needs to be upgraded to the enterprise version . For upgrade information, please contact the Databend team .
If you would like to learn more, check out the resources listed below.
- PR #13316 | feat: impl create task sql parser and planner
- PR #13344 | feat: add SQL syntax support for alter, execute, describe, show, drop task
Code Corner
Let’s explore code snippets or projects in Databend and the surrounding ecosystem.
Understanding configuration mapping in Databend
For complex database server programs like Databend, it is often necessary to support a large number of configurable options to help operation and maintenance personnel manage and tune the system according to actual usage needs. A recent blog post explains the mapping between command line options, environment variables, configuration files, and code in Databend.
Databend currently supports three configuration methods: command line, environment variables and configuration files, with decreasing priority.
- The command line is used to adjust a few conflicting configurations in the local environment.
- For K8S clusters, in order to flexibly change some configurations (such as feature switches), using environment variables may be a more elegant form.
- In general, it is recommended to use configuration files to record and manage various configurations.
It is worth noting that Databend uses opendal's built-in rollback mechanism to support environment variables like AWS_ACCESS_KEY_IDand AWS_SECRET_ACCESS_KEYthat are commonly used by cloud service developers but are not explicitly declared in the Databend configuration, simplifying the code and reducing user learning costs.
If you would like to learn more, check out the resources listed below.
Highlights
Here are some noteworthy events, maybe you can find something of interest.
- Added table function
fuse_encoding. - New string functions
splitandsplit_part. - SQLsmith supports MERGE INTO.
- databend-metactl supports viewing cluster status.
What's Up Next
We are always open to cutting-edge technologies and innovative ideas, and welcome you to join the community and inject vitality into Databend.
Support more types of comparisons for z3 solver
Previously, Databend introduced z3.rs to solve the constraint problem, that is, given some constraints, it seeks a set of solutions that satisfy the conditions. The Z3 solver is a theorem prover developed by Microsoft Research and is commonly used to solve SMT problems. However, currently Databend's z3 solution only supports comparison of integer types, and more types need to be covered.
For example, with support for string comparisons, select * from t1 left join t3 on t3.e = t1.a where t3.f >= '2000-08-23';in t3.f >= '2000-08-23'can be pushed down to t3the table.
Issue #13236 | Feature: z3 supports more type comparision
If you are interested in this topic, you can try to solve some of the problems or participate in discussions and PR reviews. Alternatively, you can click on https://link.databend.rs/im-feeling-lucky to pick a random question, good luck!
Changelog
Check out the changelog for Databend's daily builds to stay up to date on the latest developments.
Address: https://github.com/datafuselabs/databend/releases
Contributors
A big thank you to the contributors for their great work this week.
Connect With Us
Databend is an open source, flexible, low-cost, new data warehouse based on object storage that can also perform real-time analysis. We look forward to your attention and exploring cloud native data warehouse solutions together to create a new generation of open source Data Cloud.
Lei Jun announced the complete system architecture of Xiaomi's ThePaper OS, saying that the bottom layer has been completely restructured. Yuque announced the cause of the failure and repair process on October 23. Microsoft CEO Nadella: Abandoning Windows Phone and mobile business was a wrong decision. Both Java 11 and Java 17 usage rates exceeded Java 8 Hugging Face was restricted from accessing. The Yuque network outage lasted for about 10 hours and has now returned to normal. Oracle launched Java development extensions for Visual Studio Code . The National Data Administration officially unveiled Musk: Donate 1 billion if Wikipedia is renamed "Weiji Encyclopedia" USDMySQL 8.2.0 GA