Databend Open Source Weekly Issue 124

Databend is a modern cloud data warehouse. Designed for flexibility and efficiency to support your large-scale analysis needs. Free and open source. Experience the cloud service now: https://app.databend.cn .

What's On In Databend

Explore Databend's new developments this week and encounter Databend that is closer to your heart.

Added support for Delta and Iceberg table engines

Databend has added support for Delta and Iceberg format table engines to help users connect tables managed by different data lakes.

The following example shows how to use the Delta table engine and CONNECTION to create a Delta table stored on S3:

create connection my_s3_conn storage_type = 's3' access_key_id ='minioadmin' secret_access_key ='minioadmin' ENDPOINT_URL='http://127.0.0.1:9900';

create table test_delta engine = delta location = 's3://testbucket/admin/data/delta/delta-table/' connection_name = 'my_s3_conn'

Currently, the Delta table engine only supports basic read operations, and we will continue to iteratively optimize it in the future.

If you'd like to learn more, feel free to contact the Databend team or check out the resources listed below.

Code Corner

Let’s explore code snippets or projects in Databend and the surrounding ecosystem.

Databend disables ClickHouse HTTP Handler by default

Taking into account the continuous improvement of Databend's own driver and surrounding ecology, a new configuration item is introduced this week enable_clickhouse_handlerto control whether to enable ClickHouse-compatible HTTP Handler. The default value is 0.

If you need to use ClickHouse HTTP Handler, you can execute the following SQL statement to enable support:

set global enable_clickhouse_handler=1;

PR #13965 | feat: add setting to disable clickhouse handler

Use Borsh to replace Bincode

Since Bincode may cause binary inconsistency under different versions and different usage methods, after investigation, Databend used Borsh to replace Bincode for serialization of aggregate status.

The format of Borsh is almost the same as the format of Databend aggregate state, and has consistency guarantees, follows safe coding practices, and has good format specifications and performance.

PR #13997 | refactor: using borsh instead of bincode for serde agg function state

Highlights

Here are some noteworthy events, maybe you can find something of interest.

Support overflowing Sort when memory limit is reached.
Support internal stream columns.
Support SHOW LOCKS.
Read Docs | system.streams and Docs | system.stream_status to learn how to use system tables to obtain stream-related information.

What's Up Next

We are always open to cutting-edge technologies and innovative ideas, and welcome you to join the community and inject vitality into Databend.

Introducing PASSWORD POLICY

Databend plans to introduce password policies to meet users' security needs. By setting a password policy, you can deny access after an incorrect password attempt and set password complexity requirements.

CREATE PASSWORD POLICY [ IF NOT EXISTS ] <name>
  [ PASSWORD_MIN_LENGTH = <integer> ]
  [ PASSWORD_MAX_LENGTH = <integer> ]
  [ PASSWORD_MIN_UPPER_CASE_CHARS = <integer> ]
  [ PASSWORD_MIN_LOWER_CASE_CHARS = <integer> ]
  [ PASSWORD_MIN_NUMERIC_CHARS = <integer> ]
  [ PASSWORD_MIN_SPECIAL_CHARS = <integer> ]
  [ PASSWORD_MIN_AGE_DAYS = <integer> ]
  [ PASSWORD_MAX_AGE_DAYS = <integer> ]
  [ PASSWORD_MAX_RETRIES = <integer> ]
  [ PASSWORD_LOCKOUT_TIME_MINS = <integer> ]
  [ PASSWORD_HISTORY = <integer> ]
  [ COMMENT = '<string_literal>' ]

Issue #13994 | feat: PASSWORD POLICY

If you are interested in this topic, you can try to solve some of the problems or participate in discussions and PR reviews. Alternatively, you can click on https://link.databend.rs/im-feeling-lucky to pick a random question, good luck!

Changelog

Check out the changelog for Databend's daily builds to stay up to date on the latest developments.

Address: https://github.com/datafuselabs/databend/releases

Contributors

A big thank you to the contributors for their great work this week.

Connect With Us

Databend is an open source, flexible, low-cost, new data warehouse based on object storage that can also perform real-time analysis. We look forward to your attention and exploring cloud native data warehouse solutions together to create a new generation of open source Data Cloud.