Databend Open Source Weekly Issue 117

From data to large model application, on November 25th, Hangzhou Yuanchuang Conference was held to share development tips.

Databend is a modern cloud data warehouse. Designed for flexibility and efficiency to support your large-scale analysis needs. Free and open source. Experience the cloud service now: https://app.databend.cn .

What's On In Databend

Explore Databend's new developments this week and encounter Databend that is closer to your heart.

Feature Preview: Read-Only ATTACH TABLE

For a few large-scale queries, a large number of resident resources have to be maintained, which brings high maintenance costs to the enterprise. How to mix local and cloud workloads and use the instant scalability of Databend Cloud to further reduce the number of resident private instances and save maintenance costs while ensuring query reliability?

Databend recently provided read-only ATTACH TABLE, allowing users who deploy Databend locally to share table data with Databend Cloud in a read-only manner:

All table data is shared read-only.
Changes to tables, including data and schema changes, are performed and tracked on the user's local platform.

ATTACH TABLE <target_table_name> '<source-table-data-URI>' CONNECTION=(<connection_parameters>) READ_ONLY

By ATTACH TABLEappending the parameter to the statement , the target table will use the latest snapshot hint file READ_ONLYstored at the specified source table data to map the table data and structure, and the target table can only be accessed in read-only mode.URI

If you'd like to learn more, feel free to contact the Databend team or check out the resources listed below.

Code Corner

Let’s explore code snippets or projects in Databend and the surrounding ecosystem.

Use OpenTelemetry Collector to replace Jaeger Agent

Jaeger is an open source, end-to-end distributed tracing tool from Uber that can help monitor and troubleshoot microservices-based applications.

Due to port changes in Jaeger's latest All-in-One Docker image, the original distributed tracing capability is invalid. Upstream is also considering deprecating opentelemetry-jaeger and encouraging users to migrate to opentelemetry-otlp , see Consider deprecating opentelemetry-jaeger #995 .

Databend actively responded to the upstream call. While repairing the availability of distributed tracing capabilities, it opentelemetry-otlpreconstructed the original tracing export mechanism using and the corresponding environment variables were also changed to DATABEND_OTEL_EXPORTER_OTLP_ENDPOINT.

If you would like to learn more, check out the resources listed below.

Highlights

Here are some noteworthy events, maybe you can find something of interest.

Added table function quantile_tdigest_weighted.
Recluster supports distributed execution.
DELETEStatements support table aliases.
SQLsmith supports REPLACE INTOand HINT. Read the latest blog Leveraging Randomized Testing with SQLsmith to Uncover Bugs to learn about Databend's SQLsmith design.
New JSON operators ->, ->>, #>and are added #>>. Read the Docs | JSON Operators to learn more about this.

What's Up Next

We are always open to cutting-edge technologies and innovative ideas, and welcome you to join the community and inject vitality into Databend.

Support definition`CONNECTION`

In order to more elegantly handle the mapping of user roles between the access control models of computing cluster instances and storage services, Databend plans to provide CONNECTIONthe ability to define and manage.

CONNECTION can save identity credentials and access management (IAM) entities generated by external storage services, providing a convenient and consistent access control experience. In addition, the introduction CONNECTIONcan also simplify complex configurations in processes such as creating STAGE.

The reference syntax specifications are as follows:

CREATE CONNECTION [IF NOT EXISTS]
    ENDPOINT_URL = 'http://localhost:9000',
    ACCESS_KEY_ID = 'ROOTUSER',
    SECRET_ACCESS_KEY = 'CHANGEME123'
...

Issue #13489 | feat: CREATE CONNECTION

If you are interested in this topic, you can try to solve some of the problems or participate in discussions and PR reviews. Alternatively, you can click on https://link.databend.rs/im-feeling-lucky to pick a random question, good luck!

Changelog

Check out the changelog for Databend's daily builds to stay up to date on the latest developments.

Address: https://github.com/datafuselabs/databend/releases

Contributors

A big thank you to the contributors for their great work this week.

Connect With Us

Databend is an open source, flexible, low-cost, new data warehouse based on object storage that can also perform real-time analysis. We look forward to your attention and exploring cloud native data warehouse solutions together to create a new generation of open source Data Cloud.