Databend Open Source Weekly Issue 108

Databend is a modern cloud data warehouse. Designed for flexibility and efficiency to support your large-scale analysis needs. Free and open source. Experience the cloud service now: https://app.databend.cn .

What's On In Databend

Explore Databend's new developments this week and encounter Databend that is closer to your heart.

Multi-source data directory

In Databend, Catalog is the highest level abstraction of data organization, which can help users manage and access data sources efficiently.

Databend has the capability of multi-source data catalog (Multiple Catalog), and currently supports two types of external catalogs, Apache Iceberg and Apache Hive.

By mounting an external Catalog, efficient querying can be performed without loading data into Databend.

-- Create a Hive catalog
CREATE CATALOG hive_ctl 
TYPE = HIVE 
CONNECTION =(
    METASTORE_ADDRESS = '127.0.0.1:9083' 
    URL = 's3://databend-toronto/' 
    AWS_KEY_ID = '<your_key_id>' 
    AWS_SECRET_KEY = '<your_secret_key>' 
);

If you would like to learn more, check out the resources listed below.

Code Corner

Let’s explore code snippets or projects in Databend and the surrounding ecosystem.

Utilize `cargo-pgo`to implement PGO

Profile-guided optimization (PGO) is a common compilation optimization technology that can use profiling information collected when a program is running to guide the compiler to perform targeted optimizations and generate higher-quality code, thereby improving program performance.

According to tests, using PGO, Databend can achieve approximately 10% performance improvement in certain scenarios. The specific performance of PGO depends on the actual workload. You can try to apply PGO technology to optimize the performance of the Databend cluster.

If you would like to learn more, check out the resources listed below.

Docs | Profile Guided Optimization (PGO)

Highlights

Here are some noteworthy events, maybe you can find something of interest.

Implement json_prettyfunction.
Improve inner join performance.
Add more statistical indicators to the HTTP query interface.
SHOW DATABASESImplemented and supported for Hive Catalog SHOW TABLES.
Read Blog | Revolutionizing Data Archival and Query Performance for Pharmaceutical Group to learn about Databend's customer success stories in the pharmaceutical retail space.

What's Up Next

We are always open to cutting-edge technologies and innovative ideas, and welcome you to join the community and inject vitality into Databend.

Implement fuzz testing using SQLsmith

SQLsmith focuses on generating random, type-aware and column-aware SQL queries that can usually pass semantic checks to further test the execution logic of the database.

In the past, Databend has supported SQLancer as well as traditional fuzz testing with randomly generated inputs. The Databend team plans to introduce SQLsmith to implement domain-aware fuzz testing, provide more comprehensive and targeted test results, and increase the possibility of discovering vulnerabilities.

Issue #12576 | Feature: Using sqlsmith to support sql fuzzy testing

If you are interested in this topic, you can try to solve some of the problems or participate in discussions and PR reviews. Alternatively, you can click on https://link.databend.rs/im-feeling-lucky to pick a random question, good luck!

Changelog

Check out the changelog for Databend's daily builds to stay up to date on the latest developments.

Address: https://github.com/datafuselabs/databend/releases

Contributors

A big thank you to the contributors for their great work this week.

Connect With Us

Databend is an open source, flexible, low-cost, new data warehouse based on object storage that can also perform real-time analysis. We look forward to your attention and exploring cloud native data warehouse solutions together to create a new generation of open source Data Cloud.