Databend Open Source Weekly Issue 110

Databend is a modern cloud data warehouse. Designed for flexibility and efficiency to support your large-scale analysis needs. Free and open source. Experience the cloud service now: https://app.databend.cn .

What's On In Databend

Explore Databend's new developments this week and encounter Databend that is closer to your heart.

Use BendSQL to manage files in Stage

Databend recommends using PRESIGN to upload files to the stage or download files to the local. PRESIGN generates a time-limited pre-signed URL, providing a safe and efficient way to transfer data and reducing file transfer delays.

For BendSQL client users, you can use the PUT command to upload files to Stage and the GET command to download files in Stage.

root@localhost:8000/default> PUT fs:///books.parquet @~

PUT fs:///books.parquet @~

┌───────────────────────────────────────────────┐
│                 file                │  status │
│                String               │  String │
├─────────────────────────────────────┼─────────┤
│ /books.parquet                       │ SUCCESS │
└───────────────────────────────────────────────┘

GET @~/ fs:///fromStage/

┌─────────────────────────────────────────────────────────┐
│                      file                     │  status │
│                     String                    │  String │
├───────────────────────────────────────────────┼─────────┤
│ /fromStage/books.parquet                      │ SUCCESS │
└─────────────────────────────────────────────────────────┘

If you would like to learn more, check out the resources listed below.

Docs | Staging Files

Code Corner

Let’s explore code snippets or projects in Databend and the surrounding ecosystem.

Using Databend Python Binding in Jupyter Notebook

Databend provides Python Binding, which can be used without deploying a Databend instance. DataFrame can also be freely converted to Polars and Pandas formats, making it easy to integrate with data science tools.

Just execute the following command to install:

pip install databend

The following program shows how to use Databend Python Binding in Jupyter Notebook and draw a bar chart using matplotlib.

# Create a table in DataBend
ctx.sql("CREATE TABLE IF NOT EXISTS user (created_at Date, count Int32)")

# Create a table in DataBend
ctx.sql("CREATE TABLE IF NOT EXISTS user (created_at Date, count Int32)")

# Insert multiple rows of data into the table
ctx.sql("INSERT INTO user VALUES ('2022-04-01', 5), ('2022-04-01', 3), ('2022-04-03', 4), ('2022-04-03', 1), ('2022-04-04', 10)")

# Execute a query
result = ctx.sql("SELECT created_at as date, count(*) as count FROM user GROUP BY created_at")

# Display the query result
result.show()

# Import libraries for data visualization
import matplotlib.pyplot as plt

# Convert the query result to a Pandas DataFrame
df = result.to_pandas()

# Create a bar chart to visualize the data
df.plot.bar(x='date', y='count')
plt.show()

If you would like to learn more, check out the resources listed below.

Docs | Tutorial-3: Integrate with Jupyter Notebook with Python Binding Library

Highlights

Here are some noteworthy events, maybe you can find something of interest.

Implement initial support for ownership models
Hash Join supports spill
Columns in Databend now support NULL values by default, read Docs | NULL Values and NOT NULL Constraint for details.
Read Docs | databend-local to learn about Databend Local mode.

What's Up Next

We are always open to cutting-edge technologies and innovative ideas, and welcome you to join the community and inject vitality into Databend.

Supports creating UDFs using JavaScript

PR #12729 | feat: implement udf server in databend is expected to be merged this week. This means that Databend will soon support the creation of user-defined functions using Python.

CREATE FUNCTION [IF NOT EXISTS] <udf_name> (<arg_type>, ...) RETURNS <return_type> LANGUAGE <language> HANDLER=<handler> ADDRESS=<udf_server_address>

We expect to provide support for UDFs in multiple languages based on this PR, and JavaScript may be one of the options worth trying.

Issue #12746 | Feature: support javascript udf

If you are interested in this topic, you can try to solve some of the problems or participate in discussions and PR reviews. Alternatively, you can click on https://link.databend.rs/im-feeling-lucky to pick a random question, good luck!

Changelog

Check out the changelog for Databend's daily builds to stay up to date on the latest developments.

Address: https://github.com/datafuselabs/databend/releases

Contributors

A big thank you to the contributors for their great work this week.

Connect With Us

Databend is an open source, flexible, low-cost, new data warehouse based on object storage that can also perform real-time analysis. We look forward to your attention and exploring cloud native data warehouse solutions together to create a new generation of open source Data Cloud.