Amoro is a lake warehouse management system built on open data lake tables such as Apache Iceberg. It provides a set of pluggable data self-optimization mechanisms and management services, aiming to bring out-of-the-box lake warehouse usage to users. experience.
On November 7, 2023, Amoro version 0.6.0 was officially updated and released! Based on version 0.5.1, this version adds many features and improves usability and stability. It is recommended that all users and developers upgrade to this version. In this version update, 21 contributors from the community made 118 submissions. Thank you to every community partner for your contribution!
01 Important update
1.Kubernetes integration
Supports deployment of AMS and Optimizer through Kubernetes.
2. More friendly integration with S3
When registering the catalog, you can choose that the Storage is S3 and supports the AK/SK verification system.
3. Paimon format support
Apache Paimon is a real-time data lake platform with high-speed data ingestion, change log tracking and efficient real-time analysis.
- Supports registration of Paimon catalog on the Catalogs page.
- After registering the catalog, you can view the Schema, Properties, Files, Snapshots, Optimizing, Operations and other information of the table on the Tables page.
- Spark sql supported by paimon can be executed in the Terminal interface.
4. Partition and file expiration
Now you only need to perform some simple configurations on the table to enable the function of automatically expiring files or partitions in the table based on time, such as:
CREATE TABLE IF NOT EXISTS user (
id INT,
name string,
ts TIMESTAMP
) USING iceberg
PARTITIONED BY (days(ts));
ALTER TABLE user SET TBLPROPERTIES (
'data-expire.enabled' = 'true',
'data-expire.level' = 'partition',
'data-expire.field' = 'ts',
'data-expire.retention-time' = '30d'
);The above example enables the automatic expiration function of partitions on the user table, and AMS will automatically eliminate partitions that are older than 30 days. For more information about automatic expiration of partitions and files, please refer to the latest user manual : https://amoro.netease.com/docs/latest/using-tables/#configure-data-expiration .
5.Mixed Format supports ORC file format
Mixed Format Users can set the file storage format to ORC format.
6.Mixed Format supports Flink-1.16 and Flink-1.17
Support for Flink-1.12 and Flink-1.14 has been removed, and support for Flink-1.16 and Flink-1.17 has been added.
7. Optimize the memory usage of Position Delete
Reduces the memory consumption caused by indexing Iceberg's position-delete data during the Self-Optimizing process.
02 Release Note
For the complete Release Note of Amoro 0.6.0 version, please refer to:
https://github.com/NetEase/amoro/releases/tag/v0.6.0
03 Acknowledgments
The development of the Amoro community is inseparable from the active trials and feedback of a large number of users, as well as the selfless contributions of community developers. Thank you again for your efforts! More partners are also welcome to participate in the construction of the Amoro community!
Contributors of version 0.6.0 (in no particular order)
End~
If you are interested in data lake, lake-warehouse integration, table format or Amoro community, please contact us for in-depth discussions.
More information about Amoro can be found at:
- Official website: https://amoro.netease.com/
- Source code: https://github.com/NetEase/amor o
- WeChat ID : Search kllnn999, or scan and send the QR code to add the assistant WeChat to join the community
Author: Shi Dayang
Editor: Viridian