Data processing technology solves the needs of massive storage and analysis for businesses in all walks of life, but the explosive growth of data volume and the continuous enrichment of data types have put forward higher requirements for data processing technology and timeliness, which makes general Technologies such as computing engines (such as Spark and Flink), interactive analysis systems (such as ClickHouse), and data lake frameworks (such as Iceberg) are developing rapidly.

As a professional developer community, DEEPNOVA is committed to promoting technical exchanges, expanding technical horizons, establishing a technical ecology, and actively embracing open source communities, conducting in-depth research on open source technologies such as new generation data lakes and real-time data warehouses, and optimizing some functions. .

In order to better conduct technical discussions and exchanges with developers, from 14:00 to 17:30 on April 16 , DEEPNOVA and the Iceberg community jointly launched "DEEPNOVA MEETUP Online". The theme of this event is "Building a Real-time Data Lake Based on Iceberg", which brings together the strength of the DEEPNOVA community expert group, and will lead the audience to understand the complete history of Iceberg technology development and its application and practice in localized data, so as to truly integrate high-quality technical content. Give back to the community.

core content

1. Technical Interpretation: "Apache Iceberg Past, Present and Future"

Sharing guest : Apache Iceberg and HBase PMC member Hu Zheng

Highlights : Apache Iceberg, as an open and standardized data lake table format, has been selected and applied by many domestic and foreign manufacturers. Recently Apache Iceberg plans to launch a commercial version of the data lake storage service on AWS. At the same time, Snowflake, AWS, Cloudera and other companies have released Iceberg data lakes. As it turns out, after several years of development, Apache Iceberg has achieved rapid growth and great success. The content of this sharing includes Iceberg's open source process, as well as the current and future technical directions of key efforts.

2. Technical practice: "NetEase Lake Warehouse Management System Arctic"

Guest speaker : Ma Jin, head of NetEase's data lake and real-time computing team

Highlights : Arctic is an Iceberg-based lake warehouse management system developed by NetEase. At the same time, NetEase has built a stream-batch integrated data production link through Flink and Arctic, a real-time offline unified data warehouse. Based on Iceberg, Arctic has features such as support for primary keys, structure self-optimization, data consistency, real-time subscription and real-time join. This sharing will mainly introduce the core design ideas of Arctic.

3. Technical practice: "Optimization and Practice of Iceberg Index by FastData DLink"

Guest speaker : Zhang Gan, Director of Storage Engine Department of Dipu Technology

Content highlights : Z-Order is a technology that can compress multi-dimensional data into one dimension. It is widely used in spatiotemporal indexes and images, sorts multiple fields, rearranges original data, and reduces unnecessary I/O. This improves the query speed. Based on the primary key deduplication scheme proposed by the Iceberg community, the DEEPNOVA community optimized it with BloomFilter to filter eq-delete files, reduce memory usage, and improve the efficiency of merging small files. This sharing will mainly interpret the optimization capabilities of FastData in indexing technology.

4. Technical practice: "Optimization and Practice of FastData DLink Building Real-time Data Lake Based on Iceberg"

Guest speaker : Jian Yonghua, database kernel development engineer of Dipu Technology

Highlights : The Iceberg CDC capability is the core capability to support the construction of real-time data warehouses. The DEEPNOVA community has fully implemented the Iceberg CDC function, and has realized the rapid migration of Hive historical data into the lake. The method of generating metadata for community PR has been optimized in parallel, and the migration performance has been improved several times. This sharing will focus on how to build a real-time data warehouse and demonstrate the technical advantages of FastData.

Listener benefits:

1. Understand the architectural principles, features and application scenarios of Apache Iceberg

2. The technical optimization capability and business value of implementing different functions based on Iceberg

3. An open technical exchange community to discuss Hucang technology with senior technical experts

We firmly believe that the advancement of technology must be the result of the joint efforts of countless technology practitioners, and sincerely hope that the DEEPNOVA community can become a learning and exchange platform for technology enthusiasts, and use the power of more people to build a more complete community. We will also uphold the community spirit of openness and sharing, and give back to the community through more technology sharing, live broadcast events, etc., so that digital technology can bring infinite possibilities to the world.

Welcome to scan the code to watch the live broadcast, forward and share the top three in the live broadcast room invitation list, and will also have DEEPNOVA's exclusive gift box.