Official Announcement | Apache Flink 1.18 Release Announcement

From data to large model application, on November 25th, Hangzhou Yuanchuang Conference was held to share development tips.

Apache Flink PMC has officially released version 1.18.0 of Apache Flink. As always, this is a fleshed-out release with a wide range of improvements and new features. A total of 174 people contributed to this version, completing 18 FLIPs and over 700 questions. Thank you to all contributors for your support!

Towards Streaming Lakehouse

Flink SQL improvement

Flink SQL Gateway 的 JDBC Driver

Flink version 1.18 provides the JDBC Driver of Flink SQL Gateway. Therefore, you can now interact with your tables through Flink SQL using any SQL client that supports JDBC. The following is an example using SQLLine .

sqlline version 1.12.0
sqlline> !connect jdbc:flink://localhost:8083
Enter username for jdbc:flink://localhost:8083:
Enter password for jdbc:flink://localhost:8083:
0: jdbc:flink://localhost:8083> CREATE TABLE T(
. .)>      a INT,
. .)>      b VARCHAR(10)
. .)>  ) WITH (
. .)>      'connector' = 'filesystem',
. .)>      'path' = 'file:///tmp/T.csv',
. .)>      'format' = 'csv'
. .)>  );
No rows affected (0.122 seconds)
0: jdbc:flink://localhost:8083> INSERT INTO T VALUES (1, 'Hi'), (2, 'Hello');
+----------------------------------+
|              job id              |
+----------------------------------+
| fbade1ab4450fc57ebd5269fdf60dcfd |
+----------------------------------+
1 row selected (1.282 seconds)
0: jdbc:flink://localhost:8083> SELECT * FROM T;
+---+-------+
| a |   b   |
+---+-------+
| 1 | Hi    |
| 2 | Hello |
+---+-------+
2 rows selected (1.955 seconds)
0: jdbc:flink://localhost:8083>

More information

Flink connector’s stored procedure support

Stored Procedure has always been an indispensable tool in traditional databases. It provides a convenient way to encapsulate complex logic for data manipulation and task management. Stored procedures also offer the potential to enhance performance because they can trigger the processing of data operations directly in the external database. Other popular data systems such as Trino and Iceberg automate and simplify common maintenance tasks into a small set of stored procedures, thereby greatly reducing the user's administrative burden.

This update is mainly aimed at developers of Flink connectors. They can now predefine custom stored procedures into the connector through the Catalog interface. The main benefit to users is that connector-specific tasks that previously required writing custom Flink code can now be replaced with simple calls that encapsulate, standardize, and potentially optimize the underlying operations. Users can use the familiar CALL syntax to execute stored procedures and use SHOW PROCEDURES to view the connector's available stored procedures. The stored procedures in the connector improve the scalability of Flink's SQL and Table API, providing users with smoother data access and management capabilities.

Users can use the Call statement to directly call the catalog's built-in stored procedures (Note: For the catalog's built-in stored procedures, please refer to the corresponding catalog documentation). For example, the compact operation can be performed on the Paimon table through the following Call statement:

CREATE TABLE `paimon`.`default`.`T` ( 
 id BIGINT PRIMARY KEY NOT ENFORCED,  
 dt STRING, -- format 'yyyy-MM-dd'  
 v STRING
);

-- use catalog before call 
proceduresUSE CATALOG `paimon`;

-- compact the whole table using call statement
CALL sys.compact('default.T');

More information

DDL support extension

Starting from version 1.18, Flink supports the following functions:

REPLACE TABLE AS SELECT
CREATE OR REPLACE TABLE AS SELECT

Both commands, as well as the previously supported CREATE TABLE AS, now support atomicity, provided the underlying connector also supports it.

Additionally, Apache Flink now supports executing TRUNCATE TABLE in batch mode. As before, the underlying connector needs to implement and provide this functionality.

Finally, we've also implemented support for adding, removing, and listing partitions via:

ALTER TABLE ADD PARTITION
ALTER TABLE DROP PARTITION
SHOW PARTITIONS

More information

Time Traveling

Flink supports time travel SQL syntax for querying historical versions of data. Users can specify a point in time to retrieve the data and schema of the table at that point in time. With time travel functionality, users can easily analyze and compare historical versions of data.

For example, users can query data at a specified time point in a table through the following statement;

-- 查询表 `paimon_tb` 在 2022年11月11日的数据

SELECT * FROM paimon_tb FOR SYSTEM_TIME AS OF TIMESTAMP '2022-11-11 00:00:00'；

More information

Stream processing improvements

Table API & SQL supports operator level state retention time (TTL)

Starting from Flink version 1.18, Table API and SQL users can set the state retention time (TTL) individually for stateful operators. In scenarios like stream regular joins, users can now set different TTLs for left and right streams. In previous versions, state retention time could only be controlled at the pipeline level using the configuration item table.exec.state.ttl. With the introduction of operator-level state retention, users can now optimize resource usage based on their specific needs.

More information

SQL watermark alignment (Watermark Alignment) and idleness detection (Idleness Detection)

You can now configure watermark alignment and data source idle timeout using SQL Hint . Previously these features were only available in the DataStream API.

More information

Batch processing improvements

Hybrid Shuffle supports remote storage

Hybrid Shuffle supports storing Shuffle data in remote storage. You can use the configuration item taskmanager.network.hybrid-shuffle.remote.path to configure the remote storage path. Hybrid Shuffle reduces network memory usage and improves stability and ease of use by decoupling memory usage from parallelism.

More information

Performance improvements and TPC-DS benchmarks

In previous versions, the community put a lot of effort into improving Flink's batch processing performance, resulting in significant improvements. During this release cycle, community contributors have continued to make significant efforts to further improve Flink’s batch processing performance.

Flink SQL runtime filter (Runtime Filter)

Runtime Filter is a common method used to optimize join performance. It is designed to dynamically generate runtime filter conditions for certain join queries to reduce the amount of data scanned or shuffled, avoid unnecessary I/O and network transmission, and thereby speed up queries. We introduced runtime filtering in Flink version 1.18 and verified its effectiveness through the TPC-DS benchmark, observing that some queries were 3x faster when this feature was enabled.

Flink SQL operator fusion code generation (Operator Fusion Codegen)

Operator Fusion Codegen improves performance by fusing operator DAGs into an optimized single operator, eliminating virtual function calls, using CPU registers for intermediate data operations, and reducing instruction cache misses. improve query execution performance. As a technical optimization, we verified its effectiveness through TPC-DS. Some batch processing operators (Calc, HashAgg and HashJoin) have completed integrated code generation support in version 1.18, and the performance of many queries has been significantly improved.

Please note that the above two features are turned off by default. They can be enabled by using the two configuration items table.optimizer.runtime-filter.enabled and table.exec.operator-fusion-codegen.enabled.

Since Flink 1.16, the Apache Flink community has been continuously tracking the performance of its batch processing engine through the TPC-DS benchmark framework. After major improvements in Flink 1.17 (dynamic join reordering, dynamic local aggregation), the two improvements (operator fusion, runtime filtering) described earlier have improved performance on the partitioned table 10T data set compared with Flink 1.17 Compared with Flink 1.16, the performance has improved by 54%.

More information

Towards cloud-native elasticity

Elasticity describes the ability of a system to adapt to changes in workload without interruption, ideally in an automated manner. This is a defining characteristic of cloud-native systems and is especially important for long-running stream processing workloads. Therefore, the improvement of elastic performance is an area in which the Apache Flink community continues to invest. Recent proposals include the Kubernetes Autoscaler , numerous improvements to performance tuning, and the Adaptive Scheduler .

The adaptive scheduler has formed the core of a fully resilient Apache Flink deployment since it was first introduced in Flink version 1.15. Among its core capabilities, it allows jobs to change their resource requirements and parallelism while running. Additionally, it adapts adaptively based on the resources available in the cluster, rescaling only when the cluster is able to meet the job's minimum required resources.

Before Flink version 1.18, the adaptive scheduler was mainly used in reactive mode . According to the design, a single job will always use all available resources in the cluster. See this blog post to learn how to automatically scale Flink jobs using the Horizontal Pod Autoscaler on Kubernetes.

In Flink 1.18, the adaptive scheduler has become more powerful and widely applicable, and is becoming the default scheduler for Apache Flink stream processing tasks.

Control dynamic fine-grained expansion and contraction through REST API

Although the adaptive scheduler has the ability to change the resource requirements of a job at runtime, this feature has not been open to users before. Since Flink 1.18, you can change any task of the job through the Flink Web UI and REST API while the job is running. degree of parallelism.

In terms of implementation details, Apache Flink will perform expansion and contraction operations immediately after obtaining the resources required for the new degree of parallelism. The expansion and contraction operations are not based on savepoints, but on ordinary periodic checkpoints, which means that it does not introduce additional snapshots. For jobs with smaller state sizes, the rescale operation occurs almost immediately with minimal disruption.

Combined with the Apache Flink Web UI's backpressure monitoring , it's now easier to find and maintain the degree of parallelism that allows each task to run efficiently and without backpressure.

If a task is very busy (red), you can increase the parallelism.
If a task is idle most of the time (blue), you can reduce the parallelism.

More information

FLIP-291: Externalized Declarative Resource Management

Scale up and down RocksDB faster

The scaling time using the RocksDB state backend and incremental checkpoints improves by about 30% at the 99th percentile. We have improved the capability of parallel downloading, from only parallel downloading state handles to extending to parallel downloading of files. In addition, we turned off write-ahead-logging for batch inserts on temporary RocksDB instances used for scaling.

More information

Java 17 support

Released in 2021, Java 17 is the latest long-term support (LTS) version of Java and will end support in 2029. Starting with Flink version 1.18, you can now run Apache Flink on Java 17. The official Docker repository currently contains images based on Java 17:

docker pull flink:1.18.0-java17

If your cluster is running on Java 17, Java 17 features can also be used in your user programs and compiled to the Java 17 version.

More information

Other improvements

Production-available Watermark Alignment function

Watermark alignment, supported in Beta form since Flink 1.16 and Flink 1.17 versions, has been fully tested in real-world environments at scale. During this time, the community has collected and resolved discovered bugs and performance issues. With these issues resolved, we recommend the watermark alignment feature for general use.

More information

FLINK-32548

Pluggable troubleshooting

Apache Flink serves as the foundation for many companies' stream processing platforms and is also the foundation for many commercial stream processing services. Therefore, the ability to easily integrate into the broader ecosystem of internal and supplier platforms is becoming increasingly important. Catalog modification listeners and pluggable fault handlers fall into this category of improvements.

More information

SQL client improvements

In version 1.18, the SQL client has made a series of usability improvements:

The client is more colorful, enabling SQL syntax highlighting and switching between 7 different color schemes.
Easier to edit and preview large queries.
Line numbers can be turned off and on at any time.

More information

FLIP-189: SQL Client Usability Improvements

Apache Pekko Alternative Akka

A year ago, Lightbend announced that it would change the license of future versions of Akka (2.7+) from Apache 2.0 to BSL. The Akka 2.6 version used by Apache Flink will receive security updates and critical bug fixes until September 2023. Therefore, in Flink 1.18 version, we decided to switch from Akka to Apache Pekko (Incubating). Apache Pekko (Incubating) is based on Akka 2.6.x before the Akka project adopted a commercial source code license. Pekko recently released Apache Pekko 1.0.1-incubating, which we will use in Flink 1.18 immediately. While our mid-term plan is to completely drop dependency on Akka or Pekko (see FLINK-29281 ), switching to Pekko provides a good short-term solution and ensures that the Apache Pekko and Apache Flink communities can handle the entire software supply chain. of critical bug fixes and security vulnerabilities.

More information

FLINK-32468

Calcite upgrade

In Apache Flink 1.18, Apache Calcite is gradually upgraded from version 1.29 to version 1.32. Benefits of version upgrades include bug fixes, smarter optimizers, and performance improvements. At the parser level, it now allows parentheses to be used to group join operations into a tree structure (SQL-92). For example SELECT * FROM a JOIN (b JOIN c ON bx = cx) ON ay = cy, see CALCITE-35 . Additionally, upgrading to Calcite 1.31+ unlocks support for Session Windows via Table-Valued Functions (see CALCITE-4865 , FLINK-24024 ) and deprecates the old Group Window Aggregations. Flink's casting behavior has changed slightly due to CALCITE-4861 . Some edge cases may now behave differently: for example, a cast from FLOAT/DOUBLE 9234567891.12 to INT/BIGINT produces Java's overflow behavior.

More information

Important API deprecation

In preparation for Flink 2.0, which will be released next year, the community has decided to officially deprecate several APIs that are nearing the end of their life cycle.

SourceFunction is now deprecated. If you are still using a connector built on SourceFunction, migrate it to Source . SinkFunction has not been officially deprecated yet, but it is approaching the end of its life cycle and will be replaced by SinkV2.
Queryable State is now deprecated and will be removed in Flink 2.0.
The DataSet API is now deprecated. It is recommended that users migrate to the DataStream API with execution mode set to BATCH.

Upgrade Notes

The Flink community works hard to ensure seamless upgrades. However, some changes may require users to adjust parts of the program when upgrading to version 1.18. Please refer to the release notes for a comprehensive list of adjustments to make and issues to check for during the upgrade.

Contributor list

The Apache Flink community would like to thank all contributors who contributed to the implementation of this release:

Aitozi, Akinfolami Akin-Alamu, Alain Brown, Alexander Pilipenko, Alexander Fedulov, Anton Kalashnikov, Archit Goyal, Bangui Dunn, Benchao Li, BoYiZhang, Chesnay Schepler, Chris Nauroth, Colten Pilgreen, Danny Cranmer, David Christle, David Moravek, David Wysakowicz , Deepyaman Datta, Dian Fu, Dian Qi, Dong Lin, Eric Xiao, Stephen Chauchot, Feng Jin, Ferenc Csaky, Fruzsina Nagy, Gabor Somogyi, Gunnar Morling, Gyula Fora, HaiYang Chen, Hang Ruan, Hangxiang Yu, Hanyu Zheng, Hong Liang Teoh, Hongshun Wang, Huston, Jacky Lau, James Hughes, Jane Chan, Jark Wu, Jayadeep Jayaraman, Jia Liu, JiangXin, Joao Boto, Junrui Lee, Juntao Hu, KI (Dennis) Jung, Kaiqi Dong, L, Leomax_Sun, Leonard Xu, Licho, Lijie Wang, Liu Jiangang, Lyn Zhang, Maomao Min, Martin Visser, Marton Balassi, Mason Chen, Matthew de Detrich, Matthias Pohl, Min, Mingliang Liu, Mohsen Rezaei, Mrart, Mulavar, Nicholas Jiang, Nicolas Fraison , Noah, Panagiotis Garefalakis, Patrick Lucas, Paul Lin, Peter Vary, Piotr Nowojski, Qingsheng Ren, Ran Man, Rich Bowen, Robert Metzger, Roc Marshal, Roman Khachatryan, Ron, Rui Fan, Ryan Skraba, Samrat002, Sergey Nuyanzin, Sergio Morales, Shammon FY, Shammon FY, Shengkai, Shuiqiang Chen, Stefan Richter, Tartarus0zm, Timo Walther, Tzu-Li (Gordon) Tai, Venkata Krishnan Sowrirajan, Wang FeiFan, Weihua Hu, Weijie Guo, Wencong Liu, Xiaogang Zhou, Xiaogang Zhou, XuShuai, Yanfei Lei, Yu Chen, Yubin Li, Yun Gao, Yun Tang, Yuxin Tan, Zakelly, Zhanghao Chen, ZhengYiWeng, Zhu Zhu, Archzi, Baiwuchang, Cailiuyang, Chenyuzhi, Darenwkt, Dongwoo Kim, Eason.qin, Felixzh, Fengli , frankeshi, fredia, godfrey he, haishui, hehuiyuan, huangxingbo, jiangxin, jiaoqingbo, jinfeng, jingge, kevin.cyj, kristoffSC, leixin, leiyanfei, liming.1018, lincoln lee, lincoln.lil, liujiangang, liuyongvs, luoyuxia, maigeiye , mas-chen, novakov-alexey, oleksandr.nitavskyi, pegasas, sammieliu, shammon, shammon FY, shuiqiangchen, slfan1989, sunxia, tison, tsreaper, wangfeifan, wangkang, whjshj, wuqqq, xiangyu0xf, xincheng.ljr, xmzhou, xu you, xzw, yuanweining, yuchengxin, yunfengzhou-hub, yunhong, yuxia luo, yuxiqian, zekai-li, zhangmang, zhengyunhong.zyh, zzzzzzzs, 沈嘉琦