SeaTunnel releases the first version 2.3.2 after becoming a top-level project of Apache, further improving the stability and usability of the Zeta engine

Recently, Apache SeaTunnel officially released version 2.3.2. At this time, it has been more than two months since the previous version 2.3.1 was released. During this period, we collected and based on the feedback from users and developers, and performed bug fixes on SeaTunnel Zeta Engine in version 2.3.2, improving the stability of the engine and usage efficiency.

In addition, the new version optimizes the function and performance of the connectors in Connector-V2. At the same time, SQL Transfrom supports custom UDF functions, and Zeta Engine also provides cluster monitoring and query operations through the Rest API.

This article will introduce the details of the update content of Apache SeaTunnel 2.3.2 version.

• Release Note: https://github.com/apache/seatunnel/blob/2.3.2-release/release-note.md
• Download address: https://seatunnel.apache.org/download

new function

In this update, Zeta Engine supports obtaining task and system monitoring information in the form of RestAPI. Users can complete data information requests by calling HTTP requests to any node, which enhances task monitoring capabilities.

For details, please refer to: https://seatunnel.apache.org/docs/2.3.2/seatunnel-engine/rest-api

In addition, SQL Transform supports custom UDF functions in version 2.3.2. For detailed usage, please refer to: https://seatunnel.apache.org/docs/2.3.2/transform-v2/sql-udf

Core

[Core] [API] Support convert String to List option (#4362) [Core] [API] Add option check to FactoryUtil before creating source, sink and transform (#4424) [Core] [Shade] Added guava shade module (#4358)

Connector-V2

• [Connector-V2] [CDC] [SQLServer] Support multi-table reading (#4377)
• [Connector-V2] [Kafka] Kafka source supports skipping data deserialization failure (#4364)
• [Connector-V2] [Jdbc] [TiDB] Added TiDB directory (#4438)
• [Connector-V2][File] Added source and sink for file Excel (#4164)
• [Transform-V2] Added UDF SPI and a sample implementation for the SQL transform plugin (#4392)
• [Transform-V2] Support copy field list (#4404)
• [Transform-V2] Added CatalogTable support for FieldMapperTransform (#4423)
• [Transform-V2] Added CatalogTable support for ReplaceTransform (#4411)
• [Transform-V2] Added Catalog support for FilterRowKindTransform (#4420)
• [Transform-V2] Added CatalogTable support for FilterFieldTransform (#4422)

Zeta Engine

• [Zeta] Support the mixed use of Factory and Plugin SPI (#4359)
• [Zeta] Added REST API to get running job information by job ID (#4140)
• [Zeta] Added REST API to get system monitoring information (#4315)
• [Transform V2 & Zeta] Enable SplitTransform to support CatalogTable and CatalogTable evolution (#4396)

Improve

The Apache SeaTunnel 2.3.2 version fixes the stability of the Zeta engine, solves the problem of excessive memory usage and does not release it, solves the problem of a large amount of resource usage, and optimizes performance accordingly.

Core

• [Core][Spark] Push transformation operations from Spark Driver to Executors (#4503)
• [Core] [Starter] Optimize code structure, delete redundant code (#4525)
• [Core] [Translation] [Flink] Optimize code structure and delete redundant code (#4527)

Connector-V2

• [Connector-V2] [CDC] Improve startup.mode/stop.mode options (#4360)
• [Connector-V2] [CDC] Optimize jdbc fetch-size option (#4352)
• [Connector-V2][SQLServer] Fixed SQL Server directory issue (#4441)
• [Connector-V2] [StarRocks] Improve StarRocks serialization error message (#4458)
• [Connector-V2] [Jdbc] Added SQL log and updated some styles (#4475)
• [Connector-V2] [Jdbc] Fixed the problem that the table name cannot be automatically obtained when there are multiple tables (#4514)
• [Connector-V2] [S3 & Kafka] Remove unavailable S3 and Kafka directories (#4477)
• [Connector-V2] [Pulsar] support Canal format

Zeta Engine

• [Zeta] Support running server in daemon mode (#4161)
• [Zeta] Improve client SDK compatibility with ClassLoader (#4447)
• [Zeta] Client supports submitting jobs asynchronously (#4456)
• [Zeta] Add more verbose log output (#4446)
• [Zeta] Improve seatunnel-cluster.sh script (#4435)
• [Zeta] Reduce CPU consumption when tasks are not ready (#4479)
• [Zeta] Add parser log (#4485)
• [Zeta] Remove redundant code (#4489)
• [Zeta] Remove redundant code in validateSQL (#4506)
• [Zeta] Improve JobMetrics performance (#4467)

Bug fixes

Core

*[Core][API] Fixed the problem that the common class of the list was missing (#4421) *[Core][API] Fixed the problem that the parse nested row data type key becomes uppercase (#4459)

Connector-V2

• [Json-format] [Canal-Json] Fixed a null pointer exception when deserializing JSON (#4195)
• [Connector-V2] [Jdbc] Fixed field aliases not being supported in JDBC source queries (#4210)
• [Connector-V2] [Jdbc] Fixed the connection failure caused by connection timeout (#4322)
• [Connector-V2] [Jdbc] Set the default value of JdbcOption: generate_sink_sql to false (#4471)
• [Connector-V2][Pulsar] Fixed an issue where messages could not be continuously consumed (#4125)
• [Connector-V2] [Eleasticsearch] Fixed the wrong document description (#4390)
• [Connector-V2][Eleasticsearch] Fixed source deserialization errors and inappropriate issues (#4233)
• [Connector-V2] [Kafka] Fixed KafkaProducer resource never being released (#4302)
• [Connector-V2] [Kafka] Fixed client.id causing permission issues (#4246)
• [Connector-V2] [Kafka] Fixed the problem that KafkaConsumerThread exited due to wrong commit offset (#4379)

Zeta Engine

• [Zeta] Fixed LogicalDagGeneratorTest test case issue (#4401)
• [Zeta] Fixed MultipleTableJobConfigParser parsing only one transform (#4412)
• [Zeta] Fixed missing common plugin jar package (#4448)
• [Zeta] Fixed handleCheckpointError still being called when checkpoint has completed (#4442)
• [Zeta] Fixed a bug where job error messages were incorrect (#4463)
• [Zeta] Fixed the bug of finding TaskGroup deployment nodes (#4449)
• [Zeta] Fixed configuration file bug (#4488)
• [Zeta] Fixed logic issue with loading connectors from zeta (#4510)
• [Zeta] Fixed hadoop-hdfs conflict dependency issue (#4509)

E2E

*[E2E] [Kafka] Fixed the problem of Kafka end-to-end test cases (#4520) *[Container Version] fixed the risk problem of test cases that cannot be reproduced #4591

document optimization

• [Docs] Optimized part of Doris and SelectDB connector documentation (#4365)
• [Docs] Fixed code style issues in docs (#4368)
• [Docs] Updated jdbc documentation and kafka documentation (#4380)
• [Docs] Fixed max_retries defaulting to 0 (#4383)
• [Docs] Fixed Markdown syntax issue (#4426)
• [Docs] Fixed Kafka docs wrong config key "kafka." (#4427)
• [Docs] Added Transform section in quickstart v2 (#4436)

list of contributors

Thanks to Fan Jia, an open source engineer of Beluga, for his support for this version release, and the contributions of the following community partners!

Andrew Wetmore,

the pain,

Carl-Zhou-CN,

Cason-ACE,

Chengyu Yan,

CodingGPT,

dalong,

Eric,

FlechazoW,

Guangdong Liu,

Hao Xu,

J.A.R.V.I.S,

Kim,

Laglangyue,

Marvin,

TaoZex,

Tyrantlucifer,

Xiaojian Sun,

Zhilin Li,

Zongwen Li,

should,

gnehil,

hailin0,

ic4y,

kezhenxu94,

lightzhao,

lucklilili,

lvshaokang,

mengxiaoopeng,

monster,

songjianet,

stdnt-xiao,

thomasc,

will27,

wow,

xiaofan2012,

zhilinli,

foam

This article is supported by Beluga Open Source Technology !