Apache Linkis version 1.5.0 has been released. We are very grateful to all community members for their contributions and efforts to version 1.5.0.
This version is a major version, adding many features that the community has been waiting for for a long time, such as: the registration center supports switching Nacos, new engines such as Hbase, Nabula, repl, etc., and the new SPARK engine plug-in supports on Yarn Cluster mode. , SPARK and Flink engine plug-ins support on k8s submission of Jar tasks and other functional features, and have fixed multiple known issues.
For details, please check the official version information of Apache Linkis 1.5.0:
https://linkis.apache.org/zh-CN/download/release-notes-1.5.0
Apache Linkis 1.5.0 version introduction
● The registration center supports switching Eureka to Nacos
● Added Hbase engine plug-in, supports all Hbase-shell syntax and is compatible with multiple versions
● Added Nebula engine plug-in
● Added Repl interpreter engine plug-in to support running Java and Scala code snippets
● Support Spark EC on k8s to submit jar and Pyspark tasks
● Support Flink EC on k8s to submit jar tasks
● Spark engine plug-in supports on yarn cluster running mode
● Linkis JDBC driver supports the selection of multiple engines and versions
● Basic data management adds configuration item management, allowing administrators to manage the configuration items of each engine plug-in.
● The management console has added operation and maintenance tools, and added user configuration management to support administrators in adjusting user configuration and template management.
● [Experimental] Entrance supports task takeover, making Entrance stateless.
abbreviation:
● COMMON: Link Common
● ENTRANCE: Linkis Entrance
● EC: EngineConn
● ECM: EngineConnManager
● ECP: EngineConnPlugin
● DMS: Data Source Manager Service
● MDS: MetaData Manager Service
● LM: Link Manager
● PS: Linkis Public Service
● PE: Link Public Enhancement
● RPC: Linkis Common RPC
● CG: Linkis Computation Governance
● DEPLOY: Linkis Deployment
● WEB: Linked Web
● GATEWAY: Linkis Gateway
● EP: Engine Plugin
new features
● [EC] LINKIS-5008 registration center supports Nacos
● [GATEWAY] LINKIS-4992 Gateway supports access control configuration
● [EC-REPL] LINKIS-4937 Added Repl interpreter engine plug-in to support the execution of scala and Java code
● [EC-NEBULA] LINKIS-4903 adds nebula engine plug-in
● [EC-HBASE] LINKIS-48 91 adds Hbase engine plug-in
● [EC-SPARK] LINKIS-4850 SPARK engine plug-in supports On Yarn Cluster operation
● [EC-SPARK] LINKIS-4867 SPARK engine plug-in supports submitting Jar tasks on k8s
● [EC-SPARK] LINKIS-4906 SPARK engine plug-in supports submitting pyspark job on k8s
● [JDBC-DRIVER] LINKIS-4930 JDBCDriver supports multiple engine versions
● [EC-FLINK] LINKIS-4753 Flink engine plug-in upgraded to 1.16.2 and compatible with multiple versions
● [ENTRANCE] LINKIS-4282 Experimental: Entrance service supports HA
● [MONITOR] LINKIS-4905 Experimental: Add Linkis Monitor service
● [WEB] LINKIS-4940 Experimental: New architecture upgrade of the management console front-end
Function enhancement
● [ECM] LINKIS-4990 supports the management station to download EC log files
● [EC] LINKIS-4982 Enrich EC indicators and add lock idle and other time indicators
● [WEB] LINKIS-4954 management console adds user configuration management page
● [EC-SPARK] LINKIS-4961 Pyspark adds more default class imports
● [EC-PYTHON] LINKIS-4835 Python engine plug-in optimizes printing and error message printing
● [EC-SPARK] LINKIS-4896 SPARK Once task supports EngineConnRuntimeMode tag configuration
● [LM] LINKIS-4914 The LM resource sorting and selection rules are optimized from large to small
● [WEB] LINKIS-4935 management console supports configuring spark.conf parameters and multiple SPARK native parameters.
● [EC] LINKIS-4714 EC supports specifying Debug port range
● [EC-FLINK] LINKIS-5023 Flink engine plug-in supports reading user-defined log4j configuration
● [PES] LINKIS-4838 file reading and writing interface supports more parameters and operations
● [LM] LINKIS-4850 LM supports management of K8s resources
● [PE] LINKIS-4847 Optimize the number of modules and merge public data source modules
● [PE] LINKIS-4853 optimizes the number of modules and merges the public client module into the pes-client module
● [PE] LINKIS-4854 Optimize the number of modules to merge multiple modules of public services
● [PE] LINKIS-4934 data source service supports merging and separate deployment
● [EC-FLINK] LINKIS-5025 Flink engine plug-in supports loading default configuration
● [EC-JDBC] LINKIS-5007 JDBC engine plug-in supports task concatenation between multiple tasks
Repair function
● [EC-Flink]LINKIS-5041 Fix the problem of interactive Flink SQL printing status acquisition error log
● [MDS]LINKIS-4998 ES data source is compatible with 6.X and 7.X
● [EC-SPARK]LINKIS-4996 SPARK engine plug-in Scala task supports printing error information to the task log
● [ENTRANCE]LINKIS-4967 Fix the parsing error caused by semicolon in SQL parsing comment
● [EC]LINKIS-4920 Fixed the problem that result set null is returned as string NULL
● [CLI]LINKIS-4919 Client fixes concurrent NPE problem
● [CG]LINKIS-4915 Fixed the problem of high incorrect selection load in LM’s ECM selection logic.
● [LM]LINKIS-4860 Fix the problem of Chinese garbled characters in linkis-httpclient post interface
● [LM]LINKIS-4771 Linkis-Cli once job submission should be in Once mode
● [LM]LINKIS-4731 Kill EC script should skip the ECM process
Detailed guidance
● Overview of this version: https://linkis.apache.org/zh-CN/download/release-notes-1.5.0
● For detailed installation and deployment, see the guide: https://linkis.apache.org/zh-CN/docs/latest/deployment/deploy-quick
● Official download link: https://linkis.apache.org/zh-CN/download/main
community contributor
The release of Apache Linkis 1.5.0 is inseparable from the contributors of the Linkis community. Thanks to all community contributors, including but not limited to the following Contributors (listed in no particular order):
