Apache Linkis(incubating) 1.1.1 release

Introduction to Linkis Version 1.1.1

The 1.1.1 version released this time mainly supports UDF multi-version control; supports uploading jar packages/script materials of UDF functions to the hdfs file system managed by BML; submit tasks support Yarn queue resource usage statistics collection and viewing; The ECM management page can support the viewing of running engine log files; newly added support for the data virtualization engine OpenLooKeng.

GitHub ：https://github.com/apache/incubator-linkis

abbreviation:

• EC: Engineconn

• ECM: EngineConnManager

• ECP: EngineConnPlugin

• EC: EngineConn

• DMS: Data Source Manager Service

• MDS: MetaData Manager Service

• LM: Linkis Manager

• BML: BigData Material Library

What's New in the Version

New feature 1: Support proxy user mode

When linkis executes the task submitted by the user, the main process of linkis will switch to the corresponding user through sudo -u, and then execute the corresponding engine startup command, which requires creating a corresponding system user for each {submit user} in advance, and Configure the relevant environment variables. For new users, a series of environment initialization preparations are required. Frequent user changes will increase operation and maintenance costs, and there are too many users, so resources cannot be configured for a single user, and resources cannot be well managed. The support of the proxy user mode can realize that the tasks of user A can be delegated to the designated proxy user, and the execution entry can be converged uniformly. Subsequent DSS plans to integrate the code user mode of Linkis (not yet released), and can select the proxy user of the currently logged-in user through the DSS management console page.

New feature 2: UDF supports multi-version control

Before version 1.1.1, UDF multi-version control could not be supported, and the historical version of a UDF function could not be viewed and rolled back, and the UDF function jar package/script material was stored on the host of the server. If you change the server, you need to Do synchronous migration of UDF function jar package/script material. The linkis_ps_udf_version table is added to support multi-version storage of UDF functions. It relies on the existing BML material management function of linkis to upload jar/script materials of udf functions to the hdfs file system managed by BML.

For specific usage, see: https://linkis.apache.org/zh-CN/docs/latest/user_guide/udf

New feature 3: Added data virtualization openLooKeng engine plug-in

openLooKeng is used to support data exploration, ad hoc query and batch processing with near real-time latency of 100+ milliseconds to minutes without moving data. openLooKeng also supports hierarchical deployment, enabling geographically remote openLooKeng clusters to participate in the same query. With its cross-region query plan optimization capabilities, queries involving remote data can achieve near "local" performance. The newly added openLooKeng engine plug-in enables Linkis to have data virtualization capabilities, and supports the submission of cross-source heterogeneous queries, cross-domain and cross-DC query tasks.

For specific usage, see: https://linkis.apache.org/zh-CN/docs/latest/engine_usage/openlookeng

New feature 4: View engine process log file page

For the started engine process, you can directly view the content of the running log file of the process on the Management Console > ECM Management > a specific engine list page of an ECM. It is convenient for troubleshooting some abnormal problems and viewing key log information when the engine is running.

function enhancement

• [Gateway][Linkis-1430] For the Token authentication method, the Token acquisition is adjusted from the configuration file to the database table

• [Entrance][Linkis-1642] Optimize the excel export interface resultsetToExcel: support passing the number of rows of downloaded data

• [Entrance][Linkis-1733] Add support for more default time variables related to run_date

• [Entrance][Linkis-1794] Add write limit to the data size of a single row in the result set, and optimize the OOM problem caused by large result sets

• [DMS-Common][Linkis-1757] Supports the configuration of Hive's metadata administrator, the administrator can obtain the metadata information of all Hive database tables through the interface

• [Common][Linkis-1799] Optimize the segmentation of service logs: adjust the log history segmentation time from one day to one hour

• [Common][Linkis-1921] Optimize Jackson's dependency management method: manage jackson dependencies uniformly through jackson-bom, and upgrade to version 2.11.4

• [ECM][Linkis-1779] Optimize the status monitoring logic of ECM instances, increase the judgment of heartbeat reporting time, and fix the problem of wrong judgment caused by Eureka performance problems

• [ECM][Linkis-1930] ECM resource is not checked when optimizing resource check

• [Web][Linkis-1596] Optimize the use of the interface for viewing the task log of the management console, and fix the problem that the log cannot be refreshed and displayed in time for the running job

• [Web][Linkis-1650] linkis management console - global history page, support searching and filtering historical task data by creator information

Repair function

• [Entrance][Linkis-1623] Fix LogPath and ResultSetPath error using submitUser as executeUser

• [Entrance][Linkis-1640] Fix the problem that LogReader uses a singleton InputStream, the log is lost, and the latest persistent log cannot be read

• [Entrance][Linkis-2009] Fix the problem of memory leak caused by thread resources in Entrance service not being closed

• [Entrance][Linkis-1901] Replaced the cache in EntranceFactory with Guava Cache, fixed the problem that the user modified the concurrency parameter and it could not take effect

• [Entrance][Linkis-1986] Fix the abnormal number of lines obtained in the Entrance real-time log, resulting in the problem of duplicate logs obtained

• [ECM][Linkis-1714] By reducing the default memory size of EC Java and adding a retry log for EC applications, optimize the "Cannot allocate memory" exception in EC

• [ECM][Linkis-1806] Optimize the life cycle processing logic of EC. When ECM starts EC and the status is Failed due to insufficient queue resources and timeout, the EC process will be killed.

• [Common][Linkis-1721] Fix the issue that hdfsFileSystem is not refreshed when Kerberos authentication fails

• [UDF][Linkis-1728] Optimized the problem that the API interface of /api/rest_j/v1/udf/all occasionally takes a long time to query

• [Config][Linkis-1859] Fix the problem of abnormal primary key duplication in the management console parameter configuration saveFullTree interface

• [ Client ][Linkis-1739] Fix ujes-client's request, there is a misspelling of the parameter that causes the parameter to fail to pass

• [Client][Linkis-1783] Fix the problem that the default configuration of the task creator creator parameter does not take effect

• [Client][Linkis-1821] Fix ujes-client request entity class GetTableStatisticInfoAction parameter is missing

• [EC][Linkis-1765] Fix the blocking problem that EC triggers tryShutdown when the task is running

• [LM-AppManager][Linkis-1814] Fixed the problem that the response information returned by the createEngineConn interface of EngineRestfulApi was incorrect, resulting in NPE in client calls.

• [Web][Linkis-1972] Remove the dss related interface code left but not used by the management console for historical reasons

• [EC-Spark][Linkis-1729] Added SparkPreExecutionHook function, compatible with the old package name before Linkis (com.webank.wedatasphere.linkis)

• [EC-JDBC][Linkis-1851] Fix the jdbc engine, when there are multiple sql statements in one task execution, it cannot be executed normally

• [EC-JDBC][Linkis-1961] Fix the problem that the log cannot be printed normally due to the SLF4J dependency problem when the jdbc engine starts

• [Gateway][Linkis-1898] Fix the problem that the initial domain name cannot be set when the GatewaySSOUtils user successfully logs in to generate a cookie

other

• [License][Linkis-2110] Removed the binary file .mvn/wrapper/maven-wrapper.jar in the source code, and adjusted .mvn/* related LICENSE instructions

• [License][Linkis-2113] Upgrade py4j-0.10.7-src.zip to py4j-0.10.9.5-src.zip, solve the problem of unclear license agreement for some files in version 0.10.7, update py4j-*.src license file

Detailed Guidelines

• Overview of this release:  https://linkis.apache.org/en-US/docs/latest/release

• For detailed installation and deployment, see the guide: https://linkis.apache.org/zh-CN/docs/latest/deployment/quick_deploy

• Official download link: https://linkis.apache.org/zh-CN/download/main

Contributor's Message

The release of Apache Linkis(incubating) 1.1.1 is inseparable from the contributors of the Linkis community. Thanks to all the community contributors, including but not limited to the following Contributors:

• AbnerHung、Alexkun、CCweixiao

• Davidhua1996、Fuu3214、Liveipool

• barry8023 、 casionon and 、demonray

• husofskyzy、jackxu2011、legendtkl

• lizheng920625, maidangdang4 4

• peacewong、seedscoder

 

 

How to get involved

(1) Novice task: claim the introductory task, see  https://github.com/apache/incubator-linkis/issues/1161 for details ;  

(2) Precipitation of works: Publish the content related to the open source construction of WeDataSphere, including but not limited to installation and deployment tutorials, use experience, case practice, etc. The form is not limited, please submit it to the assistant. Such as: Community Developer Column | MariaCarrie: Linkis1.0.2 Installation and Usage Guide

(3) Contributed code: PR and Issue;

(4) Answering questions: enthusiastically answering questions for developers, such as community groups answering developer questions, issue answering questions, etc.;

(5) Others: sandbox experience, participating in activities, becoming a community volunteer, etc.