Introduction to Streamis

Streamis is a streaming application development management system jointly built by WeBank, Tianyi Cloud, Xianweng Technology and Samoyed Cloud. Based on the framework capabilities of DataSphere Studio and the underlying Flink engine connected to Linkis , users can complete the development, debugging, release and production management of streaming applications at low cost.

In the future, it is also planned to use a workflow-style graphical drag-and-drop development experience to connect streaming applications into a streaming workflow with Source nodes, Dimension nodes, Transform nodes, Sink nodes, and Visualis nodes, allowing users to Learning costs to complete the development, debugging and release of streaming applications.

Github：https://github.com/WeBankFinTech/Streamis

Streamis 0.3.0 New Release Notes

Streamis 0.3.0 has some important architectural updates and optimizations, mainly supporting separate Flink applications and Flink management engine connectors, adding snapshots and applications with the same name before job startup, and supporting DSS proxy users. Increase the collection and write-back of Flink application log aggregation locally. Streamis 0.3.0 is compatible with Linkis multi-version (>= 1.1.2), but the separated related feature requires 1.4.0 or later.

Streamis 0.3.0 mainly adds the following features:

Flink tasks add a separate management mode (requires Linkis 1.4.0)
Add Flink management ec support (requires Linkis 1.4.0)
Add startup check for job task startup, check the snapshot address and the application with the same name on yarn (requires Linkis 1.4.0)
Support for DSS proxy users (requires DSS 1.1.2)
Collect and aggregate Yarn streaming application log callbacks, and write them locally on the streamis side. The collection terminal supports filtering callback logs according to different levels

abbreviation:

DSS: DataSphereStudio

new features

[Streamis-jobmanager] flink task adds separate management mode
[Streamis-jobmanager] Add flink management ec support
[Streamis-jobmanager] Streamis job task startup adds startup check
[Streamis-jobmanager] When starting a job, it will automatically check the application with the same name on yarn
[Streamis-jobmanager] streamis application parameter configuration import
[Streamis-jobmanager] collects and aggregates stream application logs, and writes them locally on the streamis side
[Streamis-appconn] Streamis data interface permission adjustment
[Streamis-appconn] Cooperate with DSS to switch proxy users
[Streamis-jobmanager] jobs support label filtering

function enhancement

[Streamis-Web] Rendering and interface adjustment of task configuration interface
[Streamis-jobmanager] Realize the automatic restart of streaming applications and optimize the power problem
[Streamis-jobmanager] Streamis task abort process adjustment, optimized job stop function, added scheduling
[Streamis-jobmanager] Added keyword filtering strategy on the collection side
[Streamis-jobmanager] Application initialization supports application online import and version management
[Streamis-jobmanager] Check the failure before the task starts to warn the person
[Streamis-jobmanager] supports multiple drive letters to store streaming application logs
[Streamis-projectmanager] Upload project resource files to add md5

bug fixes

[Streamis-Web] Fix the problem that the historical task log page cannot display the next page, and display the flink ec paged log in the log interface
[Streamis] Fixed code scanning bugs and smells

contributor

The release of Streamis 0.3.0 is inseparable from the contributors of the Streamis community, thanks to all community contributors.

Installation package

Streamis0.3.0 installation package: https://osp-1257653870.cos.ap-guangzhou.myqcloud.com/WeDatasphere/Streamis/0.3.0/wedatasphere-streamis-0.3.0-dist.tar.gz

Streamis0.3.0 appconn plug-in package: https://osp-1257653870.cos.ap-guangzhou.myqcloud.com/WeDatasphere/Streamis/0.3.0/streamis-appconn-0.3.0.zip

DSS1.1.2&Linkis1.4.0 one-click installation package: https://osp-1257653870.cos.ap-guangzhou.myqcloud.com/WeDatasphere/DataSphereStudio/1.1.2/dss_linkis_one-click_install_20230809.zip

— END —

How to become a community contributor

1 ►Official document contribution. Participate in community contributions by discovering deficiencies in documentation, optimizing documentation, and continuously updating documentation. Through document contribution, let developers be familiar with how to submit PR and truly participate in the construction of the community. Reference guide: Nanny-level tutorial: How to become an Apache Linkis document contributor

2 ►Code contribution. We have sorted out the simple and easy-to-start tasks in the community, which are very suitable for newcomers to contribute code. Please refer to the novice task list: https://github.com/apache/incubator-linkis/issues/1161

3 ►Contribution: Publish content related to WeDataSphere open source components, including but not limited to installation and deployment tutorials, usage experience, case practice, etc., in any form, please contribute to the assistant. For example:

4 ►CommunityQ&A: Actively answer questions in the community, share technology, help developers solve problems, etc.;

5 ►Others: Actively participate in community activities, become community volunteers, help community publicity, provide effective suggestions for community development, etc.;

The new version of Streamis 0.3.0, the streaming application development management system, is released