Hello everyone, I am Mr. Feet
The integration of flow and batch is becoming more and more popular, and it is awesome .
Before using java package flink-sql-client to submit script files, it was very frictional.
However, Dinky was discovered during the development of the stream-batch integrated platform , and it finally became silky smooth.
Today, I will recommend the low-key but powerful Dinky to my friends! Powerful, free yourself .
Dinky
Based on the secondary development of Apache Flink, no intrusion, out of the box, real-time is the future, and batch flow is integrated.
Make good use of Dinky, infinitely silky.
The official has a detailed usage tutorial, and it is already open source.
The most important thing is that you can join the exchange group and ask the author (a big guy who is more patient) at any time.
Dinky platform
To build Dinky, it is relatively simple to follow the official building tutorial step by step.
As a FlinSQL real-time computing platform, its core functions:
- Support for Flink native syntax, connectors, UDFs, etc.: Migrate Flink jobs to Dinky at almost zero cost.
- Enhanced FlinkSQL syntax: table-valued aggregate functions, global variables, CDC multi-source merge, execution environment, statement merge, shared session, etc.
Supports multiple versions of Flink: Supports the ability to serve as multiple versions of FlinkSQL Server and OpenApi. - Support DB SQL operations of external data sources: such as ClickHouse, Doris, Hive, Mysql, Oracle, Phoenix, PostgreSql, SqlServer, etc.
- Support real-time task operation and maintenance: job online and offline, job information, cluster information, job snapshot, exception information, job log, data map, ad hoc query, historical version, alarm record, etc.
For more functions, friends, go to the official website to see at a glance.
Dinky secondary development experience
Just download the source code and import the idea directly, which is no different from a normal import project.
Dinky, developed with java, is very friendly, who knows how to use such a volume of java now.
- When using Dinky as the stream batch platform, use it as the backend, and only need to redevelop the frontend according to the needs.
The days of looking at the source code are always very tiring. (It hurts to look at other people's code)
It must be understood, because Flink is really popular .
But the biggest feeling is that I have learned a lot about Flink and understood how the author encapsulates it.
Simply looking at the source code is always tiring. Only in actual work, you need to modify the function. Looking at the source code is very efficient and not boring.
Self-developed platform
The development of the offline module is currently underway, and the scheduling is a bit difficult.
I planned to use the familiar Airflow before, but found that the little dolphin is very popular, and I also want to learn, so I arranged it.
The real-time streaming module in the platform plans to integrate the Dinky source code in the future to make the platform more complete and powerful.
I wish you all success and success in the end!