In the process of communicating with developers, I found that many developers, especially those who are new to Taier , have many misunderstandings and problems with local debugging. This article briefly introduces the three most frequently asked questions, service compilation, configuration & local operation, and how to run Flink-standalone in Taier. I hope to exchange and learn with you.
service compilation
In this chapter, we will introduce the two major plug-ins WorkerPlugins and DataSourcePlugin in service compilation , and the role of Taier's front-end and back-end UI & datadevelopment.
The role of WorkerPlugins
After running tasks in Taier-UI, the platform binds to tenants through clusters in Taier-data-develop, and then obtains different WorkerPlugins through the current tenant binding component types and version numbers in the cluster, and through different component types and versions No. to submit the task. The following figure is the overall operation structure diagram:
Compilation of WorkerPlugins
This is a necessary option when running tasks. When we need local debugging or deployment, the compilation of WorkerPlugins must be carried out. After compilation, a directory of WorkerPlugins will be obtained. For the specific compilation process, please refer to the video link at the end of the article. Demonstration.
The role of DataSourcePlugin
After introducing the WorkerPlugins plug-in, let's introduce another plug-in, DataSourcePlugin .
In Taier-UI, we can configure many different types of data sources, such as MySQL, Doris, Oracle, etc., and these functions are implemented relying on the powerful DataSourcePlugin. At the same time, when using the GUI task configuration related functions in offline synchronization , the acquisition of database information also relies on DataSourcePlugin to complete.
Compilation of DataSourcePlugin
This is a necessary option when running tasks. When we need local debugging or deployment, the compilation of DataSourcePlugin must be carried out. After compilation, a directory of DataSourcePlugin will be obtained. For the specific compilation process, please refer to the video link at the end of the article. Demonstration.
The role of Taier-UI
In Taier-UI, we can configure different types of data sources, create tasks, task operation and maintenance , submit scheduling, cluster configuration, cluster binding and other operations.
The role of TaierDataDevelop
All back-end service APIs that operate in Taier-UI are supported by TaierDataDevelop, and the service mainly interacts with the front-end and back-end.
Configure & run locally
This section mainly introduces the configuration of TaierDataDevelop. Here, related configurations such as the port ZK, WorkerPlugins, and DataSourcePlugin database of the back-end service , the startup of the front-end and back-end, and the cluster configuration (Flink-standalone) and binding are performed.
For the specific code flow, please see the demonstration explanation in the video link at the end of the article.
Run Flink-Standalone practice
Configure the cluster
When the task is running, through the configured CDH cluster, use the configured YARN to assemble the task, and submit the task to computing engines such as Flink, Doris, and Spark through ChunJun or directly.
Configure & run tasks
Assemble the task configuration through the task GUI, including data sources and destinations, and perform task configuration through field mapping, task custom parameters and other related configurations.
Video course & PPT acquisition
Video lessons:
https://www.bilibili.com/video/BV19M411L7f2/?spm_id_from=333.999.0.0
Courseware acquisition:
https://www.dtstack.com/resources/1031
"Data Stack Product White Paper": https://www.dtstack.com/resources/1004?src=szsm
"Data Governance Industry Practice White Paper" download address: https://www.dtstack.com/resources/1001?src=szsm If you want to know or consult more about Kangaroo Cloud big data products, industry solutions, and customer cases, visit Kangaroo Cloud official website: https://www.dtstack.com/?src=szkyzg
At the same time, students who are interested in big data open source projects are welcome to join "Kangaroo Cloud Open Source Framework DingTalk Technology qun" to exchange the latest open source technology information, qun number: 30537511, project address: https://github.com/DTStack