Step-by-step tutorial MO | How to use DolphinScheduler to connect to MatrixOne

[Live broadcast preview] 99 yuan cloud virtual machine × 9 yuan cloud native architecture √

1 Overview

Apache DolphinScheduler is a distributed, easily scalable open source system for visual DAG workflow task scheduling. It provides a solution that can visually operate tasks, workflows, and data processing processes throughout the life cycle.

The main goal of Apache DolphinScheduler is to resolve complex big data task dependencies. It uses the DAG (Directed Acyclic Graph) streaming method to assemble tasks, allowing you to monitor the execution status of tasks in real time, and supports operations such as task retry, specified node recovery failure, pause, resume, and termination.

MatrixOne supports connection with the visual DAG workflow task scheduling system DolphinScheduler. This article will guide you how to connect to MatrixOne through DolphinScheduler and create a task workflow.

2. Preparation before starting

Installation and startup of MatrixOne have been completed.
Installation of DolphinScheduler has been completed.

3 Configure MySQL driver

1. Download the MySQL driver and copy it to the libs directory

After the installation is complete, you need to manually download the mysql-connector-java driver (version 8.0.16), and then copy it to the four directories under the DolphinScheduler installation directory: api-server/libs, alert-server/libs, master-server/libs and worker-server/libs.

Note
推荐使用 `mysql-connector-java-8.0.16.jar` 作为 MySQL 驱动包。

2. Restart DolphinScheduler

After copying the driver package, you need to restart the DolphinScheduler service. First enter the installation directory of DolphinScheduler, and then execute the following command to restart the DolphinScheduler service:

# 停止 Standalone Server 服务
bash ./bin/dolphinscheduler-daemon.sh stop standalone-server
# 启动 Standalone Server 服务
bash ./bin/dolphinscheduler-daemon.sh start standalone-server

3. Log in to DolphinScheduler

Use the default username admin and password dolphinscheduler123 to log in to the DolphinScheduler Web user interface by accessing http:// ip:12345/dolphinscheduler /ui , as shown in the following figure:

4. Create data source

Click Data Source Center > Create Data Source and fill in the MatrixOne data connection information. After completion, click Test Connection . If the connection is successful, click OK to save:

4 Create project workflow

1. Create a tenant

In the Security Center , click Create Tenant and fill in the tenant name, as shown in the following figure:

Note

在生产环境中，不建议使用 root 作为租户。

2. Create a project

In project management , click Create Project and fill in the project name, as shown below:

3. Create a workflow and add nodes

Click the name of the project created in the previous step , then click Create Workflow . Drag the SQL node from the left to the canvas on the right, fill in the node name, data source information, SQL type, SQL statement , and click OK . As shown below:

This step creates a table creation node, and SQL statements are used to create tables.

Next, create insert data and query data nodes similarly . The dependencies of these three nodes are as shown below, you can connect them manually:

The SQL statements for the three nodes are as follows:

#create_table
CREATE TABLE IF NOT EXISTS test_table (id INT AUTO_INCREMENT PRIMARY KEY, name
VARCHAR(255) NOT NULL)
#insert_data
INSERT INTO test_table (name) VALUES ('John Doe')
#select_data
SELECT * FROM test_table

Connect these three nodes according to their dependencies and click Save . Fill in the workflow name , select the previously created tenant , select the execution strategy as parallel , and click OK .

After creating the workflow, you can see the created workflow on the workflow relationship page, and its status is Workflow offline :

Similarly, you can also see the defined workflow on the workflow definition page, and its status is offline :

4. Go online and run the workflow

The workflow must be online before it can run. Click the Go Online button to bring the previously created workflow online:

After going online, the status of the workflow is as shown below:

Next, click the Run button, set the pre-start configuration parameters, and click OK:

Finally, return to the project overview to view the workflow and whether the following three tasks ran successfully, as shown in the following figure:

About MatrixOne

MatrixOne is a multi-mode database based on cloud-native technology that can be deployed in both public and private clouds. This product uses an original technical architecture that separates storage and computing, separation of reading and writing, and separation of hot and cold. It can simultaneously support multiple loads such as transaction, analysis, flow, timing, and vector under a set of storage and computing systems, and can perform real-time and on-demand Isolated or shared storage and computing resources. The cloud-native database MatrixOne can help users significantly simplify the increasingly complex IT architecture and provide minimalist, extremely flexible, cost-effective and high-performance data services.

Since their release, MatrixOne Enterprise Edition and MatrixOne Cloud Services have been applied in many industries such as the Internet, finance, energy, manufacturing, education, and medical care. Thanks to its unique architectural design, users can reduce hardware and operation and maintenance costs by up to 70%, increase development efficiency by 3-5 times, and at the same time respond more flexibly to changes in market demand and seize innovation opportunities more efficiently. When investing in the same hardware, MatrixOne can achieve several times more performance improvements.

MatrixOne adheres to the concept of open source and ecological co-construction. All core codes are open source and fully compatible with the MySQL protocol. It has also created multiple end-to-end solutions with partners to significantly reduce user migration.

Keywords: hyper-converged database, multi-mode database, cloud native database, domestic database.

MatrixOrigin official website: A new generation of hyper-converged heterogeneous open source database-MatrixOrigin (Shenzhen) Information Technology Co., Ltd. MatrixOne

Github 仓库：GitHub - matrixorigin/matrixone: Hyperconverged cloud-edge native database