Preface
DolphinDB is a high-performance distributed time series database developed by Zhejiang Zhiyu Technology Co., Ltd. It integrates a powerful programming language and a high-capacity and high-speed streaming data analysis system to provide rapid storage, retrieval, and analysis of massive structured data. and computing to provide one-stop solutions. The DolphinDB database supports millions of data writes per second, millisecond query response for trillions of data, and high compression ratio data storage. It is suitable for industrial Internet of Things and quantitative finance and other fields.
In order to further simplify the use of DolphinDB, we built a DolphinDB low-code platform on Node-RED. Through visual programming tools, we abstract the functions of DolphinDB into easy-to-understand nodes, making the operation process clearer. This visualization method not only improves the convenience for users to use the platform, but also greatly reduces the user's learning threshold. Even if users do not understand DolphinDB's programming language, they can easily complete various operations.
On our low-code platform, users only need to use simple drag-and-drop operations to achieve modeless writing, data cleaning, downsampling, equipment monitoring, exception monitoring and other functions, making data processing more convenient and efficient.
The following content will introduce in detail the characteristics of the Node-RED platform, the situation of the DolphinDB low-code platform, and the method of developing DolphinDB nodes.
About Node-RED
Node-RED is an open source visual programming tool that makes it easy to connect, orchestrate, and automate IoT devices, APIs, and online services. It enables users to create applications by simply dragging and connecting nodes, where each node represents a specific feature or service. You can think of it like a jigsaw puzzle. By dragging and connecting different pieces, you can create various applications.
For example, people would like to be able to turn on the air conditioner on the way home and enjoy the coolness when they get home. This requirement can be easily achieved with the help of Node-RED. This process can be achieved by simply dragging the WeChat node, message processing node, and air conditioning control node, and then connecting and setting them. In this way, just send a message on WeChat to control the switch of the air conditioner in your home.
Node-RED is built on Node.js, so it is lightweight and efficient. It takes full advantage of the event-driven and non-blocking model of Node.js and performs well on low-cost hardware at the edge of the network (such as Raspberry Pi) and in the cloud. This makes Node-RED a very useful tool for a variety of scenarios.
Node-RED's node extension library includes many built-in nodes and features such as database access, web services, and email notifications that greatly simplify the development process. It provides powerful functional components that allow users to quickly build complex IoT devices, APIs, and online services without writing cumbersome code.
In addition, Node-RED uses JSON format to store processes, which makes sharing, importing and exporting very convenient. Users can easily share their processes with others and get more solutions and inspiration from the community. This openness and collaboration provide users with more possibilities and flexibility.
Overall, Node-RED is a powerful and easy-to-use tool that provides a visual programming environment that enables users to easily connect, orchestrate, and automate IoT devices, APIs, and online services. Whether in the field of IoT, automation or data stream processing, Node-RED is a very practical and popular tool, bringing great convenience and efficiency to users.
DolphinDB low-code platform
Platform Introduction
DolphinDB low-code platform is a DolphinDB function visualization platform based on Node-RED. It has strong scalability and provides users with the possibility to develop custom function nodes. Users can develop function nodes suitable for their own business scenarios according to their own needs, thereby further enhancing the functionality and adaptability of the platform.
In the case introduction chapter, we will introduce the development method of DolphinDB nodes to help users make better use of this low-code platform and achieve more personalized needs.
The DolphinDB low-code platform case provided in this article provides more than ten nodes, divided into three categories: file operation, data cleaning and DolphinDB function nodes. The file operation category provides data loading nodes to facilitate users to load local files into DolphinDB's memory table. The data cleaning category provides function nodes such as format conversion, NULL filling, data deduplication, and data normalization to help users implement various data cleaning operations. In the DolphinDB function node category, function nodes such as data storage, downsampling, schemaless writing, flow table creation, and streaming data engine are provided, allowing users to easily use the functions of DolphinDB without mastering its programming language.
Instructions for use
The DolphinDB low-code platform is built on Node-RED, so Node-RED needs to be installed first. For the official installation tutorial of Node-RED, please refer to: Getting Started: Node-RED . After completing the installation of Node-RED, follow the steps below to install the DolphinDB node.
1. Download the DolphinDB node. Download the DolphinDB node installation package in the attachment node-red-contrib-dolphindb-main . For example, in the following example, download and extract it to the D:\Node_Red_Project folder. The contents of the file are as follows:
2. Install the DolphinDB node into Node-RED. Enter the .node-red directory (this directory should be under the C:\Users<username> directory), and then install it with npm node-red-contrib-dolphindb-main. The specific instructions are as follows:
Among them, D:\Node_Red_Project\node-red-contrib-dolphindb-main is the download path of the DolphinDB node installation package.
cd C:\Users\<用户名>\.node-red
npm install D:\Node_Red_Project\node-red-contrib-dolphindb-mainAfter successful installation, the following content will appear.
3. Use the DolphinDB node. Enter the command in cmd node-red to start the Node-RED service. The following content indicates that the Node-RED service is started successfully.
4. Open the web page http://localhost:1880. The DolphinDB node will appear in the left palette. Drag it out and use it.
DolphinDB low-code platform
Case introduction
Data processing flow
In the case of the DolphinDB low-code platform in this article, users can easily implement the data cleaning process in a drag-and-drop manner. The following will introduce how to load the data in the CSV file, perform data cleaning on it, and finally store the cleaned data into the DolphinDB database.
First, we have the following data in a CSV file:
| id | date | price | qty | tmp |
|---|---|---|---|---|
| 1 | 2018.01.02 | 70.832104 | 1719 | 1.2 |
| 1 | 2018.01.03 | 12.22557 | 3.2 | |
| 3 | 2018.01.04 | 8.695886 | 1656 | 0.4 |
| 4 | 2018.01.05 | 24.324535 | 2860 | 2 |
| 5 | 2018.01.06 | 0.443173 | 6874 | 3.8 |
| 6 | 2018.01.07 | 90.302176 | 3277 | -1.8 |
| 7 | 2018.01.08 | 78.556843 | 3424 | -0.6 |
| 8 | 2018.01.09 | 45.836447 | 8636 | 0.4 |
| 9 | 2018.01.10 | 57.416425 | 707 | 1.6 |
| 10 | 2018.01.11 | 98.879764 | 2267 | 4.2 |
The specific operation process of data cleaning is as follows:
- Load data from the CSV file into a DolphinDB in-memory table.
- Remove duplicate values from the id column to ensure that the data for each id is unique.
- Convert the format of the price column and unify the data to two decimal places.
- The NULL value of the qty column is filled with the average value to ensure data integrity.
- Perform Max-Min normalization on the tmp column to scale the data to the range of 0 to 1.
- Store the cleaned data into the DolphinDB database for subsequent data analysis and application.
The following is the process of implementing appeal data processing operations on the DolphinDB low-code platform.
- Drag the inject, file loading, format conversion, null filling, deduplication, data normalization and data storage nodes into the workspace
- Connect nodes according to the logical order of data cleaning.
- Click on the node to configure the corresponding parameters for the node.
- Click the Deploy button in the upper right corner of the workspace .
- Click the timestamp node to trigger the process to start. The cleaned data can then be viewed in the DolphinDB database.
Modeless writing
Schemaless writing is a database writing method that is particularly suitable for data collection by devices in IoT scenarios. In traditional relational databases, data writing requires the structure of the data table to be defined in advance, including field names, data types, constraints and other information. However, in the real world, many data are not suitable for strict table structures, such as various types of sensor data collected at different frequencies.
Schemaless writing solves this problem by allowing data to be written directly into the database without defining the table structure in advance. Data can be flexibly stored in its original format without additional data conversion and processing, thus greatly simplifying the process of data writing.
On the DolphinDB low-code platform, the process of implementing schemaless writing can be easily completed through two nodes. The following is the specific process to achieve modeless writing:
- Drag the Mqtt In node and the modeless write node to the workspace
- Connect two nodes
- Click on the node to set parameters. Set the server and message topic in the Mqtt In node, and set the database name in the schemaless write node.
- Click the deploy button.
After that, you can send messages to mqtt to achieve modeless writing. In modeless writing, when sending a message to mqtt, the message needs to be sent according to certain format requirements. Here, multiple messages similar to the following are sent to mqtt.
{
"tags":{
"deviceCode": "361RP01",
"logicalPositionId": "1",
"physicalPositionId": "AI361RP01T"
},
"fields":{
"propertyValue":1505
},
"time": "2023-07-22 15:53:08"
}If the operation is successful, a database named "Mqtt_DDB" will appear in DolphinDB, and the database will continue to receive messages from "Mqtt".
DolphinDB node development method
DolphinDB nodes are based on Node-RED, so developing DolphinDB nodes requires mastering the basic knowledge of Node-RED node development. Please refer to: Node-RED node development tutorial .
The DolphinDB installation package contains the core files dolphindb.html and dolphindb.js . Among them, dolphindb.html is mainly used to define the properties, editing dialog boxes and help text of DolphinDB nodes, while dolphindb.js is mainly used to define the functions of DolphinDB nodes.
The html file part of the DolphinDB node is similar to the ordinary Node-RED node development process, so I won't go into details here. We focus on the node function implementation method in dolphindb.js . You can refer to: JavaScript API for development.
Connecting to DolphinDB in JavaScript
- Import DolphinDB in JavaScript, set the connection configuration, and establish a connection with DolphinDB.
import { DDB } from 'dolphindb'
// 使用 WebSocket URL 初始化连接到 DolphinDB 的实例（不建立实际的网络连接）
let ddb = new DDB('ws://127.0.0.1:8848', {
// 是否在建立连接后自动登录，默认 `true`
autologin: true,
// DolphinDB 登录用户名，默认 `'admin'`
username: 'admin',
// DolphinDB 登录密码，默认 `'123456'`
password: '123456',
// 设置 python session flag，默认 `false`
python: false,
// 设置该选项后，该数据库连接只用于流数据
streaming: undefined
})
// 建立到 DolphinDB 的连接
await ddb.connect()- Create a configuration node and connect to DolphinDB
function DolphinConfigNode(n) {
RED.nodes.createNode(this, n);
this.url = n.url
this.name = n.name
this.client = new DDB(this.url, {
autologin: true,
username: this.credentials.username,
password: this.credentials.password,
python: false,
streaming: undefined
})
}
RED.nodes.registerType("dolphindb", DolphinConfigNode, {
credentials: {
username: { type: "text" },
password: { type: "password" }
}
});- DolphinDB node construction:
a. Node constructor. Nodes are defined by a constructor function, which can be used to create new instances of the node. The function is passed an object containing the properties set in the stream editor. The first thing it must do is call this RED.nodes.createNode function to initialize the characteristics shared by all nodes, and then the node-specific code can be executed.
//构造函数
function DolphinDBNode(config) {
RED.nodes.createNode(this,config);
// 获取 DolphinDB 配置结点
this.dolphindb = n.dolphindb;
this.dolphindbConfig = RED.nodes.getNode(this.dolphindb);
var client = this.dolphindbConfig.client;
//功能语句
}
//注册节点
RED.nodes.registerType("DolphinDB",DolphinDBNode);b. Receive messages. The node registers a listener for this event input to receive messages from upstream nodes in the stream. Once the message is received, the corresponding function is executed.
this.on('input', function(msg, send, done) {
//执行功能
});- function statement. In JavaScript, we use
eval()functions to execute DolphinDB scripts. Generally speaking, we divide the DolphinDB script into two parts: functionfuncand function callfuncall, then splice them together, and calleval()the function to execute the DolphinDB script.
var func=`def foo(a,b){
return a+b;
}`
var funcall=`foo(11,11)`
var secipt=func+funcall
const result = await ddb.eval(script)Case
Finally, we take the data normalization node as an example to introduce the function implementation code of the DolphinDB node.
//构造函数
function DolphinDataNormalizationNode(n) {
RED.nodes.createNode(this, n);
// 获取 DolphinDB 配置结点
this.dolphindb = n.dolphindb;
this.dolphindbConfig = RED.nodes.getNode(this.dolphindb);
if (!this.dolphindbConfig) { //检查是否成功获取了 DolphinDB 配置节点
this.error(RED._("dolphindb.errors.missingconfig"));
return;
}
var client = this.dolphindbConfig.client;
//将当前节点的上下文保存在变量 node 中
this.name=n.name;
this.tableName=n.tableName;
this.colName=n.colName;
this.normalizeType=n.normalizeType;
var node = this;
//DolphinDB 脚本
var func=`def minMaxNormalization(tableName,colName){
pt=objByName(tableName);
minNum=min(pt[colName]);
maxNum=max(pt[colName]);
pt[colName]=(pt[colName]-minNum)/(maxNum-minNum);
}
def ZSoreNormalization(tableName,colName){
pt=objByName(tableName);
meanNum=mean(pt[colName]);
stdNum=std(pt[colName]);
pt[colName]=(pt[colName]-meanNum)/stdNum;
}
def dataNormalization(tableName,colName,type){
if(type==1){
minMaxNormalization(tableName,colName);
}else if(type==2){
ZSoreNormalization(tableName,colName);
}
}`;
var funcall =`dataNormalization("${this.tableName}",\`${this.colName},${this.normalizeType})`;
var script =func+funcall;
console.log(script) //打印生成的 DolphinDB 脚本，用于调试和查看。
//注册监听器 input，一旦收到消息，则执行 DolphinDB 脚本
node.on('input', async function (msg, send, done) {
client.eval(script).then(res => {
send(msg)
done()
}).catch(err => {
console.log(err)
done(err)
})
});
}
//注册节点
RED.nodes.registerType("数据归一化", DolphinDataNormalizationNode);
Summarize
This article introduces the Node-RED platform. Node-RED allows users to easily create various applications by simply dragging and connecting nodes without writing cumbersome code.
DolphinDB low-code platform is built on Node-RED. DolphinDB functions are visualized as nodes through visual programming, making the operation process clear. Among them, data cleaning and schemaless writing cases demonstrate the simplicity and efficiency of the DolphinDB low-code platform.
More importantly, the DolphinDB low-code platform is highly scalable. Users can develop custom function nodes according to their own needs to further enhance the functionality and adaptability of the platform. This openness provides users with more possibilities and flexibility, allowing them to implement more personalized needs on the platform.
Through the Node-RED platform and DolphinDB low-code platform, users can easily build complex IoT devices, APIs and online services, implement data processing and storage, and accelerate the process of innovation and digital transformation.