Write IoT stream processing data to Databend using LF Edge eKuiper

OSC invites you to party! 1028 Suzhou Yuan Founding Association, let’s hunt for treasures in the AI era

Author: Han Shanjie

Databend Cloud R&D Engineer

https://github.com/hantmac

LF Edge eKuiper

LF Edge eKuiper is a lightweight IoT edge analysis and streaming processing open source software implemented in Golang. It can run on various resource-limited edge devices. The main goal of eKuiper is to provide a streaming software framework at the edge (similar to Apache Flink (opens new window) ). eKuiper's rules engine allows users to provide SQL-based or graph-based (similar to Node-RED) rules to create IoT edge analytics applications in minutes. For a detailed introduction, please refer to [LF Edge eKuiper - Ultra-lightweight IoT edge stream processing software ( https://ekuiper.org/docs/zh/latest/ ).

Databend Sql Sink

eKuiper supports expansion in three aspects through Golang or Python 源 (Source). By supporting different sinks, users are allowed to send analysis results to different expansion systems. Databend as a Sink has also been integrated into the eKuiper plugin. The following is a case to show how to use eKuiper to write IoT stream processing data into Databend.SQL 函数目标 (Sink)

Compile eKuiper and Databend Sql Plugin

Kuiper

git clone https://github.com/lf-edge/ekuiper & cd ekuiper
make

Databend Sql Plugin

go build -trimpath --buildmode=plugin -tags databend -o plugins/sinks/Sql.so extensions/sinks/sql/sql.go

Copy the compiled sink plugin to the build directory:

cp plugins/sinks/Sql.so _build/kuiper-1.11.1-18-g42d9147f-darwin-arm64/plugins/sinks

Databend table creation

First create the target table ekuiper_test in Databend:

create table ekuiper_test (name string,size bigint,id bigint);

Start eKuiperd

cd _build/kuiper-1.11.1-18-g42d9147f-darwin-arm64 
./bin/kuiperd

The service starts normally:

Create streams and rules

eKuiper provides two ways to manage various flows, rules, and targets. One is to start the visual management interface through the [docker image] of ekuiper-manager ( https://hub.docker.com/r/lfedge/ekuiper) . One is to manage it through CLI tools. Here we use CLI.

create stream

Stream is the running form of data source connector in eKuiper. It must specify a source type that defines how to connect to the external resource. Here we create a stream to get data from the json file data source and send it to eKuiper.

First configure the file data source, where the connector's configuration file is located /etc/sources/file.yaml.

default:
  # 文件的类型，支持 json， csv 和 lines
  fileType: json
  # 文件以 eKuiper 为根目录的目录或文件的绝对路径。
  # 请勿在此处包含文件名。文件名应在流数据源中定义
  path: data
  # 读取文件的时间间隔，单位为ms。如果只读取一次，则将其设置为 0
  interval: 0
  # 读取后，两条数据发送的间隔时间
  sendInterval: 0
  # 是否并行读取目录中的文件
  parallel: false
  # 文件读取后的操作
  # 0: 文件保持不变
  # 1: 删除文件
  # 2: 移动文件到 moveTo 定义的位置
  actionAfterRead: 0
  # 移动文件的位置, 仅用于 actionAfterRead 为 2 的情况
  moveTo: /tmp/kuiper/moved
  # 是否包含文件头，多用于 csv。若为 true，则第一行解析为文件头。
  hasHeader: false
  # 定义文件的列。如果定义了文件头，该选项将被覆盖。
  # columns: [id, name]
  # 忽略开头多少行的内容。
  ignoreStartLines: 0
  # 忽略结尾多少行的内容。最后的空行不计算在内。
  ignoreEndLines: 0
  # 使用指定的压缩方法解压缩文件。现在支持`gzip`、`zstd` 方法。
  decompression: ""

Use the CLI to create a steam named stream1:

./bin/kuiper create stream stream1 '(id BIGINT, name STRING,size BIGINT) WITH (DATASOURCE="test.json", FORMAT="json", TYPE="file");'

The contents of the Json file are:

[
  {"id": 1,"size":100, "name": "John Doe"},
  {"id": 2,"size":200, "name": "Jane Smith"},
  {"id": 3,"size":300, "name": "Kobe Brant"},
  {"id": 4,"size":400, "name": "Alen Iverson"}
]

Create Databend Sink Rule

A rule represents a stream processing process and defines actions from inputting data into the stream's data source, through various processing logic, to inputting data into an external system. eKuiper has two ways to define the business logic of rules. Either use SQL/Action combination, or use the newly added graph API.

Here we define the business logic of the rule in a declarative manner by specifying sqlthe and attributes. actionsIn it, sqla SQL query is defined that is run against a predefined stream, which transforms the data. The output data can then be actionrouted to multiple locations.

Rules are defined by JSON. Here is the rule myRule.json ready to be created:

{
  "id": "myRule",
  "sql": "SELECT id, name from stream1",
  "actions": [
    {
      "log": {
      },
      "sql": {
        "url": "databend://databend:databend@localhost:8000/default?sslmode=disable",
        "table": "ekuiper_test",
        "fields": ["id","name"]
      }
    }
  ]
}

Execute CLI to create rules:

./bin/kuiper create rule myRule -f myRule.json

You can view the running status of the created rules:

./bin/kuiper getstatus rule myRule

After the rule is created, the data that meets the rule conditions will be sent to the target immediately. At this time, we look at the ekuiper_test table of Databend and can see that the data in the file data source has been written to Databend:

You can see that since our rule SQL only specifies the field id, nameonly these two fields are written here.

in conclusion

eKuiper is a stream processing software under EMQ. It is small in size and powerful in functions. It is widely used in many scenarios such as industrial Internet of Things, vehicle network, and public data analysis. This article describes how to use eKuiper to write IoT stream processing data to Databend.