Overview
In EdgeX Geneva version, EMQ X Kuiper-SQL-based lightweight streaming data processing software is integrated with EdgeX. Before entering this tutorial, let us take some time to understand some basic knowledge of Kuiper. EMQ X Kuiper is a lightweight IoT edge analysis and streaming open source software implemented by Golang. It can run on various resource-constrained edge devices. Kuiper supports streaming data processing based on 源 (Source), SQL (业务逻辑处理), 目标 (Sink).
-
Source: The data source of streaming data, such as data from an MQTT server . In the case of EdgeX, the data source is the EdgeX message bus, which can be from ZeroMQ or MQTT server;
-
SQL: SQL is where you specify business logic for streaming data processing. Kuiper provides SQL statements to extract, filter and transform data;
-
Sink: The target is used to send the analysis result to a specific target. For example, sending the analysis result to another MQTT server, or an HTTP Rest address;
To use Kuiper, you generally need to complete the following three steps.
- Creating a stream is where you define the data source
- Write rules
- Write SQL for data analysis
- Specify a destination to save analysis results
- Deploy and run rules
This tutorial describes how to use Kuiper to process data from the EdgeX message bus.
Kuiper EdgeX integration
Between different microservices, EdgeX uses the message bus for data exchange. It contains an abstract message bus interface, and implements ZeroMQ and MQTT respectively, and supports information exchange between different microservices. The integration of Kuiper and EdgeX consists of the following three parts,
-
An EdgeX message bus source has been extended to support receiving data from the EdgeX message bus
-
In order to analyze the data, Kuiper needs to know the format of the incoming data stream. Generally speaking, it is best for users to specify the format of the stream data to be analyzed when creating the stream. As shown below, a
demostream containing a namedtemperaturefield. This is very similar to when creating table definitions in a relational database. After the flow definition is created, Kuiper can type-check the incoming data at compile or run time, and the corresponding error will be reported to the user.CREATE STREAM demo (temperature bigint) WITH (FORMAT="JSON"...)However, in EdgeX, the data type definition
Core contract Servicehas been specified in EdgeX . In order to improve the user experience, users can not specify the data type when creating a stream. Kuiper will source initialization rules, fromCore contract Serviceacquiring all of thevalue descriptorsdefinitions (so if there is any change in the data type definition, you need to restart the rules). When receiving data from the message bus, it will be converted to the corresponding data type based on the rule . -
Extended support for EdgeX message bus destination (sink), used to write processing results back to EdgeX message bus. Users can also choose to send the analysis results to the RestAPI interface that Kuiper has previously supported.
Run EdgeX Docker instance
Open the EdgeX develop-scripts project and download the Geneva version of the Docker compose file, then start all EdgeX containers.
# docker-compose -f ./docker-compose-nexus-redis-no-secty.yml up -d --build
After all containers boot is completed, use the docker pscommand to determine all of the container has started normally.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5618c93027a9 nexus3.edgexfoundry.org:10004/docker-device-virtual-go:master "/device-virtual --p…" 37 minutes ago Up 37 minutes 0.0.0.0:49990->49990/tcp edgex-device-virtual
fabe6b9052f5 nexus3.edgexfoundry.org:10004/docker-edgex-ui-go:master "./edgex-ui-server" 37 minutes ago Up 37 minutes 0.0.0.0:4000->4000/tcp edgex-ui-go
950135a7041d emqx/kuiper:0.3.1 "/usr/bin/docker-ent…" 37 minutes ago Up 37 minutes 0.0.0.0:20498->20498/tcp, 9081/tcp, 0.0.0.0:48075->48075/tcp edgex-kuiper
c49b0d6f9347 nexus3.edgexfoundry.org:10004/docker-support-scheduler-go:master "/support-scheduler …" 37 minutes ago Up 37 minutes 0.0.0.0:48085->48085/tcp edgex-support-scheduler
4265dcc2bb48 nexus3.edgexfoundry.org:10004/docker-core-command-go:master "/core-command -cp=c…" 37 minutes ago Up 37 minutes 0.0.0.0:48082->48082/tcp edgex-core-command
4667160e2f41 nexus3.edgexfoundry.org:10004/docker-app-service-configurable:master "/app-service-config…" 37 minutes ago Up 37 minutes 48095/tcp, 0.0.0.0:48100->48100/tcp edgex-app-service-configurable-rules
9bbfe95993f5 nexus3.edgexfoundry.org:10004/docker-core-metadata-go:master "/core-metadata -cp=…" 37 minutes ago Up 37 minutes 0.0.0.0:48081->48081/tcp, 48082/tcp edgex-core-metadata
2e342a3aae81 nexus3.edgexfoundry.org:10004/docker-support-notifications-go:master "/support-notificati…" 37 minutes ago Up 37 minutes 0.0.0.0:48060->48060/tcp edgex-support-notifications
3cfc628e013a nexus3.edgexfoundry.org:10004/docker-sys-mgmt-agent-go:master "/sys-mgmt-agent -cp…" 37 minutes ago Up 37 minutes 0.0.0.0:48090->48090/tcp edgex-sys-mgmt-agent
f69e9c4d6cc8 nexus3.edgexfoundry.org:10004/docker-core-data-go:master "/core-data -cp=cons…" 37 minutes ago Up 37 minutes 0.0.0.0:5563->5563/tcp, 0.0.0.0:48080->48080/tcp edgex-core-data
9e5091928409 nexus3.edgexfoundry.org:10004/docker-support-logging-go:master "/support-logging -c…" 37 minutes ago Up 37 minutes 0.0.0.0:48061->48061/tcp edgex-support-logging
74e8668f892c redis:5.0.7-alpine "docker-entrypoint.s…" 37 minutes ago Up 37 minutes 0.0.0.0:6379->6379/tcp edgex-redis
9b341bb217f9 consul:1.3.1 "docker-entrypoint.s…" 37 minutes ago Up 37 minutes 0.0.0.0:8400->8400/tcp, 8300-8302/tcp, 8301-8302/udp, 8600/tcp, 8600/udp, 0.0.0.0:8500->8500/tcp edgex-core-consul
ed7ad5ae08b2 nexus3.edgexfoundry.org:10004/docker-edgex-volume:master "/bin/sh -c '/usr/bi…" 37 minutes ago Up 37 minutes edgex-files
Create flow
This step is to create a stream that can consume data from the EdgeX message bus. There are two ways to support the management flow, you can choose the way you like.
Method 1: Use Rest API
Please note: The Kuiper Rest interface in EdgeX uses a 48075port instead of the default 9081port. So when EdgeX calls Kuiper Rest, please replace all 9081 in the document with 48075.
Please $kuiper_serverreplace the address Kuiper instance running locally.
curl -X POST \
http://$kuiper_server:48075/streams \
-H 'Content-Type: application/json' \
-d '{
"sql": "create stream demo() WITH (FORMAT=\"JSON\", TYPE=\"edgex\")"
}'
For other APIs, please refer to this document .
Method 2: Use Kuiper command line
Use the following command to enter the running Kuiper docker instance.
docker exec -it kuiper /bin/sh
Use the following command to create a named demo-defined flow.
bin/cli create stream demo'() WITH (FORMAT="JSON", TYPE="edgex")'
For other command lines, please refer to this document .
Now the flow has been created, but you may be curious about how Kuiper know the address and port of the message bus, because such information is CREATE STREAMnot specified. In fact this information is in the configuration file etc/sources/edgex.yamlspecified, you can type in the command line window cat etc/sources/edgex.yamlcontents to view the files. If you have a different server, port and service address, please update the corresponding configuration. As mentioned before, these configuration options can be overridden when the container starts.
#Global Edgex configurations
default:
protocol: tcp
server: localhost
port: 5566
topic: events
serviceServer: http://localhost:48080
.....
For more information about configuration files, please refer to this document .
Create rules
Let's create a rule to send the analysis result to the MQTT server. For the relevant configuration of the MQTT target, please refer to this link . Similar to the process of creating a flow, you can choose to use REST or the command line to manage the rules.
The following example will select all the eventsall the data, the results will be the subject of
- Released to the public MQTT server
broker.emqx.iothemeresulton; - Print to log file
Option 1: Use Rest API
curl -X POST \
http://$kuiper_server:9081/rules \
-H 'Content-Type: application/json' \
-d '{
"id": "rule1",
"sql": "SELECT * FROM demo",
"actions": [
{
"mqtt": {
"server": "tcp://broker.emqx.io:1883",
"topic": "result",
"clientId": "demo_001"
}
},
{
"log":{}
}
]
}
Option 2: Use Kuiper command line
You can use any editor to create a rule, copy the following content into the editor, and name it rule.txt.
{
"sql": "SELECT * from demo",
"actions": [
{
"mqtt": {
"server": "tcp://broker.emqx.io:1883",
"topic": "result",
"clientId": "demo_001"
}
},
{
"log":{}
}
]
}
In the running container, execute the following command.
# bin/cli create rule rule1 -f rule.txt
Connecting to 127.0.0.1:20498...
Creating a new rule from file rule.txt.
Rule rule1 was created successfully, please use 'cli getstatus rule rule1' command to get rule status.
If you want to send the results to other targets, please refer to the Kuiper in support of other objectives . You can now look at the log/stream.logdetails in the log files, see the rules.
time="2020-04-17T06:32:24Z" level=info msg="Serving kuiper (version - 0.3.1-4-g9e63fe1) on port 20498, and restful api on port 9081. \n" file="server.go:101"
time="2020-04-17T06:32:24Z" level=info msg="The connection to edgex messagebus is established successfully." file="edgex_source.go:95" rule=rule1
time="2020-04-17T06:32:24Z" level=info msg="Successfully subscribed to edgex messagebus topic events." file="edgex_source.go:104" rule=rule1
time="2020-04-17T06:32:24Z" level=info msg="The connection to server tcp://broker.emqx.io:1883 was established successfully" file="mqtt_sink.go:161" rule=rule1
time="2020-04-17T06:32:25Z" level=info msg="Get 24 of value descriptors from service." file="edgex_source.go:223"
time="2020-04-17T06:32:25Z" level=info msg="sink result for rule rule1: [{\"int32\":-697766590}]" file="log_sink.go:16" rule=rule1
time="2020-04-17T06:32:25Z" level=info msg="sink result for rule rule1: [{\"int8\":-47}]" file="log_sink.go:16" rule=rule1
time="2020-04-17T06:32:25Z" level=info msg="sink result for rule rule1: [{\"int16\":-318}]" file="log_sink.go:16" rule=rule1
time="2020-04-17T06:32:25Z" level=info msg="sink result for rule rule1: [{\"int64\":-8680421421398846880}]" file="log_sink.go:16" rule=rule1
time="2020-04-17T06:32:31Z" level=info msg="sink result for rule rule1: [{\"bool\":true}]" file="log_sink.go:16" rule=rule1
Monitoring analysis results
Because all of the analysis results are to be published tcp://broker.emqx.io:1883, you can directly use the following mosquitto_subcommand to monitor the results, you can also refer to other MQTT client tools .
# mosquitto_sub -h broker.emqx.io -t result
[{"bool":true}]
[{"bool":false}]
[{"bool":true}]
[{"randomvalue_int16":3287}]
[{"float64":8.41326e+306}]
[{"randomvalue_int32":-1872949486}]
[{"randomvalue_int8":-53}]
[{"int64":-1829499332806053678}]
[{"int32":-1560624981}]
[{"int16":8991}]
[{"int8":-4}]
[{"bool":true}]
[{"bool":false}]
[{"float64":1.737076e+306}]
...
You can also type the following commands to view the status of the rule execution. Relevant REST API for viewing rule status is also provided, please check the relevant documentation .
# bin/cli getstatus rule rule1
Connecting to 127.0.0.1:20498...
{
"source_demo_0_records_in_total": 29,
"source_demo_0_records_out_total": 29,
"source_demo_0_exceptions_total": 0,
"source_demo_0_process_latency_ms": 0,
"source_demo_0_buffer_length": 0,
"source_demo_0_last_invocation": "2020-04-17T10:30:09.294337",
"op_preprocessor_demo_0_records_in_total": 29,
"op_preprocessor_demo_0_records_out_total": 29,
"op_preprocessor_demo_0_exceptions_total": 0,
"op_preprocessor_demo_0_process_latency_ms": 0,
"op_preprocessor_demo_0_buffer_length": 0,
"op_preprocessor_demo_0_last_invocation": "2020-04-17T10:30:09.294355",
"op_filter_0_records_in_total": 29,
"op_filter_0_records_out_total": 21,
"op_filter_0_exceptions_total": 0,
"op_filter_0_process_latency_ms": 0,
"op_filter_0_buffer_length": 0,
"op_filter_0_last_invocation": "2020-04-17T10:30:09.294362",
"op_project_0_records_in_total": 21,
"op_project_0_records_out_total": 21,
"op_project_0_exceptions_total": 0,
"op_project_0_process_latency_ms": 0,
"op_project_0_buffer_length": 0,
"op_project_0_last_invocation": "2020-04-17T10:30:09.294382",
"sink_sink_mqtt_0_records_in_total": 21,
"sink_sink_mqtt_0_records_out_total": 21,
"sink_sink_mqtt_0_exceptions_total": 0,
"sink_sink_mqtt_0_process_latency_ms": 0,
"sink_sink_mqtt_0_buffer_length": 1,
"sink_sink_mqtt_0_last_invocation": "2020-04-17T10:30:09.294423"
to sum up
In this tutorial, we introduced a very simple example of using the EdgeX Kuiper rule engine. If you find any problems during use, please report the problem to EdgeX or Kuiper Github.
More exercises
The current rules do not filter any data sent to Kuiper, so how to filter the data? Please use the delete rule , and then try to change the SQL statement, after finishing the change, redeploy the rule. At this time, if you monitor the result topic of the MQTT service, check whether the relevant rules work?
Extended reading
- Read the EdgeX source for more details, and type conversions.
- How to use the meta function to extract more information sent in the EdgeX message bus? When the device service sends data to the bus, some additional information is also sent along with it, such as time creation time, id, etc. If you want to use this information in SQL statements, please refer to this article.
- EdgeX Message Bus Target . This document describes how to use EdgeX Message Bus Target. If you want to send the analysis results to the message bus, you may be interested in this article.
For more information about EMQ X Kuiper, please refer to the following resources.
Author: EMQ X
Original link: https://www.emqx.io/cn/blog/kuiper-becomes-edgex-rule-engine
Copyright: This article is EMQ original, please indicate the source