The MQTT data sending component based on ETLCloud implements connection with BifroMQ and sends data to IoT devices

background

We have experienced ETLCloudoffline data integration, real-time data synchronization, heterogeneous data conversion, report and message push, custom conversion rules and other functions before, and can quickly complete the data integration that may be encountered in the work with zero code, visualization, and drag and drop. question.
Today let’s practice the MQTT data sending/EMQ data sending component. First use ETLCloudthe library table input component to obtain ClickHousethe data of , and then MQTTsend the data stream directly to the specified device using the protocol topicto realize sending instructions or data to IoT devices (EMQ compatible); in actual IoT scenarios, we sometimes send The client device sends messages or instructions in batches to realize remote configuration and control operations.

MQTT Broker selection

The mainstream MQTT Brokerones include: Mosquitto, EMQ, HiveMQ, VerneMQ, ActiveMQand cloud service providers that provide Internet of Things device access services, such as Alibaba Cloud, Huawei Cloud, etc. But here today we choose Baidu'sBifroMQ open source which has just been open sourced in the past two days MQTT Broker.
BifroMQIt is a Javahigh-performance, distributed message middleware based on implementation MQTT Broker, which seamlessly integrates native multi-tenant support. It is designed to support building large-scale IoT device connectivity and messaging systems. BifroMQDerived from the technology accumulation of Baidu IoT team for many years, it is currently used as the IoT Corebasic technology of Baidu Smart Cloud IoT core suite, which is a public cloud Serverlessservice. The following is a client tool EMQdeveloped using , which achieves the effects of establishing connection, subscription and publishing with .MQTTxBifroMQ

Dataset description

ClickHouseThe table creation statement in:

CREATE TABLE poetry.poetry (`id` Int32, `title` String, `yunlv_rule` String, `author_id` Int32, `content` String, `dynasty` String, `author` String) ENGINE = MergeTree() PRIMARY KEY id ORDER BY id SETTINGS index_granularity = 8192

Library table input component, used to read ClickHousedata SQL.

-- 按照朝代分组，统计不同朝代的诗词数量
SELECT dynasty, count(*) AS count FROM poetry.poetry GROUP BY dynasty;

Tool selection

ClickHouse database
Docker deploys ETLCloudV2.2
ETLCloud's library table input component, MQTT data sending/EMQ data sending component

Note：

The community version chosen here is lightweight and quick to start ETLCloudusing the deployment method: .Dockerdocker pull ccr.ccs.tencentyun.com/restcloud/restcloud-etl:V2.2
For simplicity, we also use Dockerdeployment mode to run BifroMQ:docker run -d --name biformq -p 1883:1883 bifromq/bifromq:latest

Purchase components

Because the "MQTT data sending/EMQ data sending" component is not included in ETLCloudthe community version, you first go to the official component library and choose to purchase it (the MQTT data sending/EMQ data sending component is free). Select the timing database (IOT device) - MQTT data transmission/EMQ data transmission - click to purchase in the official component library, and then you can view it on the component page I purchased (you need to log in first).

Install components

After the "MQTT Data Sending/EMQ Data Sending" component is successfully purchased, it can be installed on the "Data Processing Components" page of the management backend. Select remote installation, and a list of purchased components will appear; a new IoT componentETLCloud category is created here, select Install it under this component category and refresh the page after the installation is complete.

Create applications and processes

First create an application and fill in the basic application configuration information.

Next, create a data process and fill in the information.

Library table input and MQTT data sending practice

ClickHouseNext, reading statistical data from this database is completed through visual configuration and operation OLAP, and then the results are MQTTsent to the specified topic through the protocol. Client devices that subscribe to the topic will receive the data, thereby achieving remote management.

Data source configuration

Configuration Source: ClickHouse

Select the poetry database from which previous articles were migrated as the data source ClickHouse.

Configure Sink: MQTT

Select MQTTand fill in the server host and user password information.

Visual configuration process

After creating the process, you can click the "Process Design" button to enter the process visualization configuration page.

Library table input: ClickHouse

In the input component on the left, select "Library Table Input", drag it to the central process drawing area, and double-click to enter the configuration stage.

ClickHouseStep 1: Select the data source we configured and load ClickHousethe existing tables in it.

Step 2: You can generate a statement based on the selected table SQL. Here we use our custom query statistics statement.

Step 3: The definition of each field can be read from the table, and fields can be added and deleted. Here are the two fields we want to output.

Step 4: SQLData preview is automatically performed based on the statement. Such a check operation ensures the normal execution of subsequent operations.

IoT components: MQTT data sending

In the IoT component on the left, select "MQTT Data Sending", drag it to the central process drawing area, and double-click to enter the configuration stage.
Select the data source we created MQTT, customize a theme, fill in the data to be sent, and directly send the output data of the previous process node: {$!{data}}.

Finally, by 流程线connecting the start , library table input , MQTT data sending , and ending components respectively, the visual configuration of sending data to the specified topic using the MQTT protocol is completed. Done~

Note: Here, in order to view the output of the previous process node for MQTT data sending , the output mode of routing attributes configured with process lines is: output in the console log.

Run process

Save the process and run the process; you can then view the corresponding process logs and conversion logs, and visually monitor the migration progress. Judging from the results of the process, the client that subscribed to the corresponding topic successfully received the message.

Problem record

When running the process, an installed component error occurred: that is to say, the component we installed ourselves was not found. After that, I contacted the official technical staff and sent me a ETLSendMqttData.classfile. After placing it in the specified directory, cn.restcloud.etl.module.plugin.mqit ran successfully.

Error message: java.lang.Class-java.lang.Exception: Node: SendMqttDatadoes not exist, please download it from the component market or define it yourself!
Solution: Copy the component to the specified directory of the container: /usr/tomcat/webapps/ROOT/WEB-INF/classes/cn/restcloud/etl/module/plugin/mq.

# 从宿主机复制文件到容器的指定目录
[root@etl ~]# docker cp /opt/mq de63b29c71d0:/usr/tomcat/webapps/ROOT/WEB-INF/classes/cn/restcloud/etl/module/plugin/
                                             Successfully copied 6.66kB to de63b29c71d0:/usr/tomcat/webapps/ROOT/WEB-INF/classes/cn/restcloud/etl/module/plugin/
# 重启ETLCloud服务
[root@etl ~]# docker restart de63b29c71d0
de63b29c71d0

Summarize

BifroMQThe name is inspired by the rainbow bridge in Norse mythology Bifröst, which connects the human world Midgardand the world inhabited by gods. Asgard。 BifröstAs a solid and flexible channel between the two worlds, it BifroMQalso aims to become a hub connecting various systems or applications. Communication between them is achieved through message passing. This MQTTis very similar to the role middleware plays in distributed systems, which is to process and forward messages. In addition, Bifröstits sturdiness implies BifroMQexcellence in stability and reliability, while its flexibility symbolizes BifroMQadvantages in scalability and adaptability. Overall, 'BifroMQ' is a solid, flexible MQTTmiddleware that serves as a bridge between different systems or applications. "
The MQTT data sending/EMQ data sendingETLCloud component provided by can be used as a bridge between upper-layer business applications and remote device clients, sending messages or instructions in batches to client devices to achieve remote configuration and control operations.MQTT

Reference

If you have any questions or any bugs are found, please feel free to contact me.
Your comments and suggestions are welcome!