IoT devices usually report data according to a certain sampling period, and the data collected by different devices is also different. The dimensions, frequency, and number of devices collected are relatively large. For example, a smart grid consists of smart substations, line transmission, smart interactive terminals, smart meters, smart central systems, etc. Each part has a large number of equipment and integrates a large number of sensors. With the continuous improvement of China's smart grid construction, its The scale of measuring points reaches more than tens of millions, and different measuring points will generate data at different frequencies such as millisecond level and second level.
IoT data has obvious time series characteristics and requires time series storage. Using a time series database is a better choice. In addition, IoT data collection involves a large number of device accesses and massive time series database transmissions, which puts great pressure on the access throughput of the message server, writing of back-end time series databases, and storage space consumption.
After the data is stored in the database, it needs to be provided for different businesses at the same time, such as data visualization, AI intelligent analysis, digital twins, application interface calls, etc., which also poses great challenges to the concurrency and throughput requirements of back-end time series database queries.
The combined technology stack of EMQ X message middleware and openGemini one-stop storage and analysis platform is fully capable of device access, data storage and analysis in the scenario. And in the long run, this technology combination can still meet the increasing demand for the number of access devices and query concurrency brought about by expected business growth.
This article takes common data visualization scenarios in the Internet of Things as an example to introduce openGemini + EMQ
An Introduction
The overall structure of the solution is shown in the figure below:
EMQ X introduction
EMQ EMQ _ _ _ _ _ _ If you want to purely use open source, you can use the combination of EMQ X (open source version) + Telegraf + openGemini + Grafana, which can also be achieved. Telegraf consumes MQTT messages from EMQ
What is openGemini
openGemini is Huawei Cloud's open source cloud-native distributed time series database, which has the advantages of high performance, high scalability, and high concurrency. Mainly oriented to the Internet of Things and operation and maintenance monitoring scenarios, it provides open source solutions for massive time series database processing and analysis. openGemini supports stand-alone and clusters, with various deployment forms, including edge, cloud, and physical machine deployment. It supports mainstream development languages and binary execution. , has no dependence on third-party components, and contains 260+ system operation monitoring indicators, which can provide real-time insight into the system operation status.
Introduction to Grafana
Grafana is a cross-platform, open source metric analysis and visualization tool that can query collected data and display it visually through flexible configuration. It can quickly and flexibly create client charts. The official library has a wealth of dashboard plug-ins, such as heat maps, line charts, charts and other display methods. Supports openGemini, Graphite, InfluxDB, OpenTSDB, Prometheus and other data sources. Custom rules can be created and notified to other messaging services or components.
Business scene
Assume that there is a batch of devices, each device has a Client ID, and all devices send data to the corresponding topic on the MQTT server through the MQTT protocol. The design of the topic is as follows:
devices/{client_id}/messagesThe data format sent by each device is JSON, and the temperature and humidity data collected by the sensor are sent.
{"temperature": 23,"humidity" : 21}
Now it is necessary to store the data in real time so that it can be viewed at any time in the future. The specific requirements are as follows:
-
Each device reports data at a frequency of every 5 seconds, and the database needs to store each piece of data;
-
Through the visualization system, you can view the average, maximum, and minimum values of temperature/humidity in any time interval, as well as the average temperature/humidity in all time periods.
The final display effect of the visualization platform currently listing the scenarios is as shown below. In the upper right corner of the dashboard, you can select the time interval and automatic refresh time. At this time, the device continues to send data, and the dashboard data values will change accordingly, achieving a more comprehensive visualization effect.
Environmental preparation
The official websites of the three major components all have installation package resources and tutorials for different operating systems:
-
EMQX official website
https://www.emqx.com/zh
-
openGemini official website
http://www.opengemini.org
-
Grafana official website
https://grafana.com
Configure EMQ X to store data in openGemini
-
On the Dashboard page of EMQ X, click Data Integration on the left -> Data Bridging -> InfluxDB
2. Configure openGemini timing database information
-
Version: V1
-
Server address: 127.0.0.1:8086, replace 127.0.0.1 with the actual IP address
-
Database name: the actual database name created internally by openGemini
-
Username and password: openGemini username and password (no need to fill in by default)
-
Enable TLS: Whether to start or not depends on whether openGemini enables HTTPS.
-
Define parsed data: Set the Measurement, Fields, Timestamp and Tags of the data. The key values all support variables and can be set using the row protocol.
3. Click the Create button to complete the data bridge creation
4. Create rules
After completing the data bridge creation process, you will continue to create a rule to specify the data that needs to be written:
SELECTclientid,payload.temperature as temperature,payload.humidity as humidityFROM"devices/1/messages"
-
On the Dashboard page, click Data Integration Rules on the left to create a rule
-
Enter the rule ID ' my_rule' and enter the rule in the SQL editor. Here, select to store the MQTT messages of the 'devices/1/messages' topic to InfluxDB. Please ensure that the field selected by the rule (SELECT part) contains the fields in step 2. Define variables used when parsing data. The rule SQL here is:
-
Add an action, select the Use data bridge forwarding option in the action drop-down box, and select the previously created data bridge (test)
5. Test
Use the MQTT X client to publish messages to the 'devices/1/messages' topic to simulate data collection. This operation will also trigger rules and data bridging processes to write data to openGemini:
mqttx pub -i emqx_c -t devices/1/messages -m '{"temperature": 23, "humidity": 21}' Visual configuration
1. Configure the data source and add the data source InfluxDB on the Grafana Dashboard page
Enter the connection parameters for configuration. By default, the key configuration information is as follows:
-
URL: Here we take http://localhost:8086 as an example. Localhos needs to be replaced according to the actual IP address.
-
Database: The actual database name created internally by openGemini. Here, db0 is used as an example.
2. Add a dashboard . After adding the data source, you need to create a dashboard. There are many related tutorials on the Internet, so I won’t go into details here. It should be noted that the dashboard is composed of multiple data panels. When creating a data panel, you need to consider what data to display, and then design the query SQL according to this purpose. In addition, in order to achieve better display effects, it is also necessary to consider what form the data is best displayed in, such as numbers, line charts, pie charts, etc.
Summarize
At this point, we have introduced how to use EMQ X + openGemini + Grafana to complete the entire process of IoT data transmission, storage analysis, and data display. You can understand that EMQ Applications in collection. After in-depth learning and mastering other functions of Grafana, users can customize more complete data visualization and even monitoring systems.
The openGemini community looks forward to working with more universities and corporate partners to jointly promote technological innovation
openGemini innovates together to win the future!
references
-
https://www.emqx.com/zh/blog/build-emqx-influxdb-grafana-iot-data-visualization-solution-in-one-hour
-
https://docs.emqx.com/en/enterprise/v5.0/data-integration/data-bridge-influxdb.html#quick-starts
openGemini official website: http://www.openGemini.org
openGemini open source address: https://github.com/openGemini
openGemini public account:
Welcome to pay attention~ We sincerely invite you to join the openGemini community to build, govern and share the future together!