Build your own MQTT server to implement device cloudification (Ubuntu+EMQX)

1. Introduction to EMQX

This article teaches you how to deploy EMQX on an ECS cloud server, build your own private MQTT server, and configure EMQX to implement device cloud migration, device data forwarding, and storage. I use Huawei Cloud's ECS server as the server, and the system selects Ubuntu.

Look here for the Windows version:
https://blog.csdn.net/xiaolong1126626497/article/details/134280836

Insert image description here

1.1 Introduction to EMQX

EMQX is a large-scale elastically scalable cloud-native distributed IoT MQTT message server. Currently EMQX provides: open source version and enterprise version.

As the world's most scalable MQTT message server, EMQX provides efficient and reliable connections to massive IoT devices. It can move and process message and event stream data in real time with high performance, and can quickly build business-critical IoT platforms and applications.

Features of the open source version: Large-scale elastically scalable cloud-native distributed IoT MQTT message server, efficient and reliable connection to massive IoT devices, high-performance real-time processing of messages and events Streaming data helps build business-critical IoT platforms and applications.

Based on APL 2.0 open source license
Full MQTT 3.1.0, 3.1.1 and 5.0 specifications, support for MQTT-SN
Masterless high-availability cluster architecture
High concurrency, low latency, high performance
Extensible gateway and plug-in system

Help documentation: https://www.emqx.io/docs/zh/v5.2/getting-started/getting-started.html

EMQX supports a variety of installation methods, such as containerized deployment, installation and deployment through the EMQX Kubernetes Operator, or deployment on physical servers or virtual machines in the form of installation packages. For installation package deployment forms, the following operating systems are currently supported:

RedHat
CentOS
RockyLinux
AmazonLinux
Ubuntu
Debian
macOS
Linux
Windows

1.2 MQTT core concepts

MQTT (Message Queue Telemetry Transport) is the most commonly used lightweight messaging protocol in the Internet of Things. The protocol is based on a publish/subscribe (pub/sub) model for message communication, allowing devices and applications to exchange data in real time, using simple and effective message formats, minimizing network overhead and reducing power consumption.

As an MQTT messaging platform, EMQX Enterprise provides comprehensive support for complete MQTT messaging functionality.

1.3 Publish/subscribe model

The protocol is event-driven and uses a publish/subscribe pattern to connect devices. Unlike the traditional client/server pattern, it is a messaging pattern in which the sender (publisher) does not directly send messages to specific receivers (subscribers). Instead, publishers categorize messages into different topics, and subscribers subscribe to topics that interest them. When a publisher sends a message to a topic, the MQTT broker routes and filters all incoming messages before delivering them to all subscribers who have expressed interest in the topic.

Publishers and subscribers are decoupled from each other and do not need to know the existence of each other. The only connection between them is based on a predetermined messaging protocol. The publish/subscribe pattern enables flexible message communication as subscribers and publishers can be dynamically added or removed as needed. It also makes it easier to implement message broadcast, multicast, and unicast.

1.4 MQTT server

The MQTT server acts as a broker between publishing and subscribing clients, forwarding all received messages to the matching subscribing client. Therefore, sometimes the server is directly called an MQTT broker.

1.5 MQTT client

A client refers to a device or application that can connect to an MQTT server using the MQTT protocol. They can act as both publishers and subscribers, or they can serve in either role independently. In MQTT communication, a client can publish messages to the server and also subscribe to one or more topics to receive messages from other clients. In this way, the MQTT client can achieve efficient and real-time data transmission and interaction in the IoT environment.

1.6 Topics and Wildcards

In MQTT, topics are used to identify and distinguish different messages, forming the basis of MQTT message routing. The publisher can specify the topic of the message when publishing the message, and the subscriber can choose to subscribe to the topic of interest to receive related messages.

To achieve the goal of subscribing to multiple topics at once, subscribers can use wildcards when subscribing to topics. MQTT provides two types of topic wildcards, single-level wildcards and multi-level wildcards, to meet different subscription needs.

Single-level wildcard: Used to match one level within a topic.
Multi-level wildcard: Used to match multiple levels within a topic.

By using these two wildcards, subscribers have the flexibility to subscribe to a range of related topics rather than just one specific topic.

1.7 Quality of Service (QoS)

MQTT defines three levels of QoS to provide different levels of message reliability. Each message can independently set its own QoS when publishing.

QoS 0: The message is delivered at most once and may be lost;
QoS 1: The message is delivered at least once and is guaranteed to arrive, but may be repeated;
QoS 2: Deliver messages exactly once, ensuring arrival without duplication.

As the QoS level increases, the complexity of message transmission also increases. The appropriate QoS level needs to be selected based on the actual situation.

When choosing a QoS level, you need to weigh the reliability of message delivery and the consumption of network resources. For example, when network resources are limited but message reliability requirements are not high, QoS 0 can be selected; and in scenarios where message reliability requirements are high and a certain amount of network resource consumption can be tolerated, QoS 1 or QoS can be selected 2.

2. Purchase ECS cloud server

2.1 Log in to the official website

https://www.huaweicloud.com/

2.2 Purchase ECS server

【1】Choose ECS elastic server

【2】Select the region, configuration information, and operating system of the ECS server (I chose Ubuntu18.04 64-bit).

【3】Purchase an elastic public IP and configure bandwidth.

【4】Configure password

【5】Choose the purchase duration. I chose 1 month here.

【6】Confirm payment

received an email reminder and the server was successfully created. (In order to write tutorials, I spent 320 yuan and bought a server for one month)

【7】Return to the console of the elastic server

【8】Click on the server name to enter the details page.

2.3 Configure security group

Make sure that several ports commonly used by the MQTT server have been opened.

2.4 Install FinalShell

Install the FinalShell terminal under Windows to facilitate remote login to the cloud server using the SSH protocol. (Of course, the same applies to logging in using other methods)

2.5 Remotely log in to the cloud server terminal

【1】Create a new connection and select SSH connection.

【2】Fill in the IP address, username, and password

The host here is the public IP address of the server, the password is the password entered when creating the server, and the username is root.

【3】Click to connect to the server

[4] When logging in for the first time, a prompt box will pop up, choose to accept and save.

【5】Next, you can see that the server has been logged in successfully.

2. Install EMQX under Linux

This chapter will introduce how to download, install and start EMQX in Ubuntu system.

Supported Ubuntu versions:

Ubuntu 22.04
Ubuntu 20.04
Ubuntu 18.04

2.1 Official website address

Link: https://www.emqx.io/docs/zh/v5.2/deploy/install-ubuntu.html

2.2 Installation through Apt source

EMQX supports installation through the Apt source, which eliminates the need for users to manually handle dependencies and update software packages, etc., and has the advantages of being more convenient, secure and easy to use.

In the command line terminal, copy the following command and press Enter.

【1】Configure the EMQX Apt source through the following command:

curl -s https://assets.emqx.com/scripts/install-emqx-deb.sh | sudo bash

【2】Run the following command to install EMQX:

sudo apt-get install emqx

【3】Run the following command to start EMQX:

sudo systemctl start emqx

The process is as follows:

2.3 Commonly used commands in EMQX

sudo systemctl emqx start    启动
sudo systemctl emqx stop     停止
sudo systemctl emqx restart  重启

3. Configure EMQX server

3.1 Log in to the EMQX built-in management console

EMQX provides a built-in management console, the EMQX Dashboard. It is convenient for users to easily manage and monitor the EMQX cluster through the web page, and configure and use various required functions.

Enter in the browser: http://122.112.225.194:18083 to access the EMQX backend management page. You can manage connected clients or check operational status.

The IP address here is the public IP address of your ECS cloud server.

After opening the browser, the effect of opening after entering the address:

Default username and password:

用户名：admin
密码：public

You will be prompted to change the new password when you log in for the first time. If you do not want to set it, you can choose to skip it (for public network server deployment, it is safer to change the password).

Change the new password below:

The page for successful login is displayed as follows:

3.2 MQTT configuration

Here you can configure some parameters of MQTT and configure them according to your own needs.

3.3 Test MQTT communication

Create a new client and click Connect.

After connecting, then click Subscribe, and Publish, if the following messages can be received normally. It means that the MQTT server communication is normal and there is no problem.

And you can also see the formats of 主题发布 and 主题订阅 on this page.

3.4 MQTT client login server test

Next, open our own MQTT client and log in to the MQTT server to communicate test data.

Port selection: 1883

Fill in the parameters according to the software parameters, log in, and publish and subscribe to the topic.

Note: Client authentication has not been configured yet. As long as the IP and port are entered correctly, you can enter the MQTT triplet at will and log in to the server. The server does not verify the triplet.

Anonymous authentication is enabled in EMQ X's default configuration, and any client can access EMQX. When the authentication plug-in is not enabled or the authentication plug-in does not explicitly allow/deny (ignore) the connection request, EMQX will decide whether to allow the client to connect based on the enablement of anonymous authentication.

Then open the management background of EMQX. You can see that our device has logged into the server with the name test1.

You can also see the topics subscribed by our client devices on the topic subscription page.

3.5 Client authentication configuration

Anonymous authentication is enabled in EMQX's default configuration, and any client can access EMQX. When the authentication plug-in is not enabled or the authentication plug-in does not explicitly allow/deny (ignore) the connection request, EMQX will decide whether to allow the client to connect based on the enablement of anonymous authentication.

Authentication must be enabled in the official product, otherwise any device can access it.

The following describes how to configure client authentication.

【1】Open the client authentication page

【2】Choose password authentication

【3】Choose the built-in database

【4】Set the authentication method (all can be defaulted, no need to change), click Create directly.

【5】After successful creation, click User Management

【6】Add user

【7】Added successfully

【8】After adding it, open the MQTT client to test.

When logging in, the MQTT username and password must be entered correctly and filled in truthfully according to the information added in the previous step, otherwise you will not be able to log in to the server.

3.6 Client authorization configuration

The client authorization page can configure the topic publishing and subscription permissions of each client (device). Restricts whether it can publish topics and subscribe to topics. You can configure it if necessary.

http://127.0.0.1:18083/#/authorization/detail/built_in_database?tab=users

【1】Create data source

【2】Choose the built-in database

【3】Complete creation

【4】Click on permission management

【5】Select the client ID and click Add

【6】Configure permissions

3.7 Data forwarding (integration)

In the integration option, device data can be processed. For example: forward to your own HTTP server, forward to your other MQTT servers, create rules, certain events trigger certain actions, etc.

Select Data Bridge.

The data can be sent to the sender's own HTTP server or to other MQTT servers.

Select HTTP service (if you have an HTTP server, you can forward the data to your own HTTP server).

4. MQTT client message exchange test

4.1 Add 2 devices

In order to facilitate the test devices to subscribe to topics and send and receive data to each other, add at least 2 devices to the client authentication page. I added test1 and test2 respectively here.

4.2 Equipment room testing

Device A subscribes to the topic of device B, and device B subscribes to the topic of device A to realize mutual data transmission.

MQTT information for device A:

MQTT服务器地址：122.112.225.194
MQTT服务器端口号：1883
MQTT客户端ID：AAA
MQTT用户名：test1
MQTT登录密码：12345678

订阅主题：BBB/#
发布主题：AAA/1
发布的消息：{
    
     "msg": "我是AAA设备" }

MQTT information for device B:

MQTT服务器地址：122.112.225.194
MQTT服务器端口号：1883
MQTT客户端ID：BBB
MQTT用户名：test2
MQTT登录密码：12345678

订阅主题：AAA/#
发布主题：BBB/1
发布的消息：{
    
     "msg": "我是BBB设备" }

4.3 MQTT real-time image transmission test

The following demonstration uses the MQTT protocol for image transmission.

This is a packaged APP. You can download it with 0 points. After downloading it, you can test the image transmission:
https://download.csdn.net/download/xiaolong1126626497/88512661< /span>

This is the Qt source code of the MQTT protocol image transmission host computer. If you need the source code, you can download it here:
https://download.csdn.net/download/xiaolong1126626497/87614468

This is a Qt project. The image transmission software developed using Qt is the sender and receiver respectively.

The following demonstration uses the MQTT protocol for image transmission.

This is a Qt project. The image transmission software developed using Qt is the sender and receiver respectively.

Operation effect: The effect of image transmission is very good. The maximum size of a message that can be sent at one time is 256MB

5. MCU equipment moved to the cloud

As long as the MQTT client can communicate with the cloud normally, the same can be said for the microcontroller.

The host computer can also use the MQTT protocol to access the server, subscribe to the device's topic, and receive the device's messages in real time (of course, it can also use the HTTP protocol to access).

6. Data bridging

EMQX supports forwarding data uploaded by the device to other places, such as its own HTTP server. It is convenient for your own server to perform other processing.

Through data bridging, users can send messages from EMQX to an external data system in real time, or pull data from an external data system and send it to a topic in EMQX. EMQX Dashboard provides the ability to visually create data bridges by configuring relevant resources on the page.

This chapter introduces how to build your own HTTP server. Configure EMQX to forward data to your own HTTP server and save the processed data.

6.1 Build HTTP server

I directly use python to write code to build an HTTP server. python3 is not installed by default on the ECS server and needs to be installed first.

【1】Install python3

root@emqx:~/emqx# apt install python3 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3 is already the newest version (3.6.7-1~18.04).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

【2】Write code

from flask import Flask, json, request

app = Flask(__name__)

@app.route('/', methods=['POST'])
def print_messages():
  reply= {
    
    "result": "ok", "message": "success"}
  print("got post request: ", request.get_data())
  return json.dumps(reply), 200

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8000)

Save the above code into a file named server.py.

This code creates a web server using the Flask framework that can receive POST requests from the root path. When receiving a POST request, the server prints the requested data to the terminal and returns a JSON-formatted response to the client. The server will run locally and listen on the default 8000 port.

【3】Run the program

# 安装 flask 依赖
pip install flask
pip3 install flask
    
# 启动服务
python3 server.py

Execute python3 server.py in the command line to start a simple HTTP server that can receive and process POST requests. When a POST request occurs, the server will return the received POST data. You can further expand the logic for processing POST requests according to your own needs.

**Running example: ** (The code can be written locally and uploaded to the server, or you can directly vim server.py open the editor and write it directly)

You can test this server by sending a POST request to http://your-server-ip:8000/.

for example:

http://122.112.225.194:8000/

6.2 Data forwarding configuration

【1】In the integration option, you can process device data and forward the data to your own HTTP server.

【2】Selection Webhook.

Webhook, use Webhook to forward data to HTTP service;

Using Webhook is actually to send the data received and processed by EMQX to an HTTP service, and then process and integrate the business data according to the preset HTTP service.

Similarly, users need to have a pre-built HTTP service. They need to fill in the service address of the HTTP request on the configuration information page, select a request method POST, GET, PUT or DELETE, configure the request header, and fill in the data to be sent using template syntax. Just put it in the request body.