Why did IoT (Internet of Things) choose the MQTT protocol?

[Live broadcast preview] 99 yuan cloud virtual machine × 9 yuan cloud native architecture √

To communicate with each other, IoT devices require a set of standard communication protocols. MQTT (Message Queuing Telemetry Transport) is a set of standard message queue communication protocols specially designed for IoT devices. IoT devices using the MQTT protocol can connect to any message queue that supports the MQTT protocol for communication.

At a macro level, MQTT is similar to other MQ transport protocols. It is also a "publish-subscribe" messaging model
The network structure is also a C/S architecture. The IoT device is the client and the Broker is the server. The client communicates with the Broker to send and receive messages.

But after all, the usage scenarios are different, so there are many differences between MQTT and ordinary MQ.

1 Clients are all running on IoT devices

1.1 IoT device characteristics

① Cheap

The biggest feature is that a water cup only costs a few dozen yuan. The smart module on it costs at most ten yuan. No matter how expensive it is, it will not be sold. The memory of a ten-dollar smart device is calculated in KB. It may not have a CPU or an os. The entire device is just a SoC (System on a Chip). Such a device requires that the communication protocol should not be complex and not have too many functions.

② Wireless connection

IoT devices generally use wireless connections, and many devices are frequently moved, resulting in unstable network connections for IoT devices, which is the norm.

The MQTT protocol is designed with these characteristics in mind. The message design of the protocol is minimalist, and every word is saved as preciously as gold. The protocol function is also very simple, basically only:

Publish and subscribe topics
Send and receive messages

These two core functions. To deal with the problem of unstable network connections, MQTT adds mechanisms:

The heartbeat mechanism allows both the client and the server to keep track of the current connection status at any time. Once the connection is interrupted, they can reconnect as soon as possible.
The session mechanism saves the session state on the server side. After the client reconnects, it can resume the previous session and continue to send and receive messages. In this way, the complexity is transferred to the server, and the client implementation is simpler.

2 High requirements on the server side

In the usage scenarios faced by MQTT, the server needs to support a large number of IoT devices online at the same time.

In an ordinary message queue cluster, the service clients all run on powerful servers, so the number of clients is not particularly large. For example, JD.com’s JMQ cluster has about 100,000 daily online clients, which is enough to support people across the country to buy on JD.com.

In MQTT usage scenarios, the number of clients that need to be supported is far more than tens of thousands or hundreds of thousands. For example, if the Beijing Transportation Commission wants to connect all vehicles in the city, it will require millions of clients. The cameras on the roadside, the TVs and refrigerators in every household, and the various wearable devices that everyone carries with them all range in size from millions to tens of millions or even hundreds of millions.

3 Does not support point-to-point communication

The design goal of the MQTT protocol is to support a publish-subscribe (Publish-Subscribe) model rather than point-to-point communication.

One of the main features of MQTT is that it supports the publisher (Publisher) to publish messages to a topic (Topic), and the subscriber (Subscriber) can receive these messages by subscribing to relevant topics. This model has good scalability and flexibility in large-scale distributed systems. Therefore, MQTT is more suitable for many-to-many and many-to-one communication scenarios, such as Internet of Things (IoT) applications, message middleware, etc.

Although the design goal of MQTT is not point-to-point communication, in actual use, you can still simulate point-to-point communication through some designs. For example, use different topics to simulate point-to-point communication, or perform some additional protocols and logic at the application layer to achieve the effect of point-to-point communication.

The general approach is that each client creates a topic named after its own ID, and then the client subscribes to its own exclusive topic to receive messages specifically sent to this client. That is, in an MQTT cluster, the number of topics and the number of clients are basically of the same order of magnitude.

4 MQTT product selection

How to support a large number of online IoT devices and a large number of topics is the biggest challenge faced by every MQ that supports the MQTT protocol. When selecting MQTT server technology, you need to focus on technical points.

Open source MQTT products

Some are traditional MQ, which implement MQTT protocol support through official or unofficial extensions. There are also some specialized MQTT Server products. Most of these MQTT Servers have no problems at the protocol support level and meet the requirements in terms of performance and stability. But we haven’t found any open source products that can well support a large number of clients and themes. why?

Traditional MQ

Although it can be extended to support the MQTT protocol, at the beginning of the overall architecture design, support for a large number of clients and topics was not considered. For example, RocketMQ metadata is stored in NameServer's memory, and Kafka is stored in zk. These storages are not good at saving large amounts of data, so they cannot support too many clients and topics.

Some other open source MQTT Servers

Many have no clustering function or the clustering function is incomplete. Most products with good clustering functions put the clustering function in the enterprise version and sell it.

Therefore, when selecting MQTT Server technology, if the number of IoT devices you connect is less than 100,000, you can choose an open source product. The selection principle is the same as selecting an ordinary message queue. Give priority to a popular and familiar open source product.

If the client scale exceeds 100,000, and it needs to support such a large number of clients, a single node on the server side is definitely not enough. A cluster must be used, and the cluster must support horizontal expansion. At this time, there are almost no open source products. At this time, it can only be recommended to choose the MQTT cloud service provided by some cloud platform vendors, which is relatively low-priced. You can also choose the higher-priced commercial version of MQTT Server.

Another option is to build your own MQTT cluster based on the existing open source MQTT Server through some integration and development.

5 Build an MQTT cluster that supports a large number of clients

How does the MQTT cluster support massive online IoT devices? Generally speaking, the architecture of an MQTT cluster should be like this:

Looking from left to right, the first accessed address is preferably a domain name, so that multiple IP addresses can be configured behind the domain name for load balancing. Of course, this domain name is not required. You can also connect directly to the load balancer. For load balancing, you can choose dedicated load balancing hardware like F5, or software like Nginx, as long as it is a four-layer or seven-layer load balancing device that supports the MQTT protocol.

A Proxy cluster is deployed behind the load balancer.

Proxy cluster function

Undertake massive IoT device connections
Maintain session with client
As a proxy, forward messages between the client and the Broker

After the Proxy cluster is the Broker cluster, which is responsible for saving and sending and receiving messages.

Some MQTT Server cluster architectures:

There is no proxy in the architecture. In fact, the Proxy and Broker functions are just integrated into one process, and there is not much difference between the two architectures. It can be considered as the same structure for analysis.

Front-end Proxy can easily solve the problem of massive connections. Since Proxy can be expanded horizontally, as long as enough Proxy nodes are used, it can withstand simultaneous connections of massive clients. Each Proxy and each Broker only need one connection to communicate. For each Broker, the maximum number of connections will not exceed the number of Proxy nodes.

Proxy can learn from Tomcat's two ways of handling sessions for session processing:

The session is saved locally in the Proxy, and each Proxy node only maintains the sessions of these clients connected to itself. But this needs to be used in conjunction with load balancing. The load balancing device needs to support sticky sessions to ensure that connections of the same session are always forwarded to the same Proxy node.
Save the session in an external storage cluster, such as Redis cluster or MySQL cluster. In this way, Proxy can be designed to be completely stateless, with no special requirements for load balancing equipment. But this requires the external storage cluster to have the ability to store tens of millions of data and have good performance.

How to support massive themes?

A more feasible solution is to deploy multiple groups of Broker small clusters on the back end of the Proxy cluster. For example, it can be multiple groups of Kafka small clusters. Each small cluster is only responsible for storing a part of the topic. In this way, for each Broker small cluster, the number of topics can be controlled within an acceptable range. Since messages are forwarded through Proxy, some sharding algorithms such as consistent hashing can be used in Proxy to find the corresponding Broker cluster based on the topic name. This solves the problem of supporting a large number of themes.

UML

UML diagram of Proxy:

@startuml
package "MQTT Proxy Cluster" {
    class MQTTProxy {
        +handleIncomingMessage()
        +handleOutgoingMessage()
        +produceMessage()
        +consumeMessage()
    }

    class Client {
        +sendMessage()
        +receiveMessage()
    }

    class Broker {
        +publish()
        +subscribe()
    }

    Client --> MQTTProxy
    MQTTProxy --> Broker
}
@enduml

@startuml
actor Client
entity MQTTProxy
entity Broker

Client -> MQTTProxy : sendMessage()
activate MQTTProxy
MQTTProxy -> Broker : produceMessage()
deactivate MQTTProxy
@enduml

@startuml
entity MQTTProxy
entity Broker
actor Client

Broker -> MQTTProxy : publishMessage()
activate MQTTProxy
MQTTProxy -> Client : consumeMessage()
deactivate MQTTProxy
@enduml

Sequence diagram of Proxy sending and receiving messages:

Sequence diagram of Proxy production message process:

Sequence diagram of Proxy consumption message process:

6 Summary

MQTT is a set of standard communication protocols specially designed for IoT devices. This protocol is similar to the ordinary message queue protocol in terms of message model and function. The biggest difference lies in the different application scenarios. In IoT application scenarios, IoT devices have poor performance and unstable network connections. The main challenge facing the server is the need to support a large number of clients and themes.

Existing open source MQTT products have good support for the protocol, and can be selected when the number of clients is less than 100,000. For scenarios with a large number of clients, the server must be supported by a cluster, and you can choose paid cloud services and enterprise edition products. You can also choose to build an MQTT cluster yourself.

The most critical technical point of building a cluster by yourself is to solve massive connections, session management and massive topics through the front-end Proxy cluster:

The front-end Proxy is responsible for forwarding messages between the Broker and the client. In this way, the massive client connections are converged into a small number of connections between the Proxy and the Broker, which solves the problem of massive client connections.
The implementation principle of maintaining sessions is the same as Tomcat maintaining HTTP sessions.
Massive topics can be solved by deploying multiple groups of Broker small clusters on the backend, and each small cluster shares a part of the topic.

reference:

ChatGPT
MQTT protocol: How to support massive online IoT devices?
EMQX MQTT Tutorial
Baidu open source MQTT server (supports clustering) https://bifromq.io/zh-Hans/

This article is published by OpenWrite, a blog that publishes multiple articles !