MQTT2 - How MQTT works

Introduce the message model of the MQTT protocol, message transmission process, message publishing and subscription.

1. Introduce the message model of the MQTT protocol

The message model of the MQTT protocol is called the "topic" model. In this model, messages received by the server are categorized by topics. Clients can receive desired messages by subscribing to one or more topics.

1. Message topic in MQTT protocol

When publishing an MQTT message, the message is divided into a topic and a message payload. Subject is used to identify the category of the message and can be any string, but usually a slash-separated hierarchy. For example, a topic named "home/bedroom/temperature" represents a temperature value, while "home/kitchen/lights" represents the state of a light. Using topics, subscribers can choose to receive topics of interest to them.

The topic model of MQTT is a flexible and powerful mechanism that can be adapted to various application scenarios. The number of topics can be very large, and each topic can have multiple subscribers. Subscribers can use wildcards to subscribe to multiple topics in order to receive a wider range of messages. For example, using the wildcard "home/+/temperature", a subscriber could receive temperature values ​​from all rooms.

In MQTT, the topic is specified by the client, and there is no mandatory naming rule. Therefore, when designing a theme, you should choose a structure that is easy to understand and manage, and avoid complex or ambiguous naming. At the same time, the length of the topic name also needs to be properly considered, too long topic names may cause performance problems.

2. Message content format in the MQTT protocol

In the MQTT protocol, message content consists of two parts: message header and message body. The message header contains some control information for specifying the type of message, quality of service, etc., while the message body contains the actual message content.

The most important in the message header is the quality of service (QoS) level, which is used to specify the reliability and transmission quality of the message. The MQTT protocol supports three different QoS levels: QoS 0, QoS 1 and QoS 2. At the QoS 0 level, the message is sent once, if the sending fails, it will be discarded, and the sending will not be retried. At the QoS 1 level, the message will be sent at least once to ensure that the message is received by the receiver. If the send fails, retries are made until the receiver receives the message. Under the QoS 2 level, the message is guaranteed to be sent only once, and it is guaranteed that the message will not be sent repeatedly.

In addition, the message header also includes some other control information, such as message identifier (Message Identifier). A message identifier is a 16-bit integer used to uniquely identify a sent message. When a message has a quality of service rating of 1 or 2, the message identifier is important and is used to track the status of the message.

The message body part is the actual message content. The format of the message body can be arbitrary, but must be encoded and decoded according to the requirements of the application. In practical applications, the message body is usually a piece of data in the format of JSON, XML or binary data.

3. Message publishing and subscription mechanism in MQTT protocol

In the MQTT protocol, the publication and subscription of messages is based on the topic model. Clients receive required messages by subscribing to one or more topics, while publishers send messages by publishing messages to specific topics.

In the process of publishing and subscribing messages, a persistent connection is established between the client and the server. The client can send a subscription request to the server, and the server will add the client to the subscription list according to the request. Once a new message is published to the subscribed topic, the server will push the message to all clients subscribed to the topic.

In the MQTT protocol, there are also some special topics, such as topics starting with $SYS, which are usually used to obtain information about the server status. Additionally, there are wildcards that can be used to subscribe to subtopics of a particular topic.

In short, the publish and subscribe mechanism in the MQTT protocol enables clients to easily obtain the messages they are interested in, and also reduces the consumption of network bandwidth and resources.

2. The message transmission process of the MQTT protocol

In the MQTT protocol, the client needs to connect with the server. Once the connection is established, the client can send messages to the server or subscribe to topics. After the server receives the message, it will classify it according to the message topic, and notify the clients who have subscribed to the topic through the corresponding topic.

2.1 Establish a connection

1. The client sends a CONNECT message to the server, including the protocol version number, client identifier, clear session flag, will flag, will subject, will message, user name and password and other information.

2. The server sends a CONNACK message to the client, including information such as connection confirmation flags and session flags. If the connection fails, the server sends the client the reason for the connection rejection.

3. After the connection is established, the client can send a subscription message to the server to subscribe to one or more topics. After receiving the subscription message, the server adds the corresponding topic to the subscription list and sends a SUBACK message to the client.

4. After receiving the published message, the server will send the message to the client subscribed to the topic according to the message topic. If a client subscribes to multiple topics, the server can send messages of all topics to the client at one time through a PUBLISH message.

5. The client can send a PINGREQ message to keep the connection. After receiving the PINGREQ message, the server will return a PINGRESP message to indicate that it is still alive.

6. The client can actively disconnect by sending a DISCONNECT message, or it can wait for the server to timeout and close the connection.

2.2 Publishing a message

In the MQTT protocol, the client that publishes a message is called a "Publisher", and the client that receives a message is called a "Subscriber". Publishers can publish messages to one or more topics, and the server will send these messages to all subscribers who have subscribed to the corresponding topics. The message publishing process of the MQTT protocol is as follows:

1. The publisher sends a PUBLISH message to the server, which contains a subject and message content.
2. The server stores messages in message queues according to topics.
3. The server sends a message to all subscribers that have subscribed to the topic.
4. After the subscriber receives the message, it can perform corresponding processing operations.

It should be noted that the message publishing in the MQTT protocol is asynchronous, that is, the publisher does not need to wait for the server to send messages to all subscribers, but can return immediately and perform subsequent operations.

In addition, the MQTT protocol also provides some special message publishing methods, such as "Retained Messages" and "Last Will and Testament". Retaining messages means that the server will store the latest messages, and when a new client subscribes to the topic, the server will send the latest messages to the client. A will message is a message sent when a client disconnects to inform other clients that the client is offline.

2.3 Receive messages

When a client subscribes to a topic, it receives any messages for that topic. When a client receives a message, it can choose to take appropriate action. For example, a client can store the message locally, display it on a user interface, or transmit it to another device.

In the MQTT protocol, the client receiving the message must use a callback function to process the message. When a message arrives, the MQTT client will call this callback function and pass the message to it. Applications can implement this callback function themselves to perform desired operations.

The client can also control the reliability of message delivery through the QoS mechanism. Specifically, QoS is divided into three levels: 0, 1, and 2, among which level 0 is the fastest delivery, but the lowest reliability, and level 2 is the slowest delivery, but the highest reliability. When subscribing to a topic, clients can specify the desired QoS level. If a publisher publishes a message at a lower level than a subscriber requires, the subscriber will not receive the message.

2.4 Disconnect

In the MQTT protocol, disconnection is an important operation, which can release the resources of the server and the client, and allow other clients to connect to the server. The following are the steps to disconnect in the MQTT protocol:

1. Client sends DISCONNECT message to server
2. After the server receives the message, it disconnects from the client and releases resources
3. After the client receives the disconnection message from the server, it disconnects the local network connection

In the MQTT protocol, since the connection between the client and the server is realized through a TCP/IP connection, the client and the server can disconnect at any time. In general, the client will actively disconnect after it completes the task, and the server will manage the connection according to its own policy.

3. Message publishing and subscription in the MQTT protocol

Message publishing and subscription is one of the core concepts of the MQTT protocol. In MQTT, message publishers are called "publishers" and message receivers are called "subscribers".

In MQTT, message transmission is asynchronous, which means that after the publisher publishes the message, it does not need to wait for the subscriber to receive the message. When a subscriber goes online, the MQTT server will push all unreceived messages to the subscriber.

Through this publish-subscribe model, the MQTT protocol can effectively realize the communication between IoT devices, making the communication between devices more flexible and reliable.

3.1 News release

Publishers publish messages to a specific topic (Topic), and subscribers can subscribe to one or more topics to receive messages related to the topic. When a new message is published to a topic to which a subscriber has subscribed, the subscriber receives the message.

In MQTT, topics are organized by one or more hierarchies separated by "/". For example, "/home/bedroom/light" is a theme, where "home" is a hierarchy with a "bedroom" hierarchy below it, and a "light" hierarchy below it.

3.2 Message Subscription

Subscribers can use wildcards to subscribe to multiple topics. MQTT supports two types of wildcards: single-level wildcard "+" and multi-level wildcard "#". Use the single-level wildcard "+" to subscribe to all topics at a level, and use the multi-level wildcard "#" to subscribe to all topics at a level and all sub-levels under this level.

When a publisher publishes a message, it must specify a topic. After the message is delivered to the MQTT server, the server forwards the message to all subscribers who have subscribed to the same or matching topic.

3.3 Subscription filter

When a client subscribes to a topic, wildcards can be used to subscribe to one or more topics. MQTT defines two kinds of wildcards: "+" and "#".

The "+" wildcard means to match any character at a level. For example, subscribing to "home/+/temperature" will match topics such as "home/livingroom/temperature", "home/kitchen/temperature", etc.

The "#" wildcard means to match any character at one or more levels. For example, subscribing to "home/#" will match "home/temperature", "home/livingroom/temperature", "home/kitchen/lights", etc. all topics starting with "home/".

Note that in actual use, the use of subscription filters should be reduced as much as possible to avoid pressure on network bandwidth and processing power caused by too many subscribed topics.

4. Several characteristics of MQTT message transmission

1. Three QoS levels in the MQTT protocol

The MQTT protocol supports three Quality of Service (QoS) levels, which are used to control the reliability of message delivery. The three QoS classes are:

• QoS 0: At most one transmission, no guarantee that the message will reach the receiving end.
• QoS 1: Transmit at least once to ensure that the message reaches the receiving end, but there may be duplicate messages.
• QoS 2: Ensure that the message arrives at the receiving end and only once, but requires more network resources and time.

In the MQTT protocol, the publisher can set the QoS level of the message, and the subscriber can choose the QoS level of the subscribed topic.

The higher the QoS level, the higher the reliability of message delivery, but requires more network resources and time. Selecting an appropriate QoS level should be determined according to actual application requirements to ensure a balance between transmission efficiency and reliability.

2. Message confirmation mechanism in MQTT protocol

The message confirmation mechanism in the MQTT protocol refers to a series of measures taken to ensure the reliability and correctness of the message during the process of sending and receiving the message. The MQTT protocol supports three different QoS levels, each with a different message confirmation mechanism.

• QoS0: At most one transmission, no message confirmation is required, and the message is unreliable.
• QoS1: At least one transmission, message confirmation required, message may be repeated.
• QoS2: Exactly one transmission, message confirmation is required, and the message will not be repeated.

Under QoS1 and QoS2 levels, message publishers will receive confirmation messages from message receivers to ensure that messages are received correctly. When the message publisher receives the confirmation message, it means that the message has been received. If the message publisher does not receive the confirmation message, it needs to resend the message.

Under the QoS2 level, message retransmission is also required, that is, if the publisher does not receive the confirmation message, it needs to resend the message until the confirmation message is received.

During the message confirmation process, a timeout judgment is required. If the confirmation message is not received after the timeout, the message needs to be resent.

In addition to the QoS level, the MQTT protocol also supports will messages, which can send a predefined message when the client disconnects to notify other clients that the client is offline. This is also a message acknowledgment mechanism to ensure that other clients can correctly handle the client offline situation.

3. Will message in MQTT protocol

When the client connection is disconnected, there may be some remaining messages that need to be processed, and at this time, the will message in the MQTT protocol needs to be used. Will messages can be understood as the last will that needs to be performed by the server on behalf of the client after the client disconnects unexpectedly.

The will message needs to be set when the connection is established. When the client connection is abnormally disconnected, the server will publish the will message to the previously subscribed topic. When publishing a will message, the server will check its QoS level and the Retain flag. If the Retain flag is set, the will message will be kept on the server until another client subscribes to the corresponding topic and reads the will message.

The use of will messages can provide useful support in many situations, such as:

• When a client disconnects, send an offline notification to other subscribers
• Send timeout notifications to other subscribers when a client does not respond for a long time
• When the client disconnects abnormally, send an abnormal disconnection notification to other subscribers

When setting a will message, you need to specify its subject, payload, QoS level, and Retain flag. This information will be saved in the connection agreement in case of disconnection.

5. Comparison of MQTT protocol and other communication protocols

5.1 Comparison between MQTT protocol and HTTP protocol

Both the MQTT protocol and the HTTP protocol are commonly used Internet communication protocols, and they have different characteristics in many aspects. Here are some comparisons between the MQTT protocol and the HTTP protocol:

1. Transmission method: The HTTP protocol adopts the request-response mode, that is, the client sends a request to the server, and the server returns a response. Both the request and the response require additional network transmission and processing time. However, the MQTT protocol adopts the publish-subscribe mode. It only needs to establish a connection between the client and the server, and then the communication can be realized by subscribing and publishing messages, and the communication efficiency is higher.

2. Message size: The HTTP protocol usually needs to carry a large amount of HTTP header information and cookies and other data during transmission, so the message size is usually large. The message format in the MQTT protocol is more compact, the message header information is less, and the amount of transmitted data is smaller, which is especially suitable for use in low-bandwidth and unstable networks.

3. Connection status: In the HTTP protocol, each request requires a connection between the client and the server, and the connection is disconnected after the request ends. Such a connection process requires a lot of network resources, which is a burden for the server. In the MQTT protocol, the client only needs to maintain the connection state after the connection is established, and the subsequent message transmission is realized through this connection. It does not need to establish a connection every time, which saves server resources.

4. QoS level: The MQTT protocol supports three different quality of service (QoS) levels, and the appropriate level can be selected according to application requirements. The HTTP protocol does not provide a variety of different QoS levels like the MQTT protocol.

5. Message transmission method: The messages transmitted by the HTTP protocol are all transmitted in plain text, which is easy to be stolen by attackers. The MQTT protocol supports encrypted transmission, and the TLS protocol can be used to encrypt messages, which improves data security.

Generally speaking, the MQTT protocol is suitable for scenarios with high real-time transmission requirements, small data volume and security requirements, while the HTTP protocol is suitable for transmitting some large data and is suitable for the interaction between the client and the server.

5.2 Comparison between MQTT protocol and CoAP protocol

Both the MQTT protocol and the CoAP protocol are used for communication protocols between IoT devices, and there are some commonalities and differences between them.

common ground:

1. Both are lightweight protocols: Both the MQTT protocol and the CoAP protocol are designed to solve the communication between IoT devices, so they are both lightweight protocols with a small code base and low bandwidth occupation.
2. Both are based on the publish/subscribe model: Both the MQTT protocol and the CoAP protocol are based on the publish/subscribe model. They both support the publisher to publish data to the topic, and the subscriber can subscribe to the relevant topic and receive the data published by the publisher.

the difference:

1. Different application scenarios: The MQTT protocol is mainly used in scenarios that require low bandwidth and unstable networks, such as sensor and telemetry applications. The CoAP protocol is mainly used in scenarios that require low power consumption and low bandwidth, such as smart home and IoT devices.
2. The transport protocols are different: the MQTT protocol uses the TCP protocol as the transport layer protocol, while the CoAP protocol uses the UDP protocol as the transport layer protocol. Therefore, the MQTT protocol can guarantee the reliability and sequence of messages, while the CoAP protocol is more suitable for low-bandwidth and wireless network scenarios.
3. Different security: The MQTT protocol provides security mechanisms at both the transport layer and the application layer, such as the TLS/SSL protocol and authentication mechanisms based on user names and passwords. The CoAP protocol does not provide a security mechanism at the transport layer or the application layer, and needs to add a security mechanism at the application layer.

The above is the comparison between the MQTT protocol and the CoAP protocol. It can be seen that the MQTT protocol is suitable for scenarios that require high transmission quality and need to ensure data integrity, while the CoAP protocol is suitable for scenarios that do not require high transmission quality and require low power consumption. bandwidth scenarios.

Six, focus

The MQTT protocol is a message transmission protocol based on the publish/subscribe model. Its message model is called the "topic" model. Messages are classified by topics, and clients can receive the required messages by subscribing to topics.

In the MQTT protocol, the client needs to connect with the server. After the connection is established, the client can publish messages to the server or subscribe to topics. After the server receives the messages, it will classify them according to the topics, and notify the clients who have subscribed to the topics through the corresponding topics.

The MQTT protocol supports three different quality of service (QoS) levels, namely at most once (QoS0), at least once (QoS1) and only once (QoS2). Through the message confirmation mechanism, the reliability of the message can be guaranteed, and the will message can be set to provide additional security.

The MQTT protocol, the HTTP protocol and the CoAP protocol are all communication protocols between Internet applications. Compared with the HTTP protocol, the MQTT protocol has higher real-time performance and smaller data packet headers, and is more suitable for use in environments with limited network bandwidth. Compared with the CoAP protocol, the MQTT protocol has higher reliability and more functional scalability, and is more suitable for the connection and data transmission of large-scale IoT devices.