The MQTT protocol can be said to be the most widely used IoT application layer protocol at present. MQTT solves one of the most basic problems in the Internet of Things, that is, the communication between devices and devices, and between devices and cloud services. The main application scenario is to provide communication guarantee for a large number of low-power, unreliable Internet of Things devices. This article mainly explains some questions for beginners. For example, do you know why QoS2 messages can guarantee that the messages will not be received repeatedly? Why is it to provide reliable communication guarantee? More protocol details are not shown, you can refer to MQTT Chinese website .

2. Basic features of MQTT protocol:

Simple to implement
Provide QoS for data transmission
Lightweight, low bandwidth
Any type of data can be transmitted
Session that can be maintained (Session)

Three, MQTT protocol features

Application layer protocol based on TCP long connection;
Based on Client/Server architecture;
Use the subscription/publishing model to decouple the sender and receiver of the message;
Provides 3 types of QoS (Quality of Service) messages: at most once, at least once, and only once;
Sending and receiving messages are asynchronous, and the sender does not need to wait for the receiver to respond;

Four, MQTT protocol communication model

4.1 Client/Server architecture

The complete MQTT protocol includes two parts, MQTT Client and MQTT Server, among which MQTT Server is also Broker;

4.1.1 MQTT Client

As long as the device is connected to the MQTT Broker based on the MQTT protocol, the device is considered to be an MQTT Client. The MQTT Client can act as a publisher and a subscriber alone, or as a publisher and a subscriber at the same time.

4.1.2 MQTT Broker

MQTT Broker is the core of the MQTT protocol. Its main function is to receive messages from publishers and then forward them to corresponding subscribers. Broker can authorize Clinet access and control the rights of Client. The commonly used open source library of MQTT Broker written in C language is Mosquitto. For more information, please check https://github.com/mqtt/mqtt.github.io/wiki/servers . If you need a Broker environment to learn MQTT, you can build it yourself through these open source libraries, or you can use the Broker services provided by major cloud platforms, such as Alibaba Cloud and Tencent Cloud.

4.2 Publish/Subscribe Model

MQTT is implemented through the publish/subscribe model. Since the publisher and the subscriber are not directly connected, an intermediary is required to publish and store messages. We call this intermediary the Broker, which is simply the server implementation based on the mqtt protocol. (There are many open source implementations on github), the publisher and subscriber connected to Broker act as clients;

Both the publisher and the subscriber have established a TCP connection to the Broker;
The subscriber informs the Broker of the topic it wants to subscribe to;
The publisher sends the message to the Broker and specifies the topic of the message (Topic);
After the Broker receives the message, it detects which subscribers have subscribed to the corresponding Topic, and then sends the message to the subscriber;
The subscriber obtains the message from the Broker;
If a subscriber is offline, the Broker can save the corresponding message first, and when the subscriber connects to the Broker next time, the previous message will be sent to the subscriber;

Five, MQTT protocol format

The MQTT protocol uses a binary data packet, which consists of three parts: fixed header, variable header, and message body;

Fixed header: exists in all MQTT data packets, the length is 2~5 bytes, including 3 parts, the data packet type (4bit), the identification bit (4bit), the remaining length of the data packet (1~4Byte), specific Meaning refer to MQTT protocol
Variable header: part of the MQTT packet contains
Message body: part of the MQTT packet contains

In the MQTT protocol, the remaining length of the data packet in the fixed header includes the variable header and the length of the message body. The remaining length can be represented by 1~4Byte, and the maximum 4Byte can represent 256MB (0xFFFFFF7F). Some data packets do not have variable headers and message bodies. For example, the PINGREQ data packet has only 2 bytes, so the maximum and minimum values of the MQTT data packet can be calculated.

Maximum: 256MB+5Byte, where 256MB is the maximum remaining length, including variable header and message body; 5Byte is fixed header, data packet type (4bit), identification bit (4bit), remaining length of data packet (4Byte);
Minimum: 2Byte, such as PINGREQ data packet, only fixed header, no variable header, message body;

Six, MQTT protocol implementation principle

6.1 Establishing a connection

Before the Client can publish and subscribe to messages, it must first connect to the Broker. This connection is not only to establish a TCP connection, but also to establish a connection at the application layer. The MQTT connection establishment process is as follows:

Client sends CONNECT packet to Broker;
After the Broker receives the CONNECT packet, if access is allowed, it will reply with a CONNACK packet with a return code of 0. If it refuses to access, it will reply with a CONNACK packet with a return code other than 0. The return code indicates the reason for the failure;

6.2 Close connection

The closing of the connection can be initiated by either the Client or the Broker. There is also a third case that will close the connection, that is, the heartbeat timeout, and different closing procedures are different:

Client closes the connection: Client sends a DISCONNECT packet to the Broker, without waiting for the Broker to reply the DISCONNECT packet, the Broker will not reply, and can close the underlying TCP connection after sending it; the reason why it has to send before disconnecting the TCP connection has no interaction with the Broker The data packet is to allow Broker to recognize a normal disconnection, Broker will discard the last wish message of the current connection, otherwise it will be recognized as an abnormal connection, and the currently connected hospital message will not be lost;
Broker closes the connection: no need to send any MQTT packets, directly close the underlying TCP connection
Keepalive timeout: The MQTT protocol stipulates that the Broker should keep the connection before it receives the DISCONNECT packet from the Client. However, if the Broker does not receive any data packets from the Client during the keepalive interval, it will actively close the connection. The value of this keepalive is specified by the data packet type when the Client initiates a connection;

6.3 Publish and Subscribe

The concept of publish and subscribe must be well understood by everyone. For the specific interaction protocol data package, you can view the MQTT protocol and introduce several key features:

Support for receiving offline messages: MQTT supports subscribers accepting offline messages, but it does not mean that messages can be received offline, meaning that if the publisher publishes a message to the Broker when the subscriber is offline, subscribes to the corresponding Topic subscription After connecting to the Broker again, the publisher will receive the message published by the publisher during its offline period. The requirement for receiving offline messages is that the Client needs to use a persistent session, and the QoS when publishing is not less than 1;
Topic wildcards: wildcards can be used for the message topics of publish and subscribe. The main function of wildcards is to specify the level of the message topic name. There is a special case. The wildcard does not work when unsubscribing. The unsubscribed topic name must match every word in the subscription. The subject name specified at the time is the same;

6.4 Qos

The original intention of the MQTT protocol design is to provide a set of mechanisms to ensure the stable transmission of messages for data transmission in an environment with narrow network bandwidth and unstable signals. This set of mechanisms includes message response, storage and retransmission. Under this mechanism, three different levels of QoS (Quality of Service) are provided. QoS is an agreement reached between Sender and Receiver, not an agreement reached between Publisher and Subscriber. That is, the QoS between Publisher and Broker, and the QoS between Subscriber and Broker, there is no direct relationship between the two.

QoS0: at most once
QoS1: at least once
QoS2: Ensure only once

The above are three QoS service levels. Many beginners may not understand the implementation principle at the early stage of learning MQTT, especially how to ensure that QoS2 is only sent once. In fact, it is well understood from the analysis of the MQTT protocol itself;

6.4.1 QoS0

Qos0 is the simplest message quality service level. Sender sends a PUBLISH packet containing message data to Receiver, and then discards the sent PUBLISH packet directly regardless of the result, so Receiver receives at most once.

6.4.2 QoS1

In order to ensure that Receiver receives the message at least once, QoS1 adds a response mechanism; Sender sends a PUBLISH data packet with message data to Receiver and saves this data packet locally. Receiver will return a PUBACK data packet after receiving it, PUBACK The variable header of the data packet has the same packet identifier as the PUBLISH data packet. After receiving the PUBACK, the Sender finds the locally saved data packet according to the packet identifier (Packet Identifier) and discards it, and the transmission is completed.

If the Sender does not receive the PUBACK packet within a certain period of time after sending the PUBLISH packet, it will set the DUP flag in the PUBLISH packet to 1 (indicating that the packet is retransmitted) and resend it, repeat the above steps until it receives the PUBACK, so if Because of the network delay and other reasons, the Receiver does not receive the PUBLISH packet or the PUBACK is not sent within the timeout period, which will cause the Receiver to receive multiple times.

6.4.3 QoS2

QoS2 must not only ensure that Receiver can receive the data packet sent by Sender, but also cannot be repeated, that is, it can only be received once, so the retransmission and response mechanism is more complicated. Among the three service levels, QoS2 is the most expensive and the slowest. Yes, of course it is also the safest.

QoS2 uses 2 sets of request/response procedures, that is, a 4-segment handshake to ensure that Receiver only receives a message sent by Sender once.

1) Sender sends a PUBLISH packet with a QoS value of 2, assuming that the packet contains a Packet Identifier as P, and saves the PUBLISH packet locally;

2) After Receiver receives the PUBLISH packet, it saves the Packet Identifier of the PUBLISH packet as P locally, and replies to the Sender with a PUBREC packet. PUBREC contains the Packet Identifier as P, but there is no message body;

3) After receiving the PUBREC, the Sender can discard the PUBLISH packet with the Packet Identifier of P, save the PUBREC packet locally, and reply to the Receiver with a PUBREL packet. The Packet Identifier in the PUBREL packet is P and there is no message body; if If the PUBREC packet is not received within the Sender timeout period, the DUP flag in PUBLISH will be set to 1, and the PUBLISH packet will be sent again;

4) When Receiver receives a PUBREL packet, it can discard the Packet Identifier P stored locally, and reply to the Sender with a PUBCOMP packet. The Packet Identifier in the variable header of the PUBCOMP packet is P and there is no message body.

5) When the Sender receives the PUBCOMP data packet, the data packet transmission is completed, and the local PUBREC data packet is lost. If the Sender does not receive the PUBCOMP packet within the timeout period, it will repeat the PUBREL packet;

The above is the process of completing a QoS2 level data packet. At least 4 data packets must be sent. However, some students found in step 3 that when the Sender did not receive the Receiver's PUBREC data packet, the PUBLISH data packet would still be repeatedly sent. , Then how to ensure that Receiver will only receive it once?

The reason is that when Receiver receives a PUBLISH packet, it does not deliver it to the upper layer of the protocol immediately, but saves the message locally by persisting it, and then delivers the message to the upper layer of the protocol after receiving the PUBREL packet, and then sends the locally saved Packet Identifier P is deleted. Since there is no Packet Identifier P in the PUBREL packet, even if PUBCOMP is not sent successfully and PUBREL is retransmitted, Receiver only needs to reply to PUBCOMP, and does not need to repeatedly deliver PUBLISH packets to the upper layer of the protocol;

The implementation principle of QoS2 service level can be viewed in this blog, which is very detailed: https://blog.csdn.net/zerooffdate/article/details/78950907

6.5 keepalive and connection keepalive

6.5.1 The role of keepalive

When actually using the MQTT protocol, both Broker and Client need to be aware of whether MQTT is disconnected in time; MQTT is an application layer protocol based on the TCP protocol. In theory, the upper layer application will be notified when TCP is disconnected, but the TCP protocol has a half-open The connection problem, in this state, a segment of the TCP connection has failed, but the other end does not know it, and it takes a long time to perceive that the peer connection has been disconnected; therefore, it is not enough to rely on the TCP layer connection status monitoring. So MQTT designed a set of keepalive mechanism,

The MQTT protocol stipulates that if the Broker does not receive any data packets from the Client within the 1.5*keepalive time interval, the Broker considers the connection with the Client to be disconnected; for the same reason, the Client does not receive any data packets from the Broker within this time interval. , Client will also think that his connection with Broker is disconnected;

The MQTT protocol designs a pair of PINGREQ/PINGRESP data packets. When there is no data interaction between Broker and Client, this pair of data packets can meet the keepalive agreement and network status monitoring;

6.5.2 keepalive features

If within a Keepalive time interval, Client and Broker have data packet transmission, Client does not need to use PINGREQ data packet;
The value of Keepalive is specified by the Client when sending the CONNECT packet, and different Clients can specify different values;
The maximum value of Keepalive is 18:12:15
When the value of Keepalive is 0, it means that the Keepalive mechanism is not used

6.6 Retained message

6.6.1 The role of the retained message

There will be such a scenario, after the Publisher publishes a message, the Subscriber subscribes to this topic, then this Subscriber will not receive the message published by the Publisher before it subscribes. The Retained message is to solve this problem. The Retained message refers to the message with the Retained flag set to 1 in the PUBLISH packet. After receiving such a PUBLISH packet, the Broker will save the message for the topic as a new subscription. When the subscriber subscribes to the topic, the Broker will send this message to the subscriber.

6.6.2 Features of Retained Message

A topic has only one retained message, and publishing a new retained message will overwrite the old retained message;
If subscribers subscribe to topics using wildcards, they will receive all retained messages matching topics;
Only new subscribers will receive Retained messages. If subscribers repeatedly subscribe to a topic, they will be treated as new subscribers and receive Retained messages;
Publish a retained message with a payload length of 0 to the topic to delete the retained message of this topic;

6.6.3 Retained messages and persistent sessions

There is no relationship between the two, the retained message is stored separately by the broker for each topic, and the persistent session is stored separately by the broker for each client;

6.7 LWT (Will)

The full name of LWT is Last Will and Testament, which is the last wish, including the subject of the last wish, the QoS of the last wish, the message of the last wish, etc. See the MQTT protocol for details. The last wish has the following characteristics:

The relevant settings of the last wish are specified in the CONNECT packet when the connection is established;
The last wish is used in the case of abnormal disconnection. When the Broker detects the abnormal disconnection of the Client, it will publish a message to the subject of the Client's last wish;
The client releases the DISCONNECT packet to disconnect, which is a normal disconnection and will not trigger the LWT mechanism, and the Broker will lose the LWT parameters specified when the Client is connected

Seven, MQTT version

The above are based on MQTT 3.1.1. For more information on MQTT 5.0, please refer to the official documentation.

Getting started with the MQTT protocol

I. Introduction