Message flow of MQTT protocol

1 network failure

In any network environment, one party will fail to connect, for example, there is no WIFI signal when leaving the company's gate. But the other end of the persistent connection - the server may not immediately know that the other party has been disconnected. Similar network abnormalities may occur during the message sending process, and the message will be lost after it is sent out.

The MQTT protocol assumes that the client and server are generally stable, and the communication channels between them are unreliable. Once the client network is disconnected, the situation will be very serious and it is difficult to restore the original state.

But don’t forget that many clients will have permanent storage devices, such as flash ROM, memory card, etc., which can be used to save key data or status information when communication is abnormal.

In short, abnormal network situations are very complicated and can only be handled with care.

2 Message resending strategy

In the case of QoS > 0 , the sender needs to wait for a response message after sending PUBLISH, PUBREL, SUBSCRIBE, UNSUBSCRIBE and other types of messages. If the message is not received within a specified time period, the sender may need to retry. Resent messages require the DUP flag to be set to 1.

The timeout for waiting for a response should be counted after the message is successfully sent, and the timeout for waiting should be a configurable option so that it can be increased appropriately during the next retry. For example, the first retry timeout is 10 seconds, the next time may be 20 seconds, and the next retry may be 60 seconds. Of course, there is also a limit on the number of retries.

There is also a case where the client reconnects without setting the clean session flag in the variable header, but both sides (client and server) should retry previously unsent in-flight messages. The client is not forced to send unacknowledged messages, but the server has to resend those unacknowledged messages.

Message flow determined by QoS level

QoS level is the abbreviation of Quality of Service level, translated into Chinese, service quality level.

In the "4.1 Quality of Service levels and flows" chapter of the MQTT 3.1 protocol, only the publishing process from the client to the server is discussed, which is not complete. Because it determines the message arrival rate and can improve the sending quality, it should be the direction of the message flow where the server publishes the PUBLISH message to the subscriber.

QoS level 0

Send at most once, and discard once sent. There is no confirmation message, and it is not known whether the other party received it.

Client	Message and direction	Server
QoS = 0	PUBLISH `---------->`	Action: Publish message to subscribers then Forget Reception: <=1

The targeted message is not important, and it doesn't matter if it is lost.

At the network level, the transmission pressure is small.

QoS level 1

All QoS level 1 must append a 16-bit message ID in the variable header.

SUBSCRIBE and UNSUBSCRIBE messages use QoS level 1.

For message publishing, Qos level 1 means that the message has been transmitted at least once.

If the sender does not receive the PUBACK message within a period of time, the sender needs to turn on the DUB flag as 1, and then resend the PUBLISH message. Therefore, the receiver may receive two PUBLISH messages. The message flow for publishing messages from the client to the server:

Client Message and direction Server

Client	Message and direction	Server
QoS = 1 DUP = 0 Message ID = x Action: Store message	PUBLISH `---------->`	Actions: Store message Publish message to subscribers Delete message Reception: >=1
Action: Discard message	PUBACK `<----------`	Message ID = x

QoS = 1
DUP = 0
Message ID = x

Action: Store message

PUBLISH
---------->

Actions:

Store message
Publish message to subscribers
Delete message

Reception: >=1

Action: Discard message

PUBACK
<----------

Message ID = x

Message flow for server publishing to subscribers:

Server Message and direction Subscriber

Server	Message and direction	Subscriber
QoS = 1 DUP = 0 Message ID = x	PUBLISH `---------->`	Actions: Store message Make message available Reception: >=1
	PUBACK `<----------`	Message ID = x

QoS = 1
DUP = 0
Message ID = x

PUBLISH
---------->

Actions:

Store message
Make message available

Reception: >=1

PUBACK
<----------

Message ID = x

If the publisher (client/server) fails to receive the PUBACK message due to various abnormalities, it will resend the PUBLISH message again and set the DUP flag to 1. The receiver takes the server as an example, which may cause the server to receive duplicate messages. According to the process, the broker (server) publishes a message to the subscriber (which will cause the subscriber to receive a duplicate message), and then sends a PUBACK confirmation message to the publisher.

At the business level, it may be possible to make up for the shortcomings of the MQTT protocol: the retried message ID must be consistent, and the receiver must determine whether the currently received message ID has been accepted.

But the same cannot be completely guaranteed, the news must have arrived.

QoS level 2

Appears only in PUBLISH type messages and requires the message ID to be appended in the variable header.

The level is high, and the communication pressure is slightly greater, but it ensures that only one transmission and reception is performed.

First look at the flowchart in the agreement, Client -> Server direction, you will have an overall impression:

Client	Message and direction	Server
QoS = 2 DUP = 0 Message ID = x Action: Store message	PUBLISH `---------->`	Action(a) Store message or Actions(b): Store message ID Publish message to subscribers
	PUBREC `<----------`	Message ID = x
Message ID = x	PUBREL `---------->`	Actions(a): Publish message to subscribers Delete message or Action(b): Delete message ID
Action: Discard message	PUBCOMP `<----------`	Message ID = x

Server -> Subscriber：

Server	Message and direction	Subscriber
QoS = 2 DUP = 0 Message ID = x	PUBLISH `---------->`	Action: Store message
	PUBREC `<----------`	Message ID = x
Message ID = x	PUBREL `---------->`	Actions: Make message available
	PUBCOMP `<----------`	Message ID = x

The schemes a and b adopted by the server side both include when the message is valid and when the message is processed. Choose one of the two options, and the server side decides by itself. But no matter which method is used, it is under the scope of the QoS level 2 agreement and will not be affected. If one party does not receive the corresponding confirmation message, it will retry from the latest message that needs to be confirmed, so that the whole (QoS level 2) process can be opened.

3 Message order

The order of messages can be affected by many factors, but for server programs, it is necessary to ensure that each stage of the message delivery process is consistent with the order in which it started. For example, in the message flow defined by QoS level 2, the PUBREL flow must be sent in the same order as the PUBLISH flow:

Client	Message and direction	Server
	PUBLISH 1 `---------->` PUBLISH 2 `---------->` PUBLISH 3 `---------->`
	PUBREC 1 `<----------` PUBREC 2 `<----------`
	PUBREL 1 `---------->`
	PUBREC 3 `<----------`
	PUBREL 2 `---------->`
	PUBCOMP 1 `<----------`
	PUBREL 3 `---------->`
	PUBCOMP 2 `<----------` PUBCOMP 3 `<----------`

流动消息(in-flight messages)数量允许有一个可保证的效果：

在流动消息(in-flight)窗口1中，每个传递流在下一个流开始之前完成。这保证消息以提交的顺序传递
在流动消息(in-flight)大于1的窗口，只能在QoS level内被保证消息的顺序

4 消息的持久化

在MQTT协议中，PUBLISH消息固定头部RETAIN标记，只有为1才要求服务器需要持久保存此消息，除非新的PUBLISH覆盖。

对于持久的、最新一条PUBLISH消息，服务器不但要发送给当前的订阅者，并且新的订阅者(new subscriber，同样需要订阅了此消息对应的Topic name）会马上得到推送。

Tip：新来乍到的订阅者，只会取出最新的一个RETAIN flag = 1的消息推送，不是所有。

5 消息流的编码/解码

MQTT协议中，由目前定义的14种类型消息在客户端和服务器端之间数据进行交互。若以JAVA语言构建MQTT服务器，可选择Netty作为基础。

在Netty中，数据的进入和流出，代表了一次完整的交互。无论是要进入的还是要流出的数据(单独以服务器为例)，都可看做字节流。若把每种类型消息抽象为一个具体对象，那么处理起来就不难了。

客户端->服务器，进入的字节流，逐个字节/单位读取，可还原成一个具体的消息对象（解码的过程）。

要发送到客户端的消息对象，转换（编码）成字节流，然后由TCP通道流转到接收者。