Header information of MQTT protocol

1 Overview

MQTT (Message Queue Telemetry Transport), a telemetry transmission protocol, provides a subscription/publishing mode, which is more simple, lightweight, and easy to use. For restricted environments (low bandwidth, high network delay, and unstable network communication), it can be simply summarized as Created by the Internet of Things, the official summary features are as follows:

1.使用发布/订阅消息模式，提供一对多的消息发布，解除应用程序耦合。
2. 对负载内容屏蔽的消息传输。
3. 使用 TCP/IP 提供网络连接。
4. 有三种消息发布服务质量：
    “至多一次”，消息发布完全依赖底层 TCP/IP 网络。会发生消息丢失或重复。这一级别可用于如下情况，环境传感器数据，丢失一次读记录无所谓，因为不久后还会有第二次发送。
    “至少一次”，确保消息到达，但消息重复可能会发生。
    “只有一次”，确保消息到达一次。这一级别可用于如下情况，在计费系统中，消息重复或丢失会导致不正确的结果。
5. 小型传输，开销很小（固定长度的头部是 2 字节），协议交换最小化，以降低网络流量。
6. 使用 Last Will 和 Testament 特性通知有关各方客户端异常中断的机制。

MQTT 3.1 protocol online version: http://public.dhe.ibm.com/software/dw/webservices/ws-mqtt/mqtt-v3r1.html

Official download address: http://public.dhe.ibm.com/software/dw/webservices/ws-mqtt/MQTT_V3.1_Protocol_Specific.pdf

At present, MQTT is widely used in mobile phone push services. Based on the MQTT protocol, a more private and streamlined (some parts of the protocol are slightly redundant) transmission protocol can be created, such as a one-byte transmission overhead.

2 fixed head

Fixed header, using two bytes, 16 bits in total:

bit	7	6	5	4	3	2	1	0
byte 1	Message Type				DUP flag	QoS level		RETAIN
byte 2	Remaining Length

The first byte (byte 1)

Message type (4-7), using 4-bit binary representation, can represent 16 message types:

Mnemonic	Enumeration	Description
Reserved	0	Reserved
CONNECT	1	Client request to connect to Server
CONNACK	2	Connect Acknowledgment
PUBLISH	3	Publish message
PUBACK	4	Publish Acknowledgment
PUBREC	5	Publish Received (assured delivery part 1)
PUBREL	6	Publish Release (assured delivery part 2)
PUBCOMP	7	Publish Complete (assured delivery part 3)
SUBSCRIBE	8	Client Subscribe request
SUBACK	9	Subscribe Acknowledgment
UNSUBSCRIBE	10	Client Unsubscribe request
UNSUBACK	11	Unsubscribe Acknowledgment
PINGREQ	12	PING Request
PINGRESP	13	PING Response
DISCONNECT	14	Client is Disconnecting
Reserved	15	Reserved

Except for positions 0 and 15, which are reserved for use, there are 14 types of message events in total.

3 DUP flag (open flag)

Ensure the reliable transmission of messages. The default is 0, which only occupies one byte, indicating the first sending. It cannot be used to detect repeated sending of messages, etc. Only applicable to client or server trying to resend PUBLISH, PUBREL, SUBSCRIBE or UNSUBSCRIBE messages, note that the following conditions must be met:

 当QoS > 0
 消息需要回复确认

In this case, the message ID needs to be included in the variable header. When the value is 1, it means that the current message has been sent before.

4 QoS (Quality of Service, Quality of Service)

Use two binary representations of PUBLISH type messages:

QoS value	bit 2	bit 1	Description
0	0	0	at most once	throw away	<=1
1	0	1	at least once	Confirmation required	>=1
2	1	0	Only one time	Confirmation required	=1
3	1	1	Standby, reserved place

5 RETAIN (keep)

For PUBLISH messages only. Different values, different meanings:

1 : Indicates that the sent message needs to be kept persistently (not affected by server restart), not only to the current subscriber, but also to new subscribers who subscribe to this Topic name will be pushed immediately.

Remarks: New subscribers will only fetch the latest message push with RETAIN flag = 1.

0 : Push this message only for current subscribers.

If the server receives a PUBLISH message with an empty message body (zero-length payload), RETAIN = 1, and an existing Topic name, the server can delete the corresponding persistent PUBLISH message.

6 How to parse

Because java uses signed (the highest bit is the sign bit) data representation, byte range: -128-127. The most significant bit of the byte (the first bit on the left), may be 1. If it is directly converted to byte type, negative numbers will appear, which is a minefield. DataInputStream provides int readUnsignedByte() reading method, please pay attention. The following demonstrates how to obtain all defined information from a byte while bypassing minefields:

public static void main(String[] args) {
    byte publishFixHeader = 50;// 0 0 1 1 0 0 1 0

    doGetBit(publishFixHeader);
    int ori = 224;//1110000,DISCONNECT ,Message Type (14)
    byte flag = (byte) ori; //有符号byte       
    doGetBit(flag);
    doGetBit_v2(ori);
}


public static void doGetBit(byte flags) {
    boolean retain = (flags & 1) > 0;
    int qosLevel = (flags & 0x06) >> 1;
    boolean dupFlag = (flags & 8) > 0;
    int messageType = (flags >> 4) & 0x0f;

    System.out.format(
            "Message type:%d, DUP flag:%s, QoS level:%d, RETAIN:%s\n",
            messageType, dupFlag, qosLevel, retain);
}

public static void doGetBit_v2(int flags) {
    boolean retain = (flags & 1) > 0;
    int qosLevel = (flags & 0x06) >> 1;
    boolean dupFlag = (flags & 8) > 0;
    int messageType = flags >> 4;

    System.out.format(
            "Message type:%d, DUP flag:%s, QoS level:%d, RETAIN:%s\n",
            messageType, dupFlag, qosLevel, retain);
}

7 Processing Remaining Length (remaining length)

The number of bytes remaining in the current message, including variable headers and payloads (called content/body, more appropriately). Maximum value of a single byte: 01111111, hexadecimal: 0x7F, decimal 127. Why can't a single byte be 11111111 (0xFF)? Because the MQTT protocol stipulates that if the eighth bit (highest bit) is 1, it means that there are subsequent bytes. At the same time, the MQTT protocol allows up to 4 bytes to indicate the remaining length. Then the maximum length is: 0xFF, 0xFF, 0xFF, 0x7F, binary representation: 11111111, 11111111, 11111111, 01111111, decimal: 268435455 byte=261120KB=256MB=0.25GB The range of values ​​between four bytes:

Digits	From	To
1	0 (0x00)	127 (0x7F)
2	128 (0x80, 0x01)	16 383 (0xFF, 0x7F)
3	16 384 (0x80, 0x80, 0x01)	2 097 151 (0xFF, 0xFF, 0x7F)
4	2 097 152 (0x80, 0x80, 0x80, 0x01)	268 435 455 (0xFF, 0xFF, 0xFF, 0x7F)

How to convert it to decimal? Use the java language to express as follows:

public static void main(String[] args) throws IOException {
    // 模拟客户端写入
   ByteArrayOutputStream arrayOutputStream = new ByteArrayOutputStream();
   DataOutputStream dataOutputStream = new DataOutputStream(arrayOutputStream);
   dataOutputStream.write(0xff);
   dataOutputStream.write(0xff);
   dataOutputStream.write(0xff);
   dataOutputStream.write(0x7f);

   InputStream arrayInputStream = new ByteArrayInputStream(arrayOutputStream.toByteArray());

    // 模拟服务器/客户端解析
   System. out.println( "result is " + bytes2Length(arrayInputStream));
}

/**
* 转化字节为 int类型长度
* @param in
* @return
* @throws IOException
*/
private static int bytes2Length(InputStream in) throws IOException {
    int multiplier = 1;
    int length = 0;
    int digit = 0;
    do {
        digit = in.read(); //一个字节的有符号或者无符号，转换转换为四个字节有符号 int类型
        length += (digit & 0x7f) * multiplier;
        multiplier *= 128;
   } while ((digit & 0x80) != 0);

    return length;
}

Generally, the last byte is less than 127 (01111111), and 0x80 (10000000) is used for & operation, and the final result is 0, so the calculation will be terminated. The proxy middleware and the requester pass the byte stream Stream in the middle, so naturally they need to read from the stream and parse them one by one.

So how to parse the int type length into an indeterminate byte value?

public static void main(String[] args) throws IOException {
    // 模拟服务器/客户端写入
   ByteArrayOutputStream arrayOutputStream = new ByteArrayOutputStream();
   DataOutputStream dataOutputStream = new DataOutputStream(
             arrayOutputStream);

    // 模拟服务器/客户端解析
    length2Bytes(dataOutputStream, 128);
}

/**
* int类型长度解析为1-4个字节
* @param out
* @param length
* @throws IOException
*/
private static void length2Bytes(OutputStream out, int length)
         throws IOException {
    int val = length;
    do {
         int digit = val % 128;
        val = val / 128;
         if (val > 0)
             digit = digit | 0x80;

        out.write(digit);
   } while (val > 0);
}

digit modulo val, the maximum value may be 127, once 127 | 10000000 = 11111111 = 0xff = 255 Please note: the remaining length is only in the fixed header, whether it is one byte or four bytes, cannot be counted as a variable header.

8 variable headers

The fixed header only defines the message type and some flags, and some metadata of the message needs to be put into the variable header. Variable header content byte length + Playload/payload byte length = remaining length, this needs to be kept in mind. Variable header, including protocol name, version number, connection flag, user authorization, heartbeat time, etc. This part is repeated with the CONNECT message type to be mentioned later, so it is temporarily skipped.

9 Playload/message body/load

The message body mainly exists to cooperate with fixed/variable header commands (for example, if the CONNECT variable header User name flag is 1, the user name string needs to be appended to the message body).

Messages such as CONNECT/SUBSCRIBE/SUBACK/PUBLISH have a message body. The message body of PUBLISH is treated in binary form.

Please remember that the MQTT protocol only allows the use of custom features in the PUBLISH type message body. If you want to add custom private features in the fixed/variable header, you can avoid it. This is also to prevent the protocol from being mere formality, becoming very split and to take into account existing clients and so on. For example, to support compression, etc., you can define data support in Playload and read and process it in the application.

This part will be discussed in detail later.

10 Message Identifier/Message ID

Only when the value of the QoS level flag in the fixed header is 1 or 2 will it appear in the variable headers of messages such as PUBLISH, PUBACK, PUBREC, PUBREL, PUBCOMP, SUBSCRIBE, SUBACK, UNSUBSCRIBE, and UNSUBACK.

A 16-bit unsigned short type value (the value cannot be 0, 0 is reserved as an invalid message ID), only required in a specific direction (the server sends to the client in one direction, the client sends to the server in the other direction) One direction) must be unique in the communication message. For example, if the client sends to the server, there may be duplication in the server's sending to the client, but it doesn't matter.

In the variable header, the order of the two bytes required is MSB (Most Significant Bit) LSB (Last/Least Significant Bit), which translates into Chinese as the most significant bit and the least significant bit. The most significant bit is on the left/above the least significant bit, indicating that this is a big-endian byte/network byte order, which conforms to people's reading habits, and the high bit is on the far left.

bit	7	6	5	4	3	2	1	0
	Message Identifier MSB
	Message Identifier LSB

But everything represented in this way can be regarded as a 16-bit unsigned short type integer, represented by two bytes. Processing in JAVA is relatively simple:

DataInputStream.readUnsignedShort

or

in.read() * 0xFF + in.read();

Maximum length can be: 65535

11 UTF-8 encoding

For character strings, MQTT uses a modified version of UTF-8 encoding. The general form is as follows, which needs to be kept in mind:

bit	7	6	5	4	3	2	1	0
byte 1	String Length MSB
byte 2	String Length LSB
bytes 3 ...	Encoded Character Data

For example, AVA uses the writeUTF() method to write a string of text "OTWP". The first two bytes are a complete unsigned number, representing the byte length of the string, and the last four bytes are the real length of the string. A total of six bytes:

bit	7	6	5	4	3	2	1	0
byte 1	Message Length MSB (0x00)
	0	0	0	0	0	0	0	0
byte 2	Message Length LSB (0x04)
	0	0	0	0	0	1	0	0
byte 3	'O' (0x4F)
	0	1	0	0	1	1	1	1
byte 4	'T' (0x54)
	0	1	0	1	0	1	0	0
byte 5	'W' (0x57)
	0	1	0	1	0	1	1	1
byte 6	'P' (0x50)
	0	1	0	1	0	0	0	0

At this point, in the program, you don't need to deal with the default separately, and use the readUTF() method directly, which can automatically save the trouble of dealing with the length of the string. Of course, the string can be read manually:

// 模拟写入
dataOutputStream.writeUTF( "abcd");// 2 + 4 = 6 byte
......
// 模拟读取 
int decodedLength = dataInputStream.readUnsignedShort();//2 byte
byte[] decodedString = new byte[decodedLength]; // 4 bytes
dataInputStream.read(decodedString);
String target = new String(decodedString, "UTF-8");

Equivalent to:

String target = dataInputStream.readUTF();

MQTT, whether it is a variable header or a message body, as long as it is a string part, it adopts a modified version of UTF-8 encoding, read and write, with the help of DataInputStream/DataOutputStream, one line of statements, omitting the manual Dealing with the hassle.

Mastering the functions and meanings of the QoS level of the fixed header, the RETAIN flag, and the Connect flags of the variable header is very useful for the overall understanding of MQTT.