In this tutorial, we will introduce the MQTT protocol and the format of MQTT messages or data packets in more detail.
We will study:
MQTT message format.
MQTT message header
Message field and encoding
Control message encoding example
Introduction
MQTT is a binary-based protocol, and the control elements are binary bytes instead of text strings.
MQTT uses commands and commands to confirm the format.
This means that every command has a corresponding confirmation.
The subject name, client ID, user name and password are encoded as UTF-8 strings.
Except for MQTT protocol information (such as client ID, etc.), the payload is binary data, and the content and format are application-specific.
The MQTT data packet or message format consists of a 2-byte fixed header (always present) + variable header (not always present) + payload (not always present).
The possible packet formats are:
- Fixed Header (Control field + Length) – Example CONNACK
- Fixed Header (Control field + Length) + Variable Header -Example PUBACK
- Fixed Header (Control field + Length) + Variable Header + payload -Example CONNECT
The fixed header field is composed of a control field and a variable-length packet length field.
The minimum size of the packet length field is 1 byte, which is used for messages with a total length of less than 127 bytes (excluding the control field and the length field).
The maximum packet size is 256MB. Small data packets smaller than 127 bytes have a 1-byte data packet length field.
Data packets larger than 127 and smaller than 16383 will use 2 bytes. and many more
Note: Only 7 bits are used, and the 8th bit is used as a continuous bit.
Note: 7 bits are used with the 8th bit being a continuation bit.
The minimum data packet size is only 2 bytes , which contains a single-byte control field and a single-byte data packet length field. For example, the disconnect message has only 2 bytes.
Control field
The 8-bit control field is the first byte of the 2-byte fixed header. It is divided into two 4-bit fields, containing all protocol commands and responses.
The first 4 most significant bits are the command or message type field, and the other 4 bits are used as control flags.
The following table is taken from the MQTT 3.1.1 specification and shows examples of MQTT commands and related codes.
Because they are the most important part of the 8-bit byte field, I also displayed their byte value in decimal, as if they appeared in the data packet.
Control flag
Although there are 16 possible signs, only a few of them are actually used.
The release message makes full use of these signs, as shown in the following table:
Use the repeat flag when reposting a message using QOS or 1 or 2
Use the QOS flag when publishing a message, and indicate the QOS level -0,1,2
The reserved message flag is also used for publishing.
Length field (remaining length)
Its length is between 1 and 4 bytes. Each byte uses 7 bits as the length, and the MSB is used as a continuity flag.
The remaining length is the number of bytes following the length field, including the variable-length header and payload, as shown below:
The following illustrates the length field of the data packets with lengths of 64 and 321 bytes
The remaining data packet length of 64 bytes only needs 1 byte:
A packet length of 321 bytes requires a remaining length field of 2 bytes:
The following table taken from the specification shows the packet size and packet length fields.
Variable length header
As mentioned earlier, variable-length header fields do not always appear in MQTT messages.
Some MQTT message types or commands require this field to carry other control information.
The variable length header fields are similar, but different for all message types.
MQTT connection and disconnection message example
As an illustration, we will now look at the data packet details of the connection message.
The following is an example of a real client connection and disconnection, showing the actual byte values of the sent and received data.
CONNECT control code = 0x10
CONNACK control code = 0x20
MQTT packet = control + length + protocol name + protocol level + connection flag + keep alive + payload
(MQTT packet =control + length + protocol name + Protocol Level +Connect Flags + keep alive +Payload)
annotation:
- Note the connection (0x10) and connection confirmation (0x20) control codes.
- Note the total length of hexadecimal 0x17 or 23 bytes, which does not include the control field and the length field. The length field is only 1 byte.
- You should also be able to see the client ID (python_test) in the sent packet.
- When looking at the actual packet bytes, Python will print the hexadecimal value unless it can match ASCII characters. In the above example, the keep alive field is x00x3C, but it is displayed as x00 <. Because Ascii <= 0x3C
annotation:
- The client ID field is sent as the first part of the payload, not as part of the header.
- The client ID is followed by a length field.
- The connection flag indicates that a clean session is being requested.
- The connection flag is part of the "variable length" header and is used to indicate the presence or absence of the username, password, and message fields in the payload. It also contains the clean session logo and Will QOS.
Control package summary table
| Control Packet | Variable Header | Payload |
| CONNECT | Required | Required |
| CONNACK | None | None |
| PUBLISH | Required | Optional |
| PUBACK | Required | None |
| PUBREC | Required | None |
| PUBREL | Required | None |
| PUBCOMP | Required | None |
| SUBSCRIBE | Required | Required |
| SUBACK | Required | Required |
| UNSUBSCRIBE | Required | Required |
| UNSUBACK | Required | Required |
| PINGREQ | None | None |
| PINGRESP | None | None |
| DISCONNECT | None | None |
Wireshark network analysis
In response to readers’ questions about the TCP protocol, I created this screenshot taken from Wireshark.
It shows an MQTT client connecting and publishing (QOS 1). You can clearly see the ACK packet with a total length of 58 bytes.
We know that the ACK packet is 2 bytes.
Therefore, a TCP packet without MQTT is about 56 bytes.
Another interesting thing to note is that until I completed the packet capture, I did not expect that every MQTT command or response will get a TCP ACK, and even an MQTT ACK.
If you look at the screenshot, the MQTT connection can get the TCP ACK response and the MQTT Connect ACK response.
The MQTT Connect ACK response will have a TCP ACK response.
Reference: MQTT V3.1.1 Specification pdf
http://www.steves-internet-guide.com/mqtt-protocol-messages-overview/