Getting Started with MQTT Performance Testing: Common Test Scenarios and Metrics

introduction

In the field of Internet of Things, there are a large number of resource-constrained sensors and industrial control devices running in a low-bandwidth and unstable network environment, which makes MQTT an ideal message transmission protocol in the Internet of Things scenario. Therefore, MQTT Broker must guarantee excellent performance and high reliability to meet the requirements of IoT applications.

Before proceeding to system testing, it is crucial to understand the basic test scenarios and performance metrics. In this article, we will provide a detailed description based on the testing experience of the EMQX team, which is also applicable to other MQTT Broker tests.

Glossary

MQTT protocol: MQTT (Message Queuing Telemetry Transport) is a lightweight message transmission protocol based on the publish/subscribe model. Although it has the word "message queue" in its name, it has nothing to do with message queues. The protocol has become the preferred protocol in the field of Internet of Things because of its simplicity, flexibility, easy implementation, support for QoS, and small message size.

Performance testing: Performance testing refers to the use of testing tools to simulate various normal, peak or abnormal load conditions to evaluate the performance of the system under test on various performance indicators. Its purpose is to verify whether the system can meet user expectations, and to find performance bottlenecks and problems in the system.

Common MQTT test scenarios

MQTT Broker mainly has two test scenarios:

• Concurrent connections, including the number of concurrent connections and connection rate.
• Message throughput, including the throughput of message sending and receiving, and some factors that affect the performance of the production environment system, such as QoS, payload size, topic wildcard, etc.

When designing a specific performance test scenario, especially when conducting a PoC or pre-deployment test, the following two points must always be kept in mind:

• Try to simulate the usage in the real production environment as much as possible.
• Cover possible peak loads.

Test scenarios can be divided according to the two basic dimensions of connection and message throughput.

concurrent connection test

MQTT connection is a long connection based on TCP. The client first establishes a TCP connection with the MQTT Broker, and then sends an MQTT login request. After the connection is successfully established, the client and MQTT Broker maintain the connection state by sending heartbeat packets periodically. Therefore, establishing and maintaining an MQTT connection for a long time requires certain resources of the MQTT broker. In a high-concurrency scenario, such a long connection will consume a lot of resources of the broker. Therefore, through performance testing, we can evaluate how many concurrent connections MQTT Broker can sustain under limited resources.

In addition, the higher the connection rate (that is, the number of new connections per second), the more computing resources are required. This factor needs to be considered when formulating test scenarios, because in some scenarios, a large number of devices will go online at the same time, and the ability to test the broker Or this indicator is needed when planning system capacity.

In the concurrent connection test, it is also necessary to consider whether to use TLS/SSL encrypted transmission, because it will increase the additional resource overhead of the pressure machine and MQTT Broker. When planning your tests, you need to evaluate its impact on performance.

To sum up, in the MQTT concurrent connection test, the following three scenarios should be considered:

1. Gradually increase the number of concurrent connections at a fixed lower connection rate to test system response and resource consumption. This determines the maximum number of concurrency the system can sustain with given hardware and network resources.
2. With a given number of concurrent connections, test the response and resource consumption of the system at different connection rates.
3. When designing 1) and 2), distinguish between normal TCP connections and TLS/SSL encrypted connections.

Message Throughput Test

As mentioned earlier, MQTT is a message transmission protocol based on the publish/subscribe model. It is an asynchronous protocol that implements three types of publish-subscribe 1-to-1, 1-to-many, and many-to-1. It is widely used in Various IoT scenarios. Therefore, the message throughput test should cover the following three scenarios:

1. 1 to 1: There are equal numbers of publishers and subscribers. For each publisher, there is only one subscriber who subscribes to the topics it publishes. That is, the inflow rate of messages to MQTT Broker is the same as the outflow rate.
2. Many-to-1 (report): A typical IoT application scenario, where there are a large number of IoT devices as publishers, but only a few or a single subscriber, such as a large number of devices reporting their status or data.
3. 1-to-many: In the broadcast mode, a small number of clients publish messages, and a large number of devices subscribe to and consume messages, such as sending instructions from the control terminal.

Also, when designing message throughput scenarios, don't ignore factors such as QoS, message payload size, subscription topics with wildcards, etc. Different QoS have a great impact on the performance and resource consumption of the load test. The payload size can be determined according to the actual usage.

other scenes

For other MQTT functions, such as shared subscriptions, message dumping to databases or other message queues (MQ), massive topic subscriptions, and extreme situations such as simultaneous connection/disconnection of many MQTT clients, it can be designed and tested according to actual needs scene.

performance metrics

After designing the test scenario, it is necessary to develop metrics to evaluate the success of the test.

In performance testing, indicators can generally be divided into two categories: application system indicators (such as MQTT Broker indicators) and computing resource indicators.

• Application system indicators are related to user scenarios and requirements, such as response time (or delay), concurrency, etc.
• Computing resource metrics are related to hardware resource consumption. For the MQTT tests we discuss, these metrics are similar to those for other software performance tests, such as CPU, memory, network, disk I/O.

MQTT system indicators are closely related to test scenarios, and common indicators are shown in the table below.

performance testing tools

Large-scale performance testing needs to be able to simulate high-concurrency and high-throughput scenarios quickly, realistically, and stably. At the same time, it needs to manage and maintain many machines and resources. Choosing the right testing tool can achieve twice the result with half the effort.

The EMQX team uses two performance testing tools, emqtt_bench and XMeter.

emqtt_bench

emqtt_bench is an MQTT protocol performance testing tool written by the EMQX R&D team based on Erlang. After the installation is complete, it can be used through the command line.

用法：emqtt_bench pub | sub | conn

Compared with other tools, the advantage of emqtt_bench is that it is easy to install and use, and takes up less computing resources. However, the scenarios it supports are relatively limited, and it needs to be combined with other monitoring tools to test indicator data

For specific installation and usage methods, please refer to https://github.com/emqx/emqtt-bench

XMeter

emqtt_bench is suitable for quick performance verification during the development phase. If you want to conduct large-scale testing or formal testing, we recommend another more professional performance and load testing tool - XMeter .

XMeter is a performance testing tool based on JMeter. It has transformed the architecture of JMeter to achieve the ability of full horizontal expansion. Easily handle large amounts of data and perform high-frequency testing. XMeter not only inherits the powerful functions of JMeter, but also adds many new features. During the test, it provides rich and real-time test reports, allowing testers to view key performance indicators of MQTT at any time, such as throughput, response time, and success rate. At the same time, XMeter has a built-in monitoring system, which can monitor the resource consumption of MQTT Broker in real time.

In addition, XMeter provides automated and centralized test resource management capabilities. Test machines (containers) are created automatically at the beginning of the test and destroyed at the end of the test.

During the entire test phase, XMeter will display the MQTT performance indicators and computing resource usage in real-time graphically, as shown in Figures 1 to 5.

XMeter User Guide

XMeter is available in two editions.

• XMeter local private deployment. Ideal for organizations that need complete control over their testing environment and adhere to strict security and data privacy regulations. To use this version you need:

  • Download the open-source mqtt-jmeter plugin developed by the XMeter team from GitHub - emqx/mqtt-jmeter: MQTT JMeter Plugin .

  • Put the jar file into the JMeter directory.

  • According to the application scenario, write the test script in JMeter, as shown in Figure 6.

  • Upload the script to XMeter and start performance testing against MQTT.

    

• XMeter Cloud: fully managed MQTT load testing cloud service, easy to use:

  • Initiate MQTT performance test with one click, no need to manually deploy test resources
  • It only takes 3 steps to complete the MQTT test configuration, eliminating the burden of writing scenario scripts
  • Test resources are created on-demand on the cloud, and the test environment is highly automated, saving a lot of time and labor costs

You only need to register a free trial account on our website , and then follow the guidance of this document to start your XMeter journey.

Summarize

In this article, we discussed several common test scenarios and key indicators used to evaluate the performance of MQTT Broker. By understanding and applying these testing techniques and indicators, you can optimize MQTT system performance and reliability, and improve the overall level of IoT and messaging infrastructure.

Copyright statement: This article is original by EMQ, please indicate the source for reprinting.

Original link: https://www.emqx.com/zh/blog/getting-started-with-mqtt-performance-testing-a-primer-on-scenarios-and-metrics