introduction
We are happy to announce: The MQTT Open Benchmark Specification provided by EMQ is now officially released!
The test specification includes practical typical usage scenarios, a set of main indicators to measure Broker performance, and a tool for simulating load and collecting test results, which can help developers evaluate the scalability and performance of MQTT Broker, so as to choose the most needed and Suitable product.
Objective and practical MQTT load testing method
In the publish/subscribe mode , MQTT Broker is the core of connecting devices and transmitting IoT data. With the popularity of the Internet of Things, it is full of challenges to deal with a large number of devices and the high-speed data streams generated by them. The scalability and performance of MQTT Broker has therefore become the key to Internet of Things solutions.
In this context, EMQ has proposed a set of MQTT open benchmark test specifications, aiming to provide an objective and practical MQTT Broker testing method, which will simplify the load testing of MQTT Broker through the following main advantages:
Provide open and practical evaluation benchmarks
Although there are many research papers and technical articles evaluating and comparing MQTT Brokers, current benchmarks usually only measure some specific factors and cannot reflect real-world scenarios of large-scale IoT applications. The MQTT open benchmark specification provided by EMQ conducts a comprehensive evaluation from multiple aspects, and at the same time designs a scenario that is fair to any Broker, so as to establish an objective and neutral test benchmark.
Based on real usage scenarios and actionable metrics
We analyzed and solved a large number of actual testing requirements from our customers, based on which we proposed this set of MQTT open benchmarking specifications. We believe it can meet the vast majority of testing needs.
Comprehensive benchmarks open to the community
We are committed to building an open and collaborative community around this set of MQTT benchmark specifications. As the various usage scenarios contributed by the community continue to increase, this set of specifications will more accurately reflect the needs of the industry. Welcome to join us in building a comprehensive benchmark specification for the MQTT protocol .
Contents of the MQTT Open Benchmark Specification
The first version of the MQTT Open Benchmark Specification mainly introduces the key factors that affect the performance of MQTT brokers, and defines metrics to measure scalability, availability, latency, and computing resource costs.
The specification also divides typical usage scenarios into connection, broadcast, point-to-point, and reporting, and includes practical use cases based on real customer needs.
Furthermore, to illustrate how these use cases are applied in benchmarking, we provide some examples of detailed benchmarking results.
Example benchmark results
In order to show a practical example, below we will introduce the details of a broadcast use case and benchmark test results using NanoMQ as MQTT Broker.
In this broadcast use case, a connection of 5 publishers and 1000 subscribers is established (i.e. more subscribers than publishers for "broadcasting" purposes) and 5 topics are used for pub/sub. As soon as each subscriber finishes connecting, subscribe to all 5 topics. After each publisher completes the connection, it sends a 16-byte message to an exclusive topic. Both publishers and subscribers use QoS 1. Each publisher publishes at a rate of 50 messages per second, so the total expected subscription rate is 250K messages per second.
In this benchmark, NanoMQ is deployed on a single node. The configuration details are as follows:
This benchmark test uses XMeter as a test tool, which simulates all device connections and message throughput, analyzes data and generates metrics reports. Here are some key indicators:
The detailed diagram is as follows:
future plan
We used NanoMQ as an example above, but in fact this set of test specifications is applicable to all brokers that use the MQTT protocol. We will also continue to iterate and improve it, and plan to launch a tool for MQTT benchmarking in the future.
Interested readers are welcome to check out our GitHub homepage for more information, and look forward to your valuable comments and contributions.
Copyright statement: This article is original by EMQ, please indicate the source for reprinting.
Original link: https://www.emqx.com/zh/blog/open-mqtt-benchmark-suite-the-ultimate-guide-to-mqtt-performance-testing