Baidu Tiangong AIoT team officially announced the open source high-performance distributed MQTT IoT messaging middleware in July 2023, and renamed it BifroMQ. After three months, BifroMQ-v2.0.0 version has been released .
The new version supports cluster mode for the first time, which is officially called StandardCluster (StandardCluster for short). It mainly has the following characteristics:
- Cluster mode fully supports the MQTT protocol: In cluster mode, each node has complete MQTT protocol functions, providing higher availability and scalability.
- New HTTP API support: In addition to MQTT, this version adds support for HTTP API, making it more flexible and changeable.
- High performance without sacrificing: Focus on high-performance MQTT protocol in large-scale load environments.
- The modular architecture is further optimized: distributed cluster management adapted to various business needs.
- Strong scalability: The goal is to support large-scale multi-tenant Serverless cloud services.
BifroMQ cluster architecture analysis
The overall structure of StandardCluster
BifroMQ logically decomposes the MQTT function into several sub-services from the perspective of load, and each sub-service corresponds to a type of key load:
- bifromq-mqtt: Responsible for MQTT protocol connection load
- bifromq-dist: Responsible for subscription and message routing distribution load
- bifromq-inbox: Responsible for offline message queue load in persistent sessions
- bifromq-retain: Responsible for Retain message access load
From a deployment perspective, BifroMQ StandardCluster is a cluster mode that "encapsulates" these independent load service modules into a node service process. Logically, this is an abstraction of the Standalone operating mode (BifroMQ Standalone can be regarded as a single node BifroMQ StandardCluster). Unlike other MQTT Brokers that support clusters, BifroMQ has built-in distributed persistence capabilities, so a single BifroMQ node is "Stateful".
Horizontal expansion of message distribution capabilities
In the StandardCluster cluster mode, the distribution load module (Dist Service for short) in each node process constitutes a logical load sub-cluster (Dist-SubCluster). Dist Service stores subscription information in the built-in persistence engine and synchronizes routing information between nodes through the distributed function of the persistence engine. The announcement stated that in StandardCluster mode, by adding nodes, the message distribution capability can be horizontally expanded, especially when CleanSession is True.
Persistence, scalability and high reliability of offline message queue
Similar to the Dist Service module, the Inbox Service module responsible for the offline message queue in the MQTT persistent session within the node process constitutes another logical load sub-cluster (hereinafter referred to as Inbox-SubCluster). Inbox Service persists offline queue messages into the built-in storage engine, which can greatly reduce data loss caused by node failure. In terms of storage, Inbox Service uses the sharding function of the built-in storage engine to achieve horizontal expansion of storage scale and processing capabilities. It can also dynamically increase the number of shard copies through static configuration or runtime policies, which can further improve the reliability of offline message data. This is particularly important for certain application scenarios that require higher data reliability.
Load-based splitting strategy for Inbox service
As mentioned earlier, under StandardCluster deployment, the Inbox Service within a single node process takes advantage of the sharding function of the built-in storage engine to achieve horizontal expansion of storage scale and processing capabilities. However, the sharding strategy has a decisive impact on the actual operating results. In the BifroMQ StandardCluster version, the load-based splitting strategy (hereinafter referred to as Load-based Splitting) is built-in out of the box. This strategy determines the division of KV Range by counting the load conditions in the recent period, and can be regarded as a "posterior" splitting strategy. When the usage scenario has planned and understood the distribution of offline message load in advance, dividing the Range in advance can often bring more stable performance when the load arrives. For users who use BifroMQ deeply, this can be achieved through the SPI mechanism." A priori "split strategy."
HTTP API module
The BifroMQ StandardCluster version also introduces the HTTP API function, and each cluster node can configure an open API access port. As a stateless global interface, BifroMQ HTTP API is designed to support the integration of management control logic at the business level. Accessing the API of any node can operate the entire cluster.
Impact of mixed loads on performance
Under the StandardCluster deployment, a single node has complete MQTT protocol functions and can bear various types of loads. Therefore, this model is very suitable for the following two types of enterprise-level application scenarios:
- The type of load generated by the business is relatively simple
- The load types generated by the business are diversified, but the generation time is relatively scattered. For situations where the load form is complex and concentrated in the time dimension, it is officially recommended that users simulate load testing to obtain the optimal resource configuration and parameter settings under a single BifroMQ StandardCluster cluster, or consider using multiple BifroMQ StandardClusters to host different types of business load.
Simple deployment and simplified operation and maintenance
BifroMQ clusters are built on a set of built-in decentralized technologies (base-cluster) without relying on external node registration and discovery services. Therefore, the deployment process of building BifroMQ StandardCluster is very simple. You only need to specify any node in the cluster as a seed node to complete the addition of new nodes. In addition, BifroMQ also has built-in self-healing capabilities after cluster splits, which can greatly simplify operation and maintenance operations when failures such as network partitions occur.