https://github.com/hui6075/mosquitto-cluster/blob/master/readme-zh_cn.md

Mosquitto Cluster

In a Mosquitto cluster, clients can subscribe to topics on any node, or publish messages on any node, and the cluster will ensure that messages are forwarded to the correct node on demand.
To balance load and avoid single points of failure, Mosquitto clusters are implemented in a fully decentralized and autonomous manner.

Compile and install

> git clone https://github.com/hui6075/mosquitto-cluster.git

> cd mosquitto-cluster && vi config.mk

Install

Find the file docbook.xsl through the command, and it is not found in this machine, so it is basically determined that this file cannot be found:

[root@localhost mosquitto-1.2]# find / -name docbook.xsl

Solution:

1. Install docbook-style-xsl

Use the command: yum -y install docbook-style-xsl

2. Check where the file docbook.xsl is installed:

E.g:

[root@localhost tools]# find / -name docbook.xsl
/usr/share/sgml/docbook/xsl-stylesheets-1.78.1/epub/docbook.xsl
/usr/share/sgml/docbook/xsl-stylesheets-1.78.1/fo/docbook.xsl
/usr/share/sgml/docbook/xsl-stylesheets-1.78.1/html/docbook.xsl
/usr/share/sgml/docbook/xsl-stylesheets-1.78.1/manpages/docbook.xsl
/usr/share/sgml/docbook/xsl-stylesheets-1.78.1/xhtml/docbook.xsl
/usr/share/sgml/docbook/xsl-stylesheets-1.78.1/xhtml-1_1/docbook.xsl

3. After searching, modify the address in the red letter above to the corresponding position in the file manpage.xsl;

# WITH_BRIDGE:=yes
WITH_CLUSTER:=yes

> make && make install

deploy

Install Mosquitto on all nodes, and write the node name, IP, and port number into the configuration file mosquitto.conf, for example:

node_name node1
node_address 192.168.1.1:1883

node_name node2
node_address 192.168.1.2:1883

Then configure the load balancer to use the addresses of all nodes: port numbers as the backend service addresses. Mosquitto is a single-process implementation, it is recommended to deploy with multiple instances on a single machine, and terminate TLS in the load balancer.

Mosquitto Cluster Features

The client's connection/subscription/unsubscription messages are broadcast to other nodes in the cluster, and the published messages are forwarded on demand.

loop avoidance

Any node will only send publish messages from other nodes to the client, and subscribe/unsubscribe messages from other nodes will not be forwarded.

Avoid duplicate subscriptions

对于每个来自客户端的订阅主题，以引用计数的方式保存在本地，只有在收到新的客户端主题订阅消息，或者主题不再被任何客户端订阅时进行转发。

私有消息

集群会话支持

每次客户端连接时，若本地没有发现会话信息，则广播SESSION REQ消息，其他节点收到此请求时若发现此客户端的会话信息，则断开/清除此客户端/上下文，若会话清除标识被设为FALSE，则返回此客户端的订阅状态及未完成的QoS为0和1的发布消息。此特性可以通过配置文件禁用，但压测和性能采样结果表明此特性在大并发的情况下并不会带来特别大的开销。

集群保留消息支持

每次收到来自于客户端的订阅时，节点会广播PRIVATE SUBSCRIBE，如果存在保留消息，其他节点会把此保留消息及接收时间通过PRIVATE RETAIN消息返回给此节点，因此客户端得以收到集群中绝对时间戳最晚的保留消息。

QoS支持

所有发布消息以原始QoS在集群内转发，但被节点当做QoS为0进行处理。实际转发给客户端的发布消息按照协议取发布QoS和订阅QoS两者较小值。

其他特性

集群间消息处理时不进行校验。集群间建立连接时发送所有本地先前的订阅关系。

设计思想

Mosquitto集群目标

水平扩展，避免单点故障，整个集群对外表现为一个完整的逻辑MQTT代理。

Pic1. Mosquitto集群概览

每次收到客户端的订阅消息时，通知其他节点此话题被订阅；每次收到客户端的发布消息时，把消息路由到正确的节点上。

Pic2. Mosquitto集群私有消息

节点与其他节点间只有一条逻辑通道，为了承载不同客户端的订阅/发布消息，引入了一些私有消息，包含客户端标识符、订阅标识符、时间戳等等，以帮助节点做出正确的路由和转发。

Pic3. Mosquitto集群内部消息流

转发规则：
转发来自于客户端的本节点新发生的订阅；当话题不再被任何客户端订阅时广播取消订阅消息；不转发任何系统话题。

Benchmark

使用krylovsk/mqtt-benchmark对集群进行简单的基准测试，集群吞吐率在3个以上节点时受限于客户端从而停止增长。

Pic4. Mosquitto集群平均吞吐率

n=10k表示每个客户端发送10000条消息, c=100表示总共启动100个客户端。
消息长度1000字节，QoS选取为2。

更详细的Tsung压测报告： https://github.com/hui6075/mosquitto-cluster/tree/master/benchmark

其他

Pic5. Mosquitto源码分析

mosquitto-cluster