RabbitMQ distributed cluster architecture and high availability (HA)
(1) Function and principle
Purpose of designing a cluster
- Allow consumers and producers to continue running in the event of a RabbitMQ node crash
- Scale the throughput of message communication by adding more nodes
1 Cluster configuration
RabbitMQ can deploy distributed cluster systems in three ways: cluster, federation, shovel
cluster:
- Cross-network segment is not supported, used for LANs within the same network segment
- Can be dynamically increased or decreased at will
- Nodes need to run the same version of RabbitMQ and Erlang
federation: Applied to a wide area network, it allows a switch or queue on a single server to receive messages published to a switch or queue on another server, either a single machine or a cluster. A federation queue is similar to a one-way point-to-point connection, and messages are forwarded between federation queues any number of times until they are accepted by the consumer. Usually federation is used to connect to intermediate servers on the internet, which are used as subscriptions to distribute messages or work queues.
shovel: The connection method is similar to that of the federation, but it works at a lower level. Can be applied to wide area network.
2 Node Types
RAM node: The memory node stores all the metadata definitions of queues, exchanges, bindings, users, permissions and vhosts in memory, which has the advantage of making operations like switch and queue declarations faster.
Disk node: Metadata is stored on disk. A single-node system only allows disk-type nodes to prevent system configuration information from being lost when RabbitMQ is restarted.
Problem description: RabbitMQ requires at least one disk node in the cluster, and all other nodes can be memory nodes. When a node joins or leaves the cluster, it must notify at least one disk node of the change. If the only disk node in the cluster crashes, the cluster can still keep running, but no other operations (add, delete, modify, check) can be performed until the node recovers.
Solution: Set up two disk nodes, at least one of which is available and can hold metadata changes.
3 Erlang Cookie
Erlang Cookie is a key that ensures that different nodes can communicate with each other. To ensure that different nodes in the cluster communicate with each other, they must share the same Erlang Cookie. The specific directory is stored in /var/lib/rabbitmq/.erlang.cookie.
Description: This starts with the working principle of the rabbitmqctl command. The bottom layer of RabbitMQ is implemented through the Erlang architecture, so rabbitmqctl will start the Erlang node and use the Erlang system to connect to the RabbitMQ node based on the Erlang node. The correct connection process is required. Erlang cookie and node name, Erlang nodes exchange Erlang cookies for authentication.
4 Mirror queue
Function and principle
RabbitMQ's Cluster cluster mode is generally divided into two types, normal mode and mirror mode.
Normal mode: The default cluster mode. Take two nodes (rabbit01, rabbit02) as an example to illustrate. For Queue, the message entity only exists in one of the nodes, rabbit01 (or rabbit02). The two nodes of rabbit01 and rabbit02 only have the same metadata, that is, the structure of the queue. After the message enters the Queue of the rabbit01 node, when the consumer consumes from the rabbit02 node, RabbitMQ will temporarily transmit the message between rabbit01 and rabbit02, take out the message entity in A and send it to the consumer through B. Therefore, the consumer should try to connect to each node and get messages from it. That is, for the same logical queue, physical queues must be established on multiple nodes. Otherwise, regardless of whether the consumer is connected to rabbit01 or rabbit02, the exit will always be at rabbit01, which will cause a bottleneck. When the rabbit01 node fails, the rabbit02 node cannot obtain the unconsumed message entities in the rabbit01 node. If the message is persisted, the rabbit01 node must be restored before it can be consumed; if it is not persisted, the message will be lost.
Mirror mode: The queue that needs to be consumed is changed into a mirror queue, which exists on multiple nodes, so that the HA high availability of RabbitMQ can be achieved. The effect is that the message entity will actively synchronize between mirror nodes, instead of temporarily reading it when the consumer consumes data like the normal mode. The disadvantage is that synchronous communication within the cluster consumes a lot of network bandwidth.
The implementation mechanism
mirror queue realizes the high availability (HA) of RabbitMQ. The specific implementation strategy is as follows:
| ha-mode | ha-params | Function |
|---|---|---|
| all | null | The mirrored queue will be replicated across the cluster. When a new node is added, a copy is also made on this node. |
| exactly | count | The mirrored queue will be replicated count on the cluster. If the number of clusters is less than count, the queue will be replicated to all nodes. If the cluster is larger than Count, after a node crashes, the newly entered node will not do a new image. |
| nodes | node name | The mirror queue is replicated in node name. This will not trigger an error if the name is not one of the clusters. If no node in this node list is online, then the queue will be declared on the node the client is connected to. |
Examples are listed:
queue_args("x-ha-policy":"all") //定义字典来设置额外的队列声明参数
channel.queue_declare(queue="hello-queue",argument=queue_args)If you need to set a specific node (take rabbit@localhost as an example), add another parameter
queue_args("x-ha-policy":"nodes",
"x-ha-policy-params":["rabbit@localhost"])
channel.queue_declare(queue="hello-queue",argument=queue_args)You can view which master node has been synchronized through the command line
rabbitmqctl list_queue name slave_pids synchronised_slave_pids(2) RabbitMQ Cluster configuration
1 Single-machine multi-node deployment
After starting the RabbitMQ node, the default node name of the server is Rabbit and listening port 5672. If you want to start multiple nodes on the same machine, other nodes will fail to start because the node name and port conflict with the default ones. This can be achieved by setting environment variables as follows:
- First set up two nodes rabbit and rabbit_01 on the machine
rabbitmqctl stop //先停止运行节点，再进行集群部署
RABBITMQ_NODE_PORT=5672 RABBITMQ_NODENAME=rabbit //设置环境变量指定端口和节点名称
rabbitmq-server -detached //后台启动节点
RABBITMQ_NODE_PORT=5673 RABBITMQ_NODENAME=rabbit_01 //设置环境变量指定端口和节点名称
rabbitmq-server -detached //后台启动节点Or set it by adding the /etc/rabbitmq/rabbitmq-env.conf file:
NODE_PORT=5672
NODENAME=rabbit
NODE_PORT=5673
NODENAME=rabbit_01- Add the rabbit_01 node to the first cluster node rabbit
rabbitmqctl -n rabbit_01@localhost stop_app //停止rabbit_01节点的应用
rabbitmqctl -n rabbit_01@localhost join_cluster rabbit@localhost //将rabbit_01添加到集群节点rabbit中去
rabbitmqctl cluster_status //查看集群节点的状态
rabbitmqctl -n rabbit_01@localhost start_app //启动rabbit_01节点的应用//可以看到如下信息，说明节点添加成功，表明都是磁盘类型的节点
Cluster status of node rabbit@localhost ...
[{nodes,[{disc,[rabbit@localhost,rabbit_01@localhost]}]},
{running_nodes,[rabbit@localhost]},
{cluster_name,<<"rabbit@localhost">>},
{partitions,[]},
{alarms,[{rabbit@localhost,[]}]}]2 Multi-machine and multi-node deployment
Different from the single-machine multi-node situation, in a multi-machine environment, if you want to deploy multiple nodes in a cluster cluster, you need to pay attention to two aspects:
- Ensure that the nodes to be deployed are in the same LAN
- It is necessary to have the same Erlang cookie, otherwise communication cannot be performed. In order to ensure the complete consistency of the cookie, the method of copying from one node is adopted.
Environment introduction :
| RabbitMQ node | IP address | Operating mode | operating system |
|---|---|---|---|
| rabbitmqCluster | 186.16.195.24 | DISK | CentOS 7.0 - 64-bit |
| rabbitmqCluster01 | 186.16.195.25 | DISK | CentOS 7.0 - 64-bit |
| rabbitmqCluster02 | 186.16.195.26 | DISK | CentOS 7.0 - 64-bit |
Cluster deployment process :
- The LAN configuration
sets the same configuration information under /etc/hosts of the three nodes respectively
186.16.195.24 rabbitmqCluster
186.16.195.25 rabbitmqCluster01
186.16.195.26 rabbitmqCluster02- Set the Erlang Cookie of the same authentication between different nodes by copying
from the master node to maintain the consistency of the cookie
[root@rabbitmqCluster01]# scp /var/lib/rabbitmq/.erlang.cookie 186.16.195.25:/var/lib/rabbitmq
[root@rabbitmqCluster02]# scp /var/lib/rabbitmq/.erlang.cookie 186.16.195.26:/var/lib/rabbitmq- 使用 -detached运行各节点
rabbitmqctl stop
rabbitmq-server -detached - 查看各节点的状态
[root@rabbitmqCluster]#rabbitmqctl cluster_status
[root@rabbitmqCluster01]#rabbitmqctl cluster_status
[root@rabbitmqCluster02]#rabbitmqctl cluster_status- 创建并部署集群，以rabbitmqCluster01节点为例：
[root@rabbitmqCluster01]#rabbitmqctl stop_app
[root@rabbitmqCluster01]#rabbitmqctl join_cluster rabbit@rabbitmqCluster
[root@rabbitmqCluster01]#rabbitmqctl start_app- 查看集群状态
[root@rabbitmqCluster]#rabbitmqctl cluster_statusRabbitMQ负载均衡配置
前言：从目前来看，基于RabbitMQ的分布式通信框架主要包括两部分内容，一是要确保可用性和性能，另一个就是编写当节点发生故障时知道如何重连到集群的应用程序。负载均衡就是解决处理节点的选择问题。
安装HAProxy
选择开源的HAProxy为RabbitMQ集群做负载均衡器，在CentOS 7.0中安装HAProxy。
- 安装epel
rpm -ivh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm//- 安装HAProxy
yum -y install haproxy- 配置HAProxy
cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak
vim /etc/haproxy/haproxy.cfg- 添加配置信息
listen rabbitmq_local_cluster 127.0.0.1:5670 //前段IP，供product和consumer来进行选择，由于5672端口已经默认使用，这里选择5670端口
mode tcp //负载均衡选项
balance roundrobin //轮询算法将负载发给后台服务器
server rabbit 127.0.0.1:5672 check inter 5000 rise 2 fall 3//负载均衡中的集群节点配置，这里选择的rabbit节点
listen private_monitoring :8100
mode http
option httplog
stats enable
stats uri /stats
stats refresh 60s- 访问http://localhost:8100/stats就可以看到具体的控制界面。