High-availability RabbitMQ cluster construction and principle analysis

Preface

Any service, if only standalone deployment, the performance always have an upper limit, RabbitMQis no exception, when a single RabbitMQtime service the ability to handle messages reach the bottleneck, can be achieved through high availability and load balancing cluster.

How much do you know about the RabbitMQ cluster

Typically, in each cluster we have a service called a node in RabbitMQthe cluster, the node type can be divided into two types:

• Memory node: Metadata is stored in memory. In order to synchronize data after restart, the memory node will store the address of the disk node in the disk. In addition, if the message is persisted, it will also be stored in the disk, because the memory node reads and writes fast, and the general customer The end will connect to the memory node.
• Disk node: Metadata is stored in disk (default node type), and at least one disk node needs to be guaranteed. Otherwise, once it goes down, the data cannot be restored, and the purpose of high availability of the cluster cannot be achieved.

PS: Metadata refers to basic information including queue name attributes, switch type name attributes, binding information, vhost, etc. It does not include message data in the queue.

RabbitMQ There are two main modes of clusters in, the normal cluster mode and the mirror queue mode.

Normal cluster mode

In the normal cluster mode, each node in the cluster will only synchronize metadata with each other, that is, message data will not be synchronized. So the question becomes, if we are connected to the Anode, but the message they are stored in the Bnode and how to do it?

Regardless of whether it is a producer or a consumer, if the connected node does not store queue data, it will be forwarded internally to the node storing the queue data for storage. Although it can be forwarded internally, because the message is only stored in one node, if the node goes down, will the message be gone? This problem does exist, so this common cluster mode does not achieve the purpose of high availability.

Mirror queue mode

In the mirror queue mode, not only the metadata will be synchronized between nodes, but the message content will also be synchronized between the mirror nodes, and the availability is higher. While this solution improves availability, it also brings network overhead between synchronized data, which will affect performance to a certain extent.

RabbitMQ cluster construction

Let's try together to build a RabbitMQcluster:

1. If you had to start before the stand-alone version, then delete the old data rm -rf /var/lib/rabbitmq/mnesiaor delete the installation in the directory var/lib/rabbitmq/mnesia, my machine is installed in the installation directory, the command is executed rm -rf /usr/local/rabbitmq_server-3.8.4/var/lib/rabbitmq/mnesia/.

2. Next you need to start the following three commands to start three different port number of RabbitMQservices, in addition to specifying RabbitMQwhen the service ports also need to specify additional port back office systems, and must specify the nodeprefix name, because the name of the cluster node is to communicate , So the node name must be unique, the default node name is rabbit@hostname, the following command indicates that the prefix is ​​specified:

RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit1 rabbitmq-server -detached
RABBITMQ_NODE_PORT=5673 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15673}]" RABBITMQ_NODENAME=rabbit2 rabbitmq-server -detached
RABBITMQ_NODE_PORT=5674 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15674}]" RABBITMQ_NODENAME=rabbit3 rabbitmq-server -detached

After starting to enter the /usr/local/rabbitmq_server-3.8.4/var/lib/rabbitmq/mnesia/catalog, find creating a 3node information:

Also through ps -ef | grep rabbitcan also be found three service process is started.

3. The three services started now have no connection with each other. Now we need to use one of the nodes as the master node, and then the other two nodes need to join the master node to form a cluster service. It should be noted that before joining the cluster, it needs to be reset. Node information, that is, nodes with data are not allowed to join the cluster.

//rabbit2 节点重置后加入集群
rabbitmqctl -n rabbit2 stop_app
rabbitmqctl -n rabbit2 reset
rabbitmqctl -n rabbit2 join_cluster --ram rabbit1@`hostname -s` //--ram 表示这是一个内存节点
rabbitmqctl -n rabbit2 start_app

rabbitmqctl -n rabbit3 stop_app
rabbitmqctl -n rabbit3 reset
rabbitmqctl -n rabbit3 join_cluster --disc rabbit1@`hostname -s` //--disc表示磁盘节点（默认也是磁盘节点）
rabbitmqctl -n rabbit3 start_app

4. After a successful, execute the command rabbitmqctl cluster_statusquery node rabbit1status, you can see the following figure, two disk nodes a memory node:

5. It should be noted that the cluster started here is just the default normal cluster. If you want to configure a mirrored cluster, you need to execute the following command:

rabbitmqctl -n rabbit1 set_policy ha-all "^" '{"ha-mode":"all"}'

Here RabbitMQa cluster even if the build is complete, but should be noted that, because here is a stand-alone version, so do not consider .erlang.cookiethe file consistent.

High-availability cluster based on HAProxy + Keepalived

If a RabbitMQcluster, there are multiple memory nodes, where a node is connected we should do? If this strategy of choice is done on the client side, there will be great drawbacks. The most serious one is that the client code must be modified every time the cluster is expanded. Therefore, this method is not very desirable, so we are deploying the cluster. when you need an intermediate proxy component, this component to enable service monitoring and forwarding, such as Redisthe Sentinel(Sentinel) cluster mode, you can monitor sentinel Redisnode and failover.

In RabbitMQa cluster, by Keepalivedand HAProxytwo components to achieve high availability and load balancing cluster.

HAProxy

HAProxyIt is an open source, high-performance load balancing software, which can also be used as load balancing software nginx, lvsetc. HAproxySupport 7layer load balancing and 4layer load balancing.

Load balancing

So-called 7layer load balancing and 4layer load balancing for the OSIpurposes of the model, shown below is a OSIcommunication model:

We see the figure above, the first 7layer corresponds to the application layer, the second 4layer corresponds to the transport layer. Commonly used as load-balancing software nginxgenerally work in the first 7layer, lvsgenerally work (Linux Virtual Server) in the first 4layer.

• 4 Layer load:

4Layer load uses NAT(Network Address Translation) technology, namely: Network Address Translation. When receiving the client request, by modifying the source of the data package IPand the port, and then forward the packet to the corresponding target server. 4Layer load balancing can only forward requests based on the destination address and source address in the message, and cannot determine or modify the specific types of requested resources.

• 7 Layer load:

According to the resource path requested by the client, it is forwarded to different target servers.

Highly available HAProxy

HAProxyAlthough load balancing is achieved, if only one is deployed HAProxy, there is a risk of downtime. Once HAProxydown, it will cause the entire cluster is unavailable, so we need to HAProxyalso implement a cluster, if HAProxyalso realized the cluster, the client should connect Which services? The problem seems to be back to the beginning, stuck in an infinite loop...

Keepalived

In order to achieve HAProxyhigh availability, the need to introduce a Keepalivedcomponent, Keepalivedcomponent mainly has the following characteristics:

• With load function, it can monitor the status of nodes in the cluster. If a node in the cluster goes down, failover can be realized.
• Cluster itself can also be achieved, but only one masternode.
• masterForeign node will provide a virtual IPapplication side only need to connect this one IPon the line. It can be understood as the cluster HAProxynodes will also compete for the virtual IP, which node to the competition, which will be served by a node.

VRRP protocol

VRRPThe protocol is the Virtual Router Redundancy Protocol (Virtual Router Redundancy Protocol). KeepalivedProvided a virtual IPmechanism belongs to VRRP, it is to avoid the emergence of a fault-tolerant protocol router single point of failure.

to sum up

This paper describes the RaabbitMQrelevant knowledge clusters, and compared the difference between ordinary cluster and cluster mirror, and finally through practice to build a RabbitMQcluster, but also introduces some drawbacks common clusters can be combined HAProxyand Keepalivedcomponents to achieve true high availability distributed Cluster service.