RabbitMQ learning (6) "RabbitMQ cluster and high availability"

1. Why do clusters

The cluster is mainly used to achieve high availability and load balancing .

• High availability : If some MQ servers in the cluster are unavailable, the client can also connect to other MQ servers.
• Load balancing : In a high concurrency scenario, a single MQ server can process limited messages and can be distributed to multiple MQ servers.

2. How does RabbitMQ support clusters

The application as a cluster needs to face the problems of data synchronization and communication . Because Erlang is inherently distributed, RabbitMQ naturally supports clusters and does not need to introduce ZK or databases to achieve data synchronization.

RabbitMQ uses /var/lib/rabbitmq/.erlang.cookie to verify identity, which needs to be consistent on all nodes.

3.RabbitMQ node type

There are two types of nodes in the cluster, one is a disk node (Disc Node) and the other is a memory node (RAM Node) .

• Disk node : Put metadata (including queue name attribute, switch type name attribute, binding, vhost) (not including message data) on disk.
• Memory node : Put metadata in memory.
  The memory node will store the address of the disk node on the disk (otherwise there will be no way to synchronize data after restarting). If it is a persistent message, it will be stored in both memory and disk.

At least one disk node in the cluster is required to persist metadata, otherwise, when all memory nodes crash, there will be no way to synchronize metadata. If the type is not specified, the default is the disk node.

We generally connect the application to the memory node (fast reading and writing), and the disk node is used for backup.
The cluster communicates in pairs through port 25672 , and the firewall port needs to be opened.
It should be noted that the RabbitMQ cluster cannot be built on the WAN unless plug-ins such as federation or shovel are used (this is not necessary, the cluster is in the same computer room).
Cluster configuration steps:

1. Configure hosts
2. Synchronize erlang.cookie
3. Join cluster

4. Cluster mode

RabbitMQ has two cluster modes: normal cluster mode and mirror queue mode .

4.1 Common cluster mode

In the normal cluster mode, different nodes will only synchronize metadata with each other.

Question: Why not directly copy the content (message) of the queue on all nodes?
Mainly because of the network overhead of storing and synchronizing data, if all nodes store the same data, it will not achieve linear increase in performance And the purpose of storage capacity (heap machine).

If the producer is connected to node 3, and the message is routed to queue 1 through switch A, the message will eventually be forwarded to node 1 for storage, because the content of queue 1 is only on node 1.
In the same way, if the consumer connection is node 2, and the message needs to be pulled from queue 1, the message will be forwarded from node 1 to node 2. Other nodes act as a route, similar to pointers.

The normal cluster mode cannot guarantee the high availability of the queue , but can only achieve load, because the content of the queue will not be copied. If the node fails, the related queue will be unavailable, so we need the second cluster mode.

4.2 Mirror queue mode

In the mirror queue mode, the message content will be synchronized between the mirror nodes, and the availability is higher. However, there are some side effects, the system performance will be reduced, and the cost of synchronization will be higher when there are too many nodes.

5. High availability

After the cluster is successfully built, if there are multiple memory nodes, which memory node should the producer and consumer connect to? If we choose the server to be used according to a certain strategy in our code, then every place must be modified. There will be a lot of duplication in the client code, and it is troublesome to modify.

Therefore, a load balancing component (such as HAProxy, LVS, Nignx) is required, and the load component is used for routing. At this time, you only need to connect to the IP address of the load component.

5.1 Four-tier load

Four-layer load: It works on the fourth layer of the OSI model, the transport layer (TCP is in the fourth layer), which is forwarded according to the IP port (LVS supports four-layer load). RabbitMQ is TCP port 5672.

5.2 Seven-layer load

Seven-layer load: working in the seventh layer, application layer (HTTP is located in the seventh layer). It can be allocated to the back-end server according to the requested resource type (Nginx supports seven-layer load; HAProxy supports four-layer and seven-layer load).

However, what if the components of this load also hang? The client cannot connect to any MQ server. Therefore, the load software itself needs to be a cluster. The new problem comes again. If there are two loadable software, which one should the client connect to?

Load on top of the load? It's in an endless loop. At this time we have to change our thinking.

5.3 VRRP protocol (Virtual Router Redundancy Protocol)

We should need such a component:

1. It has a routing (load) function, which can monitor the status of nodes in the cluster (such as monitoring HAProxy). If a node is abnormal or fails, it will be removed.
2. In order to improve availability, it can also deploy multiple services, but there is only one automatically elected MASTER server (called the master router), which is achieved by broadcasting heartbeat messages.
3. The MASTER server provides a virtual IP externally and provides various network functions. That is, whoever seizes the IP will provide network services to the outside world. The application side only needs to connect to this IP.

This protocol is called VRRP protocol (Virtual Router Redundancy Protocol), this component is Keepalived , it has Load Balance and High Availability functions.

6. HAproxy load + Keepalived high availability

Let's take a look at how to use HAProxy and Keepalived to achieve high availability of RabbitMQ (MySQL, Mycat, Redis similar).

Planning:
Memory node 1: 192.168.1.1
Memory node 2: 192.168.1.2
Disk node: 192.168.1.3
VIP: 192.168.77.77

1. We planned two memory nodes and one disk node. All nodes synchronize data by mirroring queues. Memory nodes are used for application access, and disk nodes are used to persist data.
2. In order to achieve the load on the two memory nodes, we installed two HAProxy, listening on two ports of 5672 and 15672.
3. Install two Keepalived, one active and one standby. Two Keepalived preempt a VIP192.168.77.77. Whoever seizes this VIP, the application will connect to whom to execute the load on the MQ.
   In this case, our Keepalived hangs a node, which has no effect, because BACKUP will become a MASTER, preempting VIP. HAProxy hangs a node, there is no impact, our VIP will automatically route the available HAProxy services. RabbitMQ suspends a node, which has no effect, because HAProxy will automatically load the available nodes.