[System Operation and Maintenance] Centos deploys Haproxy+Keepalived+RabbitMQ high-availability cluster

1.RabbitMQ high availability cluster solution

(1) The role of RabbitMQ in building a cluster: improve availability, reliability and processing capabilities, and ensure that the system provides efficient messaging services

• High availability: Through clustering, even if one node fails, other nodes can still continue to provide services, ensuring reliable delivery of messages.
• Load balancing: The cluster can evenly distribute the message load to multiple nodes, improving the system's processing capabilities while avoiding overloading a single node.
• Fault recovery: If a node in the cluster fails, the node can be removed from the cluster through fault detection and automatic recovery mechanisms, and its load can be redistributed to other nodes to ensure the continuity and availability of the entire system.
• Scalability: When the generation and consumption of messages gradually increases, more nodes can be added to the cluster to expand the system's processing capabilities to cope with the growing load.

(2) Common cluster solutions

• RabbitMQ ordinary cluster
  • The default cluster mode, for example, there are nodes node1 and node2. The two nodes are ordinary clusters, but they only have the same metadata, that is, the structure of switches and queues;
  • The message only exists in one of the nodes. If message A is stored in the node1 node, when the consumer connects to the node1 node to consume the message, it can be taken out directly;
  • But if the consumer is connected to other nodes, rabbitmq will take the message in the queue from the node where it is stored, and forward it through the connecting node before sending it to the consumer;
  • If node1 fails, node2 cannot obtain unconsumed messages stored in node1;
  • If node1 fails after persistence, you need to wait for node1 to recover before normal consumption can occur. If node1 fails after persistence, the message will be lost;
  • In this case, high availability cannot be achieved, and communication between nodes will increase to obtain messages, resulting in performance bottlenecks;
  • This mode is more suitable for scenarios where messages do not need to be persisted, such as log transmission queues;
• RabbitMQ mirror cluster
  • The queue is made into a mirror queue so that each queue exists on multiple nodes;
  • The biggest difference from a normal cluster is that [the message of the queue] will be synchronized between the nodes of the cluster;
  • And it is not pulled temporarily when the consumer obtains the data, but the ordinary cluster temporarily pulls the corresponding data from the storage node;
  • High availability is achieved, and normal consumption will not be affected after some nodes hang up;
  • It can guarantee that 100% of messages will not be lost. It is recommended to use 3 odd nodes and combine LVS+Keepalive or Haproxy for IP drift to prevent single points of failure;
  • Due to the excessive number of messages in the mirror queue mode, a large number of message synchronization will also increase network bandwidth overhead;
  • Suitable for projects with relatively high high availability requirements. If there are too many nodes, the performance will be further affected;

2.RabbitMQ high-availability cluster architecture

Two RabbitMQ disk nodes and one RabbitMQ memory node form a built-in cluster. The reason why two disk nodes are used is to prevent queues and switches from being rebuilt after the only disk node hangs up. Use HAProxy as a load balancer for the RabbitMQ cluster. In order to prevent HAProxy single points of failure, Keepalived is used to make two HAProxy nodes one master and one backup. When an application uses VIP (virtual IP) to access the HAProxy service, it connects to the HAProxy of the master (Master) by default. When the HAProxy on the master (Master) fails, the VIP will drift to the backup machine (Backup), and it will connect to the backup machine (Backup). ) on the HAProxy service.

VRRP protocol
The VRRP protocol is the Virtual Router Redundancy Protocol. The virtual IP mechanism provided in Keepalived belongs to VRRP, which is a fault-tolerant protocol to avoid single points of failure in routers.

3. RabbitMQ high-availability cluster construction

(1) Machine preparation

IP address	CPU name	Remark
192.168.140.101	mq01	Disk node RabbitMQ
192.168.140.102	mq02	Memory node keepalived+haproxy+RabbitMQ
192.168.140.103	mq03	Memory node keepalived+haproxy+RabbitMQ

(2) Docker deployment of RabbitMQ on three machines

Parameter Description:

--hostname：自定义Docker容器的hostname
--privileged=true：使用该参数，container内的root拥有真正的root权限，否则容器出现permission denied
-v：宿主机和容器路径映射
RABBITMQ_ERLANG_COOKIE：Erlang Cookie 值，同一个集群内该值必须相同，相当于不同节点之间通讯的密钥
--add-host：修改容器内部的hosts，添加的集群服务器都需要添加进来
--restart=unless-stopped docker 容器重启后重启MQ

• Node one

docker run  -d --hostname rabbitmq1 --add-host=rabbitmq2:192.168.140.102 --add-host=rabbitmq3:192.168.140.103 --restart=unless-stopped --name rabbitmq1 --net host -p 15672:15672  -p 5672:5672 -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=admin -e RABBITMQ_ERLANG_COOKIE='rabbit_mq'  rabbitmq:management

• Node two

docker run  -d --hostname rabbitmq2 --add-host=rabbitmq1:192.168.140.101 --add-host=rabbitmq3:192.168.140.103 --restart=unless-stopped --name rabbitmq2 --net host -p 15672:15672  -p 5672:5672 -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=admin -e RABBITMQ_ERLANG_COOKIE='rabbit_mq'  rabbitmq:management

• Node three

docker run  -d --hostname rabbitmq3 --add-host=rabbitmq1:192.168.140.101 --add-host=rabbitmq2:192.168.140.102 --restart=unless-stopped --name rabbitmq3 --net host -p 15672:15672  -p 5672:5672 -e RABBITMQ_DEFAULT_USER=admin -e RABBITMQ_DEFAULT_PASS=admin -e RABBITMQ_ERLANG_COOKIE='rabbit_mq'  rabbitmq:management

• Node one configures the cluster.

docker exec -it rabbitmq1 bash
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
exit

• Node 2 joins the cluster. –ram is added in memory mode. If this parameter is ignored, it defaults to a disk node.

docker exec -it rabbitmq2 bash
rabbitmqctl stop_app
rabbitmqctl join_cluster --ram rabbit@rabbitmq1
rabbitmqctl start_app
exit

• Node three joins the cluster. –ram is added in memory mode. If this parameter is ignored, it defaults to a disk node.

docker exec -it rabbitmq3 bash
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster --ram rabbit@rabbitmq1
rabbitmqctl start_app
exit

• Check the cluster node status. The configuration has started 3 nodes, 1 disk node and 2 memory nodes.

rabbitmqctl cluster_status

• Memory node (ram): Put all metadata in the memory. For memory nodes, as long as the service is restarted, all data on the node will be lost.
• Hard disk node (disc): Put all metadata on the hard disk, so if the service is restarted, the data will still exist.

(3) Build a mirror cluster

• We built a normal cluster earlier. If the disk node hangs up, the data will be lost if persistence is not enabled, and other nodes will not be able to obtain the message.
• Therefore, our cluster solution needs to be further transformed into a mirror mode cluster .
• The policy of rabbitmq is used to control and modify the vhost, queue and Exchange replication behavior of the cluster. It is to set which Exchange or queue data needs to be copied and synchronized, and how to copy and synchronize.

Configuration steps: rabbitmq management page—> Admin—> Policies—> Add/update a policy

The policy will synchronize exchange and queue data in the same VirtualHost

• name: Custom policy name
• Pattern: ^ matching character, which means matching all
• Definition: ha-mode=all is the matching type, divided into 3 modes: all (representing all queues)
• ha-mode: Specifies the mode of the mirror queue, select one of the following
  • all: Indicates mirror synchronization on all nodes in the cluster (this parameter is generally used)
  • exactly: Indicates mirror synchronization on the specified number of nodes. The number of nodes is specified by ha-params.
  • nodes: Indicates mirror synchronization on the specified node. The node name is specified through ha-params.
  • ha-sync-mode: Mirror message synchronization mode automatic (automatic), manually (manual)

The strategy will synchronize the exchange and queue data in the same VirtualHost, create a queue on MQ node one, and synchronize to each node.

Cluster restart sequence:

• The order of cluster restarts is fixed and reversed
• Startup sequence: disk node => memory node
• Shutdown sequence: memory node => disk node
• The last shutdown must be the disk node, otherwise it may easily cause abnormal situations such as cluster startup failure and data loss.

4.Haproxy high availability cluster construction

(1) Architectural pain points of MQ cluster

• A RabbitMQ high-availability cluster, producer and consumer has been built. Multiple node configuration information needs to be written in the business code.

• If a node fails, or multiple nodes are added, multiple business programs need to be restarted to update configuration information.

• You can use Haproxy to act as a load balancer in RabbitMQ, providing functions such as load balancing, failover, reliability and availability enhancement, and dynamic scaling.

(2) Create configuration file

Create the directories /usr/local/haproxy on nodes 102 and 103 respectively.

Create configuration file haproxy.cfg

#增加下面配置文件
global
  log 127.0.0.1 local0 info
  maxconn 10240
  daemon
  
defaults
  log global
  mode http
  timeout connect 3000 
  timeout client 3000
  timeout server 3000
  timeout check 2000

listen  admin_stats
    #haproxy监控界面的访问的IP和端口    
    bind  0.0.0.0:8081
    mode        http
    stats uri   /stats
    stats realm     Global\ statistics
    #haproxy登陆帐户信息
    stats auth  admin:admin

listen rabbitmq_admin
    #rabbit的监控页面
    bind    0.0.0.0:8082
    server  rabbit_admin1 192.168.140.101:15672
    server  rabbit_admin2 192.168.140.102:15672
    server  rabbit_admin3 192.168.140.103:15672

listen haproxy
  #对外提供的端口，spring boot连接mq的端口
  bind 0.0.0.0:8091
  option tcplog
  mode tcp
  balance roundrobin
  server rabbitmq1 192.168.140.101:5672 check inter 5000 rise 2 fall 2
  server rabbitmq2 192.168.140.102:5672 check inter 5000 rise 2 fall 2
  server rabbitmq3 192.168.140.103:5672 check inter 5000 rise 2 fall 2

Detailed explanation of configuration files

全局配置部分（global）：
  - 设置日志输出目标为本地地址 `127.0.0.1` 的 `local0` 日志设备，并设置日志级别为 `info`。
  - 设置最大连接数为 `10240`。
  - 启用后台运行模式。
默认配置部分（defaults）：
  - 设置日志输出为全局设定的日志设备。
  - 指定工作模式为HTTP模式。
  - 设置连接超时时间为 `3000` 毫秒。
  - 设置客户端超时时间为 `3000` 毫秒。
  - 设置服务器超时时间为 `3000` 毫秒。
  - 设置健康检查超时时间为 `2000` 毫秒。
`admin_stats` 监听器：
  - 监听 `0.0.0.0:8081` 地址和端口。
  - 使用HTTP模式。
  - 设置监控统计信息的路径为 `/stats`。
  - 设置统计信息页面的显示名称为 `Global statistics`。
  - 设置访问统计信息页面的认证用户名和密码为 `admin:xdclass.net168`。
`rabbitmq_admin` 监听器：
  - 监听 `0.0.0.0:8082` 地址和端口。
  - 配置了三个RabbitMQ管理节点。
  - 每个节点的名称和地址:端口为 `rabbit_admin1 120.24.7.58:15672`、`rabbit_admin2 120.24.7.58:15673` 和 `rabbit_admin3 120.24.7.58:15674`。
`haproxy` 监听器：
  - 监听 `0.0.0.0:5666` 地址和端口。
  - 启用TCP日志记录。
  - 使用TCP模式。
  - 使用轮询算法进行负载均衡。
  - 配置了三个RabbitMQ节点。
  - 每个节点的名称和地址:端口为 `rabbitmq1 120.24.7.58:5672`、`rabbitmq2 120.24.7.58:5673` 和 `rabbitmq3 120.24.7.58:5674`。
  - 设置了健康检查的参数：每5000毫秒检查一次，如果连续2次检查成功就认为节点上线，如果连续2次检查失败则认为节点宕机。

Start haproxy on both machines respectively

docker run -d -i --name haproxy-rabbitmq -p 8081:8081 -p 8091:8091 -p 8082:8082 -v /usr/local/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg --privileged=true -u root  haproxy:latest 

access path

• Haproxy: http://192.168.140.103:8081/stats,http://192.168.140.102:8081/stats
• RabbitMQ control console: http://192.168.140.102:8082, http://192.168.140.103:8082

5.Keepalived high availability cluster construction

(1) Nodes 102 and 103 download Keepalived respectively.

#yum安装
yum -y install keepalived

(2) Edit configuration file

! Configuration File for keepalived
global_defs {
    
    
   router_id LVS_DEVEL # 设置lvs的id，在一个网络内应该是唯一的
   enable_script_security #允许执行外部脚本
}
#配置vrrp_script，主要用于健康检查及检查失败后执行的动作。
vrrp_script chk_real_server {
    
    
    weight -5
    fall 3
    rise 2
    user root
}
vrrp_instance VI_1 {
    
    
		#配置节点是master还是backup，102配置MASTER，103配置BACKUP
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    #配置虚拟IP
    virtual_ipaddress {
    
    
        192.168.140.100
    }
}
virtual_server 192.168.140.100 8082 {
    
    
    # 设置健康检查时间，单位是秒
    delay_loop 6
    # 设置负载调度的算法为rr
    lb_algo rr
    # 设置LVS实现负载的机制，有NAT、TUN、DR三个模式
    lb_kind NAT
    # 会话保持时间
    persistence_timeout 50
   #指定转发协议类型(TCP、UDP)
    protocol TCP
    # 指定real server1的IP地址
    real_server 192.168.140.102 8082 {
    
    
        weight 1
    }
    real_server 192.168.140.103 8082 {
    
    
        weight 1
    }
}

(3) Start keepalived and access the MQ cluster

systemctl start keepalived.service

Ok, it is found that the mq node has been successfully accessed through keepalived.

When we stop the keepalived of the 102 master node, the virtual IP will drift to the 103 node.

OK, now the RabbitMQ high-availability cluster has been set up.

Remember to give the blogger three in a row! ! !