Rabbitmq集群及使用keepalived实现rabbitmq高可用性集群

1 rabbitmq安装前准备工作

1.1 安装环境的说明

  rabbitmq集群在大并发的生产环境中都是由多台机器构建，使用keepalived+lvs实现负载均衡集群。在并发不是很大或者公司无法提供多态服务器实现keepalived+lvs高可用性集群时，但rabbitmq在整个服务架构中作用很重要，此时，可使用keepalived+rabbitmq实现高可用性集群。

1.2 安装环境准备

机型	DELL R630
CPU	E5+2.00GHz 2颗56核
内存	64 GB
硬盘	1TB
操作系统	Cnetos6.9
IP地址	主机1: 192.168.1.125  主机2:192.168.1.126  virtual IP:192.168.1.127

1.3 安装前环境准备

修改两台节点主机hosts文件使之能互相解析（两台主机上同时操作）

#修改主机hosts文件
cat >> /etc/hosts <<EOF
192.168.1.125 RABBITMQ01
192.168.1.126 RABBITMQ02
EOF

2 rabbitmq的安装

2.1 安装所依赖的软件

1、安装socat

#下载socat
wget http://dl.fedoraproject.org/pub/epel/6/x86_64//socat-1.7.2.3-1.el6.x86_64.rpm
#安装socat
yum localinstall -y socat-1.7.2.3-1.el6.x86_64.rpm

2、安装erlang

#安装erlang依赖软件
    yum -y install make ncurses-devel gcc gcc-c++ unixODBC unixODBC-devel openssl openssl-devel
#下载erlang
wget  http://erlang.org/download/otp_src_19.2.tar.gz
#解压
tar -xf otp_src_19.2.tar.gz
cd opt_src_19.2
#编译安装
./configure --prefix=/usr/local/erlang --enable-smp-support --enable-threads --enable-sctp --enable-kernel-poll --enable-hipe --with-ssl
make && make install
#验证erlang安装，输入erl,能进入erl并显示erl信息则说明安装成功
erl

erlang安装也可通过创建erlang yum仓库来安装

#下载仓库安装包
wget http://packages.erlang-solutions.com/erlang-solutions-1.0-1.noarch.rpm
#安装yum仓库
rpm -Uvh erlang-solutions-1.0-1.noarch.rpm
#生成缓存
yum makecache
#安装erlang
sudo yum install erlang

2.2 安装rabbitmq

#下载rabbitmq
wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.6.2/rabbitmq-server-3.6.2-1.noarch.rpm
#安装rabbitmq
rpm -ivh --nodeps rabbitmq-server-3.6.2-1.noarch.rpm

2.3 配置启动rabbitmq

1、启动rabbitmq

#启动rabbitmq 
service rabbitmq-server start    
#查看rabbitmq的运行状态
service rabbitmq-server status 

2、开启扩展管理，开启后可以通过网页登录管理rabbitmq，登录地址为服务器加端口号15672

#开启扩展管理
rabbitmq-plugins enable rabbitmq_management 
#开启扩展管理后，查看端口
ss -tanl | grep 5672    
LISTEN     0      128                       *:15672                  *:*     

3、安装队列延迟插件（具体可参考：http://blog.csdn.net/u014308482/article/details/53036770）

#进入插件存放目录
 cd /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.8/plugins/
#下载插件
Wget https://bintray.com/rabbitmq/community-plugins/rabbitmq_delayed_message_exchange
#启用插件
rabbitmq-plugins enable rabbitmq_delayed_message_exchange

3 rabbitmq的安装

3.1 创建集群前准备

将192.168.1.125的elrang cookie同步到192.168.1.126。

#在同步前停止rabbitmq
rabbitmqctl stop   
#同步elrang cookie
scp [email protected]:/var/lib/rabbitmq/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie 
#在同步完成后启动rabbitmq
service rabbitmq-server start  

3.2 创建集群

创建rabbitmq,只需在192.168.1.126操作，将192.168.1.126加入到集群即可。

#停止rabbitmq
rabbitmqctl stop_app
rabbitmqctl reset
#将rabbitmq加入集群，默认加入为磁盘节点，如果创建为内存节点需要在此命令后面加--ram，rabbitmq中至少需要一个磁盘节点
rabbitmqctl join_cluster rabbit@proxy-02
#创建完成后重启rabbitmq
rabbitmqctl start_app

3.3 验证集群

Rabbitmq集群创建完成后可通过命令“ rabbitmqctl cluster_status”来验证集群。

#查看rabbitmq集群状态信息 
rabbitmqctl cluster_status

3.4 为rabbitmq添加用户

由于为rabbitmq做了集群，所以在一台设备上的操作，会同步到其他节点，所以添加用户只需在一台节点上添加，会同步到其他节点。

#为rabbitmq添加用户并设置密码
rabbitmqctl  add_user admin admin  
#为创建的用户添加标签，即将用户加入到某个组中
rabbitmqctl set_user_tags admin administrator  

3.5 配置镜像队列

配置的RabbitMQ默认集群模式,不包括队列的高可用性。对rabbitmq基本操作可以复制到集群里的任何一个节点。队列内容并不会复制。虽然该集群模式解决了节点压力，但队列节点宕机直接会导致该队列无法应用，只能重启。所以要想在队列节点宕机或故障也能正常应用，就要复制队列内容到集群里的每个节点，即要创建镜像队列。

镜像队列的创建可通过rabbitmq管理界面或者使用命令来创建。

#创建镜像队列，将匹配到的队列在其他所有节点上做镜像
rabbitmqctl set_policy  ha-my2 "^a|^q"  '{"ha-mode":"all"}' 

4 keepalived的安装与配置

4.1 keepalived的安装

4.1.1安装前准备工作

#安装keepalived依赖包openssl-devel和kernel-devel
yum install openssl-devel
yum install kernel-devel
ln -s /usr/src/kernels/2.6.32-431.el6.x86_64 /usr/src/linux

4.1.2 安装keepalived

#下载keepalived
wget http://keepalived.org/software/keepalived-1.3.6.tar.gz
#解压keepalived
tar -xf keepalived-1.3.6.tar.gz
cd /keepalived-1.3.6
#编译keepalived
./configure --prefix=/usr/local/keepalived --with-kernel-dir=/usr/src/linux --enable-sha1
make && make install 

4.2 配置keepalived

4.2.1 创建keepalived的配置文件

#创建配置文件存放目录
mkdir /etc/keepalived
#将安装目录的配置文件拷贝到配置文件存放目录
cp /usr/local/keepalived/etc/keepalived/samples/keepalived.conf.virtualhost /etc/keepalived/keepalived.conf

4.2.1 修改keepalived的配置文件

1、修改rabbitmq master（192.168.1.125）的keepalived配置文件（/etc/keepalived/keepalived.conf）,修改后的内容如下：

vrrp_script chk_rabbitmq {
    script "/etc/keepalived/scripts01/rabbitmq_check.sh"
    interval 2
    }
vrrp_instance VI_1 {
    state MASTER
    interface eth0   
    virtual_router_id 52
    priority 100 
    authentication {
        auth_type PASS
        auth_pass 12345
    }
    track_script {
         chk_rabbitmq
    }
    virtual_ipaddress {
       192.168.1.127
    }
        notify_master /etc/keepalived/scripts/redis_master.sh
        #notify_backup /etc/keepalived/scripts/redis_backup.sh
        notify_fault  /etc/keepalived/scripts/redis_fault.sh
        notify_stop  /etc/keepalived/scripts/redis_stop.sh
}

2、修改 rabbitmq master （ 192.168.1.126 ）的 keepalived 配置文件（ /etc/keepalived/keepalived.conf ）,修改后的内容如下：

vrrp_script chk_rabbitmq {
    script "/etc/keepalived/scripts01/rabbitmq_check.sh"
    interval 2
    }
vrrp_instance VI_1 {
    state MASTER
    interface eth0   
    virtual_router_id 52
    priority 80 
    authentication {
        auth_type PASS
        auth_pass 12345
    }
    track_script {
         chk_rabbitmq
    }
    virtual_ipaddress {
       192.168.1.127
    }
        notify_master /etc/keepalived/scripts01/rabbitmq_master.sh
        #notify_backup /etc/keepalived/scripts01/rabbitmq_backup.sh
        notify_fault  /etc/keepalived/scripts01/rabbitmq_fault.sh
        notify_stop  /etc/keepalived/scripts01/rabbitmq_stop.sh
}

4.3 创建keepalived检测rabbitmq的脚本

#创建脚本文件存放目录
mkdir /etc/keepalived/scripts01

创建keepalived对rabbitmq状态监测脚本/etc/keepalived/scripts01/rabbitmq_check.sh（两台主机上均需要创建）。

脚本内容如下：

#!/bin/bash
k1=`/usr/bin/nmap -sS 127.0.0.1 -p 5672 | grep 5672 | awk '{printf $2}'`
if [ "$k1"x == "open"x ]; then
    echo ${k1}
    exit 0
else
    echo ${k1}
    exit 1
fi

4.4 创建记录rabbitmq状态切换的脚本

Rabbitmq各节点的数据是实时互相同步的，主节点发生故障及故障恢复后状态切换时，不涉及到主从的关系，所以在状态切换时只需将切换信息记录到日志即可。

1、切换为master时

#!/bin/bash 
 
LOGFILE="/var/log/keepalived-rabbitmq-state.log"
echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE 2>&1

2、切换为slave时（主节点上可以不用创建）

#!/bin/bash 
LOGFILE="/var/log/keepalived-rabbitmq-state.log"
 
echo "[backup]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1
 
echo "Run Backup ......" >> $LOGFILE

3、rabbitmq异常时

#!/bin/bash 
LOGFILE=/var/log/keepalived-rabbitmq-state.log
echo "[fault]" >> $LOGFILE
date >> $LOGFILE

4、Rabbitmq停止时

#!/bin/bash
GFILE=/var/log/keepalived-redis-rabbitmq.log
echo "[stop]" >> $LOGFILE
date >> $LOGFILE

4.5 重启rabbitmq使配置生效

#重启keepalived
/etc/init.d/keepalived restart
#查看keepalived进程信息
ps -ef | grep keepalived
#keepalived启动后master上配有virtual IP,查看master eth0网卡信息
ifconfig eth0