RocketMQ 异步复制集群部署安装

防伪码：他日若遂凌云志，敢笑黄巢不丈夫。

1.RacketMQ多种部署模式

RocketMQ Broker集群部署方式主要有以下几种：（Slave 不可写，但可读）
1.1.单个Master
这种方式风险较大，一旦Broker 重启或者宕机时，会导致整个服务不可用，不建议线上环境使用。
1.2.多Master模式
一个集群无 Slave，全是 Master，例如 2 个 Master 或者 3 个 Master。

优点：配置简单，单个Master 宕机或重启维护对应用无影响，在磁盘配置为 RAID10 时，即使机器宕机不可恢复情况下，由与 RAID10 磁盘非常可靠，消息也不会丢（异步刷盘丢失少量消息，同步刷盘一条不丢）。性能最高。

缺点：单台机器宕机期间，这台机器上未被消费的消息在机器恢复之前不可订阅，消息实时性会受到受到影响。

# 先启动 NameServer，例如机器 IP 为：”10.162.23.70:9876;10.162.23.71:9876”
# nohup ./bin/mqnamesrv >/dev/null 2>&1 & 
# 在机器 A，启动第一个 Master
# nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-noslave/broker-a.properties >/dev/null 2>&1 & 
# 在机器 B，启动第二个 Master
# nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-noslave/broker-b.properties >/dev/null 2>&1 & 
1.3.多Master多Slave模式，异步复制
每个 Master 配置一个 Slave，有多对Master-Slave，HA 采用异步复制方式，主备有短暂消息延迟，毫秒级。

优点：即使磁盘损坏，消息丢失的非常少，且消息实时性不会受影响，因为 Master 宕机后，消费者仍然可以从 Slave 消费，此过程对应用透明。不需要人工干预。性能同多 Master 模式几乎一样。

 缺点：Master 宕机，磁盘损坏情况，会丢失少量消息。

# 先启动 NameServer，例如机器 IP 为：”10.162.23.70:9876;10.162.23.71:9876”
# nohup ./bin/mqnamesrv >/dev/null 2>&1 & 
# 在机器 A，启动第一个 Master
# nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 & 
# 在机器 B，启动第二个 Master
# nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-2s-async/broker-b.properties >/dev/null 2>&1 & 
#在机器 C，启动第一个 Slave
nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-2s-async/broker-a-s.properties >/dev/null 2>&1 & 
# 在机器 D，启动第二个 Slave
# nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-2s-async/broker-b-s.properties >/dev/null 2>&1 & 
1.4.多Master多Slave模式，同步双写
每个 Master 配置一个 Slave，有多对Master-Slave，HA 采用同步双写方式，主备都写成功，向应用返回成功。

优点：数据与服务都无单点，Master宕机情况下，消息无延迟，服务可用性与数据可用性都非常高

缺点：性能比异步复制模式略低，大约低 10%左右，发送单个消息的 RT 会略高。目前主宕机后，备机不能自动切换为主机，后续会支持自动切换功能。

# 先启动 NameServer，例如机器 IP 为：”10.162.23.70:9876;10.162.23.71:9876”
# nohup ./bin/mqnamesrv >/dev/null 2>&1 & 
# 在机器 A，启动第一个 Master
# nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 & 
# 在机器 B，启动第二个 Master
# nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-2s-sync/broker-b.properties >/dev/null 2>&1 & 
# 在机器 C，启动第一个 Slave
# nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 & 
# 在机器 D，启动第二个 Slave
# nohup ./bin/mqbroker -n “10.162.23.70:9876;10.162.23.71:9876” -c ./conf/2m-2s-sync/broker-b-s.properties >/dev/null 2>&1 & 

注：
以上 Broker 与 Slave 配对是通过指定相同的brokerName 参数来配对，Master 的 BrokerId 必须是 0，Slave 的BrokerId 必须是大与 0 的数。另外一个 Master 下面可以挂载多个 Slave，同一 Master 下的多个 Slave 通过指定不同的 BrokerId 来区分。
除此之外，nameserver也需要集群。我们的环境采用多nameserver多master多slave异步复制模式

2.环境准备

角色	主机名	IP	port	OS	配置
Nameserver1	INVOICE-HX-01	10.162.23.70	9876	Redhat7.2	16C32G100G
Nameserver2	INVOICE-HX-02	10.162.23.71	9876	Redhat7.2	16C32G100G
Master1	INVOICE-HX-01	10.162.23.70	10911 10922	Redhat7.2	16C32G100G
Master2	INVOICE-HX-02	10.162.23.71	10911 10922	Redhat7.2	16C32G100G
Slave1	INVOICE-HX-03	10.162.23.72	10911 10922	Redhat7.2	16C32G100G
Slave2	INVOICE-HX-04	10.162.23.73	10911 10922	Redhat7.2	16C32G100G

moosefs安装前准备：
# RocketMQ软件包：alibaba-rocketmq-3.2.6.tar.gz
# JDK 1.7.79
# SELINUX ：disable
# 安装路径：/opt/server/rocketmq
# 日志路径：/opt/var/logs/rocketmq

# 所有主机配置hosts
# cat /etc/hosts
10.162.23.70 INVOICE-HX-01
10.162.23.71 INVOICE-HX-02
10.162.23.72 INVOICE-HX-03
10.162.23.72 INVOICE-HX-04

注：hosts解析必须是真实的hostname

3.RocketMQ多M多异步复制集群安装

3.1.JDK安装
# 所有角色上安装
# 上传jdk-7u79-linux-x64.gz
# mkdir /opt/server -p
# tar zvxf jdk-7u79-linux-x64.gz -C /opt/server

# 修改 /etc/profile
# vi /etc/profile
# jdk
JAVA_HOME=/opt/server/jdk1.7.0_79
CLASSPATH=$JAVA_HOME/jre/lib/rt.jar:.
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME PATH CLASSPATH

# source /etc/profile
# 检查java版本：
# java -version
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

3.2.RocketMQ 解压安装
# 所有角色上安装
# 上传alibaba-rocketmq-3.2.6.tar.gz
# tar zxvf alibaba-rocketmq-3.2.6.tar.gz
# mv alibaba-rocketmq  /opt/server/rocketmq
#安装包bin下文件没有执行权限
# chmod +x /opt/server/rocketmq/bin/*

# 创建数据存储目录
# mkdir -p /opt/data/rocketmq/data/store/commitlog
# 创建日志目录
# mkdir -p /opt/var/logs/rocketmq/
# 修改日志目录
# cd /opt/server/rocketmq/conf
# sed -i  's#${user.home}/logs#/opt/var/logs/rocketmq#g' *.xml
# 查看修改
# grep logs *.xml
3.3.RocketMQ Nameserver启动
# nohup /opt/server/rocketmq/bin/mqnamesrv &
# 验证
# cat nohup.out
The Name Server boot success.
# netstat -anlp |grep 9876
tcp        0      0 :::9876                     :::*                        LISTEN      23750/java

# 开机启动
# cat /etc/rc.local
# RocketMQ nameserver
nohup /opt/server/rocketmq/bin/mqnamesrv &
3.4.RocketMQ Master1 Slave1
# 修改master1（10.162.23.70）配置
# vim /opt/server/rocketmq/conf/2m-2s-async/broker-a.properties
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=0
namesrvAddr=10.162.23.70:9876;10.162.23.71:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=50000000
destroyMapedFileIntervalForcibly=120000
redeleteHangedFileInterval=120000
diskMaxUsedSpaceRatio=88

storePathRootDir=/opt/data/rocketmq/data/store
storePathCommitLog=/opt/data/rocketmq/data/store/commitlog
maxMessageSize=65536
flushCommitLogLeastPages=4
flushConsumeQueueLeastPages=2
flushCommitLogThoroughInterval=10000
flushConsumeQueueThoroughInterval=60000

checkTransactionMessageEnable=false
sendMessageThreadPoolNums=128
pullMessageThreadPoolNums=128

brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH

# 启动master1
# nohup /opt/server/rocketmq/bin/mqbroker -c /opt/server/rocketmq/conf/2m-2s-async/broker-a.properties &
# netstat -anlp|grep java        
tcp        0      0 :::10911                    :::*                        LISTEN      24220/java         
tcp        0      0 :::10912                    :::*                        LISTEN      24220/java

# 开机启动（nameserver必须为启动状态，建议手动启动）
# cat /etc/rc.local
# RocketMQ master-a
nohup /opt/server/rocketmq/bin/mqbroker -c /opt/server/rocketmq/conf/2m-2s-async/broker-a.properties &

# 修改slave1（10.162.23.72）配置
# vim /opt/server/rocketmq/conf/2m-2s-async/broker-a.properties
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=1
namesrvAddr=10.162.23.70:9876;10.162.23.71:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=50000000
destroyMapedFileIntervalForcibly=120000
redeleteHangedFileInterval=120000
diskMaxUsedSpaceRatio=88

storePathRootDir=/opt/data/rocketmq/data/store
storePathCommitLog=/opt/data/rocketmq/data/store/commitlog
maxMessageSize=65536
flushCommitLogLeastPages=4
flushConsumeQueueLeastPages=2
flushCommitLogThoroughInterval=10000
flushConsumeQueueThoroughInterval=60000

checkTransactionMessageEnable=false
sendMessageThreadPoolNums=128
pullMessageThreadPoolNums=128

brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH

# 启动salve1
# nohup /opt/server/rocketmq/bin/mqbroker -c /opt/server/rocketmq/conf/2m-2s-async/broker-a.properties &

# netstat -anlp|grep java
tcp        0      0 :::10911                    :::*                        LISTEN      23774/java         
tcp        0      0 :::10912                    :::*                        LISTEN      23774/java

# 开机启动（nameserver必须为启动状态，建议手动启动）
# cat /etc/rc.local
# RocketMQ slave-a
nohup /opt/server/rocketmq/bin/mqbroker -c /opt/server/rocketmq/conf/2m-2s-async/broker-a.properties &

3.5.RocketMQ Master2 Slave2
# 修改master2（10.162.23.71）配置
# vim /opt/server/rocketmq/conf/2m-2s-async/broker-b.properties
brokerClusterName=DefaultCluster
brokerName=broker-b
brokerId=0
namesrvAddr=10.162.23.70:9876;10.162.23.71:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=50000000
destroyMapedFileIntervalForcibly=120000
redeleteHangedFileInterval=120000
diskMaxUsedSpaceRatio=88

storePathRootDir=/opt/data/rocketmq/data/store
storePathCommitLog=/opt/data/rocketmq/data/store/commitlog
maxMessageSize=65536
flushCommitLogLeastPages=4
flushConsumeQueueLeastPages=2
flushCommitLogThoroughInterval=10000
flushConsumeQueueThoroughInterval=60000

checkTransactionMessageEnable=false
sendMessageThreadPoolNums=128
pullMessageThreadPoolNums=128

brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH

# 启动master2
# nohup /opt/server/rocketmq/bin/mqbroker -c /opt/server/rocketmq/conf/2m-2s-async/broker-b.properties &

# 验证
# cat nohup.out
The Name Server boot success.
load config properties file OK, /opt/server/rocketmq/conf/2m-2s-sync/broker-b.properties
The broker[broker-a, 10.162.23.71:10911] boot success.
# netstat -anlp|grep java        
tcp        0      0 :::10911                    :::*                        LISTEN      24220/java         
tcp        0      0 :::10912                    :::*                        LISTEN      24220/java

# 开机启动（nameserver必须为启动状态，建议手动启动）
# cat /etc/rc.local
# RocketMQ master-b
nohup /opt/server/rocketmq/bin/mqbroker -c /opt/server/rocketmq/conf/2m-2s-async/broker-b.properties &

# 修改slave2（10.162.23.73）配置
# vim /opt/server/rocketmq/conf/2m-2s-async/broker-b.properties
brokerClusterName=DefaultCluster
brokerName=broker-b
brokerId=1
namesrvAddr=10.162.23.70:9876;10.162.23.71:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=50000000
destroyMapedFileIntervalForcibly=120000
redeleteHangedFileInterval=120000
diskMaxUsedSpaceRatio=88

storePathRootDir=/opt/data/rocketmq/data/store
storePathCommitLog=/opt/data/rocketmq/data/store/commitlog
maxMessageSize=65536
flushCommitLogLeastPages=4
flushConsumeQueueLeastPages=2
flushCommitLogThoroughInterval=10000
flushConsumeQueueThoroughInterval=60000

checkTransactionMessageEnable=false
sendMessageThreadPoolNums=128
pullMessageThreadPoolNums=128

brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH

# 启动salve1
# nohup /opt/server/rocketmq/bin/mqbroker -c /opt/server/rocketmq/conf/2m-2s-async/broker-b.properties &

# 验证
# cat nohup.out
load config properties file OK, /opt/server/rocketmq/conf/2m-2s-sync/broker-b.properties
The broker[broker-b, 10.162.23.73:10911] boot success. and name server is 10.162.23.70:9876;10.162.23.71:9876
# netstat -anlp|grep java
tcp        0      0 :::10911                    :::*                        LISTEN      23774/java         
tcp        0      0 :::10912                    :::*                        LISTEN      23774/java

# 开机启动（nameserver必须为启动状态，建议手动启动）
# cat /etc/rc.local
# RocketMQ slave-b
nohup /opt/server/rocketmq/bin/mqbroker -c /opt/server/rocketmq/conf/2m-2s-async/broker-b.properties &

注：服务停止直接采用kill java 进程的方法

3.6.RocketMQ nameserver 接口
10.162.23.70:9876;10.162.23.71:9876