RocketMQ cluster deployment installation

RcoketMQ：【

1. Low latency: under high pressure, within 1 millisecond delay more than 99.6% of the reaction.

2. for Finance: High Availability with tracking and auditing capabilities.

3. Industry Sustainable development: ensuring the capacity of one trillion messages.

4. The vendor-neutral: a new open distributed messaging and streaming media standards from the latest 4.1 version.

5. BigData friendly: the bulk transfer and integration of multi-purpose flood throughput.

6. A lot of accumulation: As long as there is enough disk space, you can accumulate messages without loss of performance.

】

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

server: 172.26.141.69 (from a master b)

server: 172.26.141.75 (b master from a)

Monitoring and management platform: 172.26.141.85

1. Installation Configuration Environment jdk1.8
wget http://172.26.107.14/newyum/jdk-8-1.el7.x86_64.rpm
the sudo yum the install local JDK -Y-8-1.el7.x86_64.rpm

2. Installation Configuration Environment MAVEN
the tar-zxf Apache-3.6.2-Maven-bin.tar.gz
Music Videos Apache-3.6.2-Maven / apprun /
Music Videos Apache-3.6.2-Maven Maven

环境配置(sudo vi /etc/profile)
MAVEN_HOME=/apprun/maven
PATH=$PATH:$MAVEN_HOME/bin
export JAVA_HOME=/apprun/jdk
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH=$JAVA_HOME/bin:$HOME/bin:$HOME/.local/bin:$PATH

3.rocketmq install :( http://rocketmq.apache.org/ )

http://rocketmq.apache.org/release_notes/release-notes-4.4.0/

unzip -d ./ ./rocketmq-all-4.4.0-source-release.zip
cd rocketmq-all-4.4.0
mvn -Prelease-all -DskipTests clean install -U
时间比较久
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:24 h
[INFO] Finished at: 2019-11-25T16:58:53+08:00
[INFO] ------------------------------------------------------------------------
[apprun@rocketmq01 rocketmq-all-4.4.0]$

Distribution CD / target /
Music Videos Apache-rocketmq / apprun /
next first directory built into rocketmq-4.4.0 directory (apache-rocketmq), the logs directory, data / store / ........ follows file folder shown built
CD / apprun / Apache-rocketmq /
mkdir -pv {logs, Data / Store / {commitlog, consumequeue, index}, DATA2 / Store / {commitlog, consumequeue, index}}

Parameters to change the startup script (virtual machine)
1.vim runbroker.sh (because the default value is suitable for use in the production) -server -Xms512m -Xmx512m -Xmn256m
2. vim runserver.sh (same reason) -server -Xms512m -XX -Xmn126m -Xmx512m: PermSize -XX = 128M: 320M MaxPermSize =
3. Vim tools.sh -Xms256m -Xmx256m -Xmn128m -server -XX: 128M PermSize = -XX: MaxPermSize = 128M

Modify the configuration file (/ apprun / apache-rocketmq / conf / 2m-2s-async /) [ two main cross machine from]
the main machine 172.26.141.69 file contents are two broker-a.properties broker-bs.properties as follows:
vim broker-a.properties
# ========================================= ================================================== =============
brokerClusterName = JiaTingYunRocketMQCluster
brokerName = Broker-A
brokerId = 0
# deleteWhen = 04
# = 48 fileReservedTime
brokerRole = ASYNC_MASTER
flushDiskType = ASYNC_FLUSH

= 10911 the listenPort
#nameserver address, separated by semicolons
namesrvAddr = 172.26.141.69: 9876; 172.26.141.75: 9876
# sending the message, the server automatically creates a topic does not exist, the default number of queues created
defaultTopicQueueNums. 4 =
# whether to allow automatic Broker Topic created, the turned recommendation line, off line
autoCreateTopicEnable to false =
# allow Broker automatically subscribed to create, open suggestion line, off line
autoCreateSubscriptionGroup to false =
brokerIP1 = 172.26.141.69
storePathRootDir = / apprun / Apache-rocketmq / Data / store
storePathCommitLog = / apprun / Apache-rocketmq / Data / store / commitlog
# consumer queue storage path storage paths
storePathConsumerQueue = / apprun / Apache-rocketmq / Data / store / consumequeue
# message index storage path
storePathIndex = / apprun / apache-rocketmq / Data / store / index
#checkpoint file storage path
= storeCheckpoint / apprun / the Apache-rocketmq / the Data / Store / checkpoint
#abort file storage path
abortFile = / apprun / the Apache-rocketmq / the Data / Store / ABORT
# delete files point in time, the default 4:00
deleteWhen = 04
# Files retention time default 48 hours
fileReservedTime = 120
# commitLog default size of each file. 1G
mapedFileSizeCommitLog = 1073741824
#ConsumeQueue 30W each file in the default storage section, adjusted according to the service circumstances
mapedFileSizeConsumeQueue = 300000
# destroyMapedFileIntervalForcibly = 120000
# 120000 redeleteHangedFileInterval =
# detect a physical disk file space
diskMaxUsedSpaceRatio = 88 #
# ============================================= ================================================== =========

vim broker-b-s.properties
#========================================================================================================
brokerClusterName=JiaTingYunRocketMQCluster
brokerName=broker-b
brokerId=1
#deleteWhen=04
#fileReservedTime=48
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH

= 10921 the listenPort
#nameserver address, separated by semicolons
namesrvAddr = 172.26.141.69: 9876; 172.26.141.75: 9876
brokerIP1 = 172.26.141.69
# sending the message, the server automatically creates a topic does not exist, the default number of queues created
defaultTopicQueueNums = 4
# allow Broker to automatically create Topic, turn on the advice of the line, the line is closed
autoCreateTopicEnable = false
# whether to allow the Broker to automatically create a subscription group, the advice line open, online close
autoCreateSubscriptionGroup = false

storePathRootDir = / apprun / Apache-rocketmq / DATA2 / Store
storePathCommitLog = / apprun / Apache-rocketmq / DATA2 / Store / commitlog
# consumer queue storage path storage paths
storePathConsumerQueue = / apprun / Apache-rocketmq / DATA2 / Store / consumequeue
# message index storage path
storePathIndex = / apprun / Apache-rocketmq / DATA2 / store / index
#checkpoint file storage path
storeCheckpoint = / apprun / Apache-rocketmq / DATA2 / store / the checkpoint
#abort file storage path
abortFile = / apprun / apache-rocketmq / data2 / Store / ABORT
# delete the file time point, the default 4:00
deleteWhen = 04
# retention time files, the default 48 hours
fileReservedTime = 120
# commitLog default size of each file. 1G
mapedFileSizeCommitLog = 1073741824
#ConsumeQueue 30W each file in the default storage section, The traffic situation adjusted
mapedFileSizeConsumeQueue = 300000
destroyMapedFileIntervalForcibly = 120000 #
# redeleteHangedFileInterval = 120000
# detect a physical disk file space
# diskMaxUsedSpaceRatio = 88
# =============================== ================================================== =======================

172.26.141.75 on the machine broker-b.properties broker-as.properties two main contents of the file are as follows:
vim broker-b.properties
# ==================== ================================================== ==================================
brokerClusterName = JiaTingYunRocketMQCluster
brokerName = Broker B-
brokerId = 0
# 04 = deleteWhen
# = 48 fileReservedTime
brokerRole = ASYNC_MASTER
flushDiskType = ASYNC_FLUSH

= 10911 the listenPort
#nameserver address, separated by semicolons
namesrvAddr = 172.26.141.69: 9876; 172.26.141.75: 9876
# sending the message, the server automatically creates a topic does not exist, the default number of queues created
defaultTopicQueueNums. 4 =
# whether to allow automatic Broker Topic created, the turned recommendation line, off line
autoCreateTopicEnable to false =
# allow Broker automatically subscribed to create, open suggestion line, off line
autoCreateSubscriptionGroup to false =
brokerIP1 = 172.26.141.75
storePathRootDir = / apprun / Apache-rocketmq / Data / store
storePathCommitLog = / apprun / Apache-rocketmq / Data / store / commitlog
# consumer queue storage path storage paths
storePathConsumerQueue = / apprun / Apache-rocketmq / Data / store / consumequeue
# message index storage path
storePathIndex = / apprun / apache-rocketmq / Data / store / index
#checkpoint file storage path
= storeCheckpoint / apprun / the Apache-rocketmq / the Data / Store / checkpoint
#abort file storage path
abortFile = / apprun / the Apache-rocketmq / the Data / Store / ABORT
# delete files point in time, the default 4:00
deleteWhen = 04
# Files retention time default 48 hours
fileReservedTime = 120
# commitLog default size of each file. 1G
mapedFileSizeCommitLog = 1073741824
#ConsumeQueue 30W each file in the default storage section, adjusted according to the service circumstances
mapedFileSizeConsumeQueue = 300000
# destroyMapedFileIntervalForcibly = 120000
# 120000 redeleteHangedFileInterval =
# detect a physical disk file space
diskMaxUsedSpaceRatio = 88 #
# ============================================= ================================================== =========

vim broker-a-s.properties
#========================================================================================================
brokerClusterName=JiaTingYunRocketMQCluster
brokerName=broker-a
brokerId=1
#deleteWhen=04
#fileReservedTime=48
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH

= 10921 the listenPort
#nameserver address, separated by semicolons
namesrvAddr = 172.26.141.69: 9876; 172.26.141.75: 9876
brokerIP1 = 172.26.141.75
# sending the message, the server automatically creates a topic does not exist, the default number of queues created
defaultTopicQueueNums = 4
# allow Broker to automatically create Topic, turn on the advice of the line, the line is closed
autoCreateTopicEnable = false
# whether to allow the Broker to automatically create a subscription group, the advice line open, online close
autoCreateSubscriptionGroup = false

Add start-stop pace (1.nameServer.sh 2.master.sh 3.slave.sh)
start: 1. First start two sets of nameServer.sh
cd / apprun / the Apache-rocketmq

Add nameServer.sh script
bash nameServer.sh
2. restart master.sh two stations

Add master.sh script

master.sh bash
3. The last two units from the start

Add slave.sh script

bash slave.sh

View

[apprun@rocketmq02 apache-rocketmq]$ jps
8849 NamesrvStartup
9041 BrokerStartup
8917 BrokerStartup
14934 Jps
[apprun@rocketmq02 apache-rocketmq]$

[apprun@rocketmq01 ~]$ /apprun/apache-rocketmq/bin/mqadmin clusterList -n 172.26.141.69:9876
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
#Cluster Name #Broker Name #BID #Addr #Version #InTPS(LOAD) #OutTPS(LOAD) #PCWait(ms) #Hour #SPACE
JiaTingYunRocketMQCluster broker-a 0 172.26.141.69:10911 V4_4_0 0.00(0,0ms) 0.00(0,0ms) 0 437426.29 0.0108
JiaTingYunRocketMQCluster broker-a 1 172.26.141.75:10921 V4_4_0 0.00(0,0ms) 0.00(0,0ms) 0 437426.29 0.0108
JiaTingYunRocketMQCluster broker-b 0 172.26.141.75:10911 V4_4_0 0.00(0,0ms) 0.00(0,0ms) 0 437426.29 0.0108
JiaTingYunRocketMQCluster broker-b 1 172.26.141.69:10921 V4_4_0 0.00(0,0ms) 0.00(0,0ms) 0 437426.29 0.0108
[apprun@rocketmq01 ~]$

4。安装配置监控管理（172.26.141.85）
cd /apprun/soft/wangbocheng
git clone https://github.com/apache/rocketmq-externals.git
ln -s /apprun/soft/wangbocheng/rocketmq-externals /apprun/

修改配置文件
vim /apprun/rocketmq-externals/rocketmq-console/src/main/resources/application.properties
#=======================================================================================================
server.contextPath=
server.port=8080

### SSL setting
#server.ssl.key-store=classpath:rmqcngkeystore.jks
#server.ssl.key-store-password=rocketmq
#server.ssl.keyStoreType=PKCS12
#server.ssl.keyAlias=rmqcngkey

#spring.application.index=true
spring.application.name=rocketmq-console
spring.http.encoding.charset=UTF-8
spring.http.encoding.enabled=true
spring.http.encoding.force=true
logging.config=classpath:logback.xml
#if this value is empty,use env value rocketmq.config.namesrvAddr NAMESRV_ADDR | now, you can set it in ops page.default localhost:9876
rocketmq.config.namesrvAddr=172.26.141.69:9876;172.26.141.75:9876
#if you use rocketmq version < 3.5.8, rocketmq.config.isVIPChannel should be false.default true
rocketmq.config.isVIPChannel=
#rocketmq-console's data path:dashboard/monitor
rocketmq.config.dataPath=/tmp/rocketmq-console/data
#set it false if you don't want use dashboard.default true
rocketmq.config.enableDashBoardCollect=true
#set the message track trace topic if you don't want use the default one
rocketmq.config.msgTrackTopicName=
rocketmq.config.ticketKey=ticket

#Must create userInfo file: ${rocketmq.config.dataPath}/users.properties if the login is required
rocketmq.config.loginRequired=false

#=======================================================================================================

接着对rocketmq-console进行编译打包运行（mvn的编辑过程需要等待一段时间）
cd /apprun/rocketmq-externals/rocketmq-console/
mvn clean package -Dmaven.test.skip=true

[INFO] Building jar: /apprun/soft/wangbocheng/rocketmq-externals/rocketmq-console/target/rocketmq-console-ng-1.0.1-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:23 min
[INFO] Finished at: 2019-11-26T09:55:23+08:00
[INFO] ------------------------------------------------------------------------

编译成功后在target下找到文件：rocketmq-console-ng-1.0.1.jar

创建启动脚本
cd /apprun/
vim MQ_jiankong.sh
#=======================================================================================================
#!/bin/sh
base_dir=$(cd "$(dirname "$0")"; pwd)

nohup java -jar /apprun/rocketmq-externals/rocketmq-console/target/rocketmq-console-ng-1.0.1.jar --server.port=8080 --rocketmq.config.namesrvAddr="172.26.141.69:9876;172.26.141.75:9876" &
#=======================================================================================================

启动
bash MQ_jiankong.sh
查看
ps aux |grep rocketmq-console-ng
或
netstat -ntlp | grep 8080

#==========空跑俩天，发现内存消耗过大==========
RocketMQ性能调优
1.开启异步刷盘
flushDiskType=ASYNC_FLUSH
同步刷盘TPS过低，较难满足业务发展需求
2.开启堆外内存设置
transientStorePoolEnable=true
消息写入到堆外内存，消费时从pageCache消费，读写分离，提升集群性能
3.开启文件预热
warmMapedFileEnable=true
开启文件预热，避免日志文件在分配内存时缺页中断
4.开启Slave读权限
slaveReadEnable=true
消息占用物理内存的大小通过accessMessageInMemoryMaxRatio来配置默认为40%；
如果消费的消息不在内存中，开启slaveReadEnable时会从slave节点读取；提高Master内存利用率
5.关闭堆内存据传输
transferMsgByHeap默认true设置为false
Broker响应消费请求时，不必将数据重新读到堆内存再发送给客户端；
直接从PageCache将数据发送给客户端

RocketMQ cluster deployment installation

Guess you like