1. Introduction to RocketMQ
1.1 Introduction
RocketMQ is a distributed, queue model messaging middleware with the following features:
Able to guarantee strict message ordering
Provides rich message pull mode
Efficient subscriber horizontal scalability
Real-time message subscription mechanism
Billion-level message accumulation capability
Reason for selection:
- Emphasizes that the cluster has no single point, is scalable, highly available at any point, and horizontally scalable
- Mass message accumulation capability, low write latency after message accumulation
- Supports tens of thousands of queues
- Message failure retry mechanism
- Message can be queried
- Active open source community
- mature
1.2 Key Concepts
1.2.1 Topics and tags
Topic: the first-level message type, the title of the book
Tags: the second-level message type, the directory of the book, you can filter messages based on tags
E.g:
theme:
order transaction
Label:
Order Transaction - Create
Order Transaction - Payment
Order Transaction - Complete
1.2.2 Send and subscribe to groups
Production group : for sending messages
Consumer group : for subscription processing of messages
The production group and the consumption group are convenient for scaling machines, increasing or decreasing processing power, and the name of the cluster group, which is used to mark a member of the purpose. Only one member of each cluster will be sent randomly at a time.
2. RocketMQ cluster method
Several recommended Broker cluster deployment methods, the Slave here is not writable, but readable, similar to the Mysql master-slave mode.
2.1 Single Master
This method is more risky. Once the Broker restarts or goes down, the entire service will be unavailable. It is not recommended to use it in an online environment.
2.2 Multi-master mode
A cluster has no slaves, all masters, such as 2 masters or 3 masters
Advantages: Simple configuration, single Master downtime or restart maintenance has no impact on the application, when the disk is configured as RAID10, in the event that the machine is down and cannot be recovered in time, because the RAID10 disk is very reliable, messages will not be lost (asynchronous disk flushing is lost). A small amount of messages, synchronously flushing a disk without losing one) has the highest performance.
Disadvantages: During the downtime of a single machine, unconsumed messages on this machine cannot be subscribed until the machine is restored, and the real-time nature of messages will be affected.
###Start NameServer first
###On machine A, start the first Master
###On machine B, start the second Master
2.3 Multi-Master multi-slave mode, asynchronous replication
Each Master is configured with one Slave, and there are multiple pairs of Master-Slave. HA adopts the asynchronous replication method, and the master and backup have a short message delay, which is in milliseconds.
Advantages: Even if the disk is damaged, very few messages are lost, and the real-time performance of messages will not be affected, because after the Master is down, consumers can still send messages from the Slave. This process is transparent to the application, does not require manual intervention, and has the same performance as multiple Masters. The pattern is almost the same.
Disadvantages: Master downtime, disk damage, a small amount of messages will be lost
### Start NameServer first
### On machine A, start the first Master
### On machine B, start the second Master
### On machine C, start the first slave
### On machine D, start the second slave
2.4 Multi-Master multi-slave mode, synchronous double write
Each Master is configured with one Slave, and there are multiple pairs of Master-Slave. The HA adopts the synchronous double-write method. Both the master and the slave write successfully and return success to the application.
Advantages: There is no single point for data and services. In the case of Master downtime, there is no delay in messages, and service availability and data availability are very high.
Disadvantages: The performance is slightly lower than the asynchronous replication mode, about 10% lower, and the RT for sending a single message will be slightly higher. At present, after the main machine is down, the standby machine cannot automatically switch to the main machine, and the automatic switch function will be supported in the future.
### Start NameServer first
### On machine A, start the first Master
### On machine B, start the second Master
### On machine C, start the first slave
### On machine D, start the second slave
The above Broker and Slave are paired by specifying the same brokerName parameter. The BrokerId of the Master must be 0, and the BrokerId of the Slave must be a number greater than 0. Another Master can mount multiple slaves, and multiple slaves under the same Master can be distinguished by specifying different BrokerIds.
3. RocketMQ deployment [dual master mode]
3.1 Server Environment
| serial number | IP | username | password | Role | model |
| 1 | 192.168.1.114 | root | nameServer1,brokerServer1 | master | |
| 2 | 192.168.2.2 | root | nameServer1,brokerServer1 | master |
3.2 Hosts add information
| IP | NAME |
| 192.168.1.114 | rocketmq-nameserver1 |
| 192.168.1.114 | rocketmq-master1 |
| 192.168.2.2 | rocketmq-nameserver2 |
| 192.168.2.2 | rocketmq-master2 |
Modify the relevant information of the linux server through the ssh link;
# vi /etc/hosts
After adding the corresponding information, save and exit
3.3 Upload and decompress [two machines]
Upload alibaba-rocketmq-3.2.6.tar.gz file to /usr/local
Unzip the file to the execution directory
# tar -zxvf alibaba-rocketmq-3.2.6.tar.gz -C /usr/local
Rename and increase the version number for easy sorting in the future
# mv alibaba-rocketmq alibaba-rocketmq-3.2.6
Create a soft connection, no -s here is to create a hard connection
# ln -s alibaba-rocketmq-3.2.6 rocketmq
# ll .usr/local View the file, you will see the soft link identifier
3.4 Create a storage path [two machines]
# mkdir /usr/local/rocketmq/store
# mkdir /usr/local/rocketmq/store/commitlog
# mkdir /usr/local/rocketmq/store/consumequeue
# mkdir /usr/local/rocketmq/store/index
3.5 RocketMQ configuration file [two machines]
# vim /usr/local/rocketmq/conf/2m-noslave/broker-a.properties
# vim /usr/local/rocketmq/conf/2m-noslave/broker-b.properties
Delete the original configuration, copy the configuration below, paste it, make the corresponding modification, then save and exit
#The name of the cluster that belongs to
brokerClusterName=rocketmq-cluster
#broker name, note that the different configuration files here are filled differently, a.properties, just write broker-a, and b, fill in b
brokerName=broker-a|broker-b
# 0 means Master, >0 means Slave
brokerId=0
#nameServer address, separated by semicolon
namesrvAddr=rocketmq-nameserver1:9876;rocketmq-nameserver2:9876 #When
sending a message, a topic that does not exist on the server is automatically created, and the number of queues created by default
defaultTopicQueueNums=4 #Whether
to allow Broker to automatically create Topic, it is recommended to open offline , close online
autoCreateTopicEnable =true #Whether
to allow Broker to automatically create subscription groups, it is recommended to open offline, close online
autoCreateSubscriptionGroup=true
#Broker listening port for external services
listenPort =10911 #Delete
file time point, the default is 4 am
deleteWhen=04 #File
retention time, default 48 hours
fileReservedTime=120
#commitLog The default size of each file is 1G
mapedFileSizeCommitLog=1073741824
#ConsumeQueue stores 30W per file by default, adjust
mapedFileSizeConsumeQueue=300000 according to the business situation
#destroyMapedFileIntervalForcibly=120000
#redeleteHangedFileInterval=120000 #Detect
physical file disk space
diskMaxUsedSpaceRatio=88 #Storage
path
storePathRootDir=/usr/local/rocketmq/store
#commitLog storage Path
storePathCommitLog=/usr/local/rocketmq/store/commitlog #Consume
queue storage path storage path
storePathConsumeQueue=/usr/local/rocketmq/store/consumequeue #Message
index storage path
storePathIndex=/usr/local/rocketmq/store/index
# checkpoint file storage path
storeCheckpoint=/usr/local/rocketmq/store/checkpoint
#abort file storage path
abortFile=/usr/local/rocketmq/store/abort #Limited
message size
maxMessageSize=65536
#flushCommitLogLeastPages=4
#flushConsumeQueueLeastPages=2
#flushCommitLogThoroughInterval=10000
#flushConsumeQueueThoroughInterval=60000 #
Broker's role
#- ASYNC_MASTER asynchronous replication Master
#- SYNC_MASTER synchronous double write Master
#- SLAVE
brokerRole=ASYNC_MASTER #Brush
method #
- ASYNC_FLUSH asynchronous brush
- SYNC_FLUSH synchronous flush disk
flushDiskType=ASYNC_FLUSH
#checkTransactionMessageEnable=false #Number of
message thread pools
#sendMessageThreadPoolNums=128 #Number
of pull message thread pools
#pullMessageThreadPoolNums=128
3.6 Modify the log configuration file [two machines]
# mkdir -p /usr/local/rocketmq/logs
# cd /usr/local/rocketmq/conf && sed -i 's#${user.home}#/usr/local/rocketmq#g' *.xml
3.7 Modify startup script parameters [two machines]
# vim /usr/local/rocketmq/bin/runbroker.sh
Since the hardware configuration of the machine is not enough, the minimum configuration is adopted. Note that -Xms -Xmx cannot be lower than 1g
# vim /usr/local/rocketmq/bin/runserver.sh
3.8 Start NameServer [two machines]
# cd /usr/local/rocketmq/bin
# nohup sh mqnamesrv &
You can enter the command jps to view
NamesrvStartup
3.9 Start BrokerServer A [192.168.1.114]
# cd /usr/local/rocketmq/bin
# nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-noslave/broker-a.properties >/dev/null 2>&1 &
# netstat -ntlp
# jps
# tail -f -n 500 /usr/local/rocketmq/logs/rocketmqlogs/broker.log
# tail -f -n 500 /usr/local/rocketmq/logs/rocketmqlogs/namesrv.log
3.10 Start BrokerServer B [192.168.2.2]
# cd /usr/local/rocketmq/bin
# nohup sh mqbroker -c /usr/local/rocketmq/conf/2m-noslave/broker-b.properties >/dev/null 2>&1 &
# netstat -ntlp
# jps
# tail -f -n 500 /usr/local/rocketmq/logs/rocketmqlogs/broker.log
# tail -f -n 500 /usr/local/rocketmq/logs/rocketmqlogs/namesrv.log
3.11 Deploy RocketMQ Console
wget https://github.com/duomu/rocketmq-console/raw/master/rocketmq-console.war download
You can also copy the following address to download directly, after completion
Deploy rocketmq-console.war to tomcat. Before starting, modify the configuration in the config.properties file of \WEB-INF\classes in the war package.
I modified the configuration here to:
rocketmq.namesrv.addr=192.168.2.2:9876;192.168.1.114:9876
Then start tomcat and view the startup log of tomcat
Then the browser accesses the console, my project is deployed at 192.168.1.114:10000
In this way, the environment is set up, and later, it is to write HelloWorld.