rocketMq cluster (source: http://blog.csdn.net/zhu_tianwei/article/details/40949523/)

Broker cluster deployment methods mainly include the following: (Slave is not writable, but readable)
(1) This method of a single Master
is more risky. Once the Broker restarts or goes down, the entire service will be unavailable. It is not recommended to go online. environmental use.
(2) Multi-Master mode
A cluster has no slaves, but all masters, such as 2 masters or 3 masters.
Advantages: simple configuration, single Master downtime or restart maintenance has no impact on the application, when the disk is configured as RAID10, even if the machine is down and cannot be recovered, the RAID10 disk is very reliable, and messages will not be lost (asynchronous disk flushing is lost For a small amount of messages, one will not be lost by synchronizing the disk). Highest performance.
 Disadvantages: During the downtime of a single machine, unconsumed messages on this machine cannot be subscribed until the machine is restored, and the real-time nature of messages will be affected.
#Start NameServer first, for example, the machine IP is: 192.168.36.189:9876
nohup ./bin/mqnamesrv >/dev/null 2>&1 & #On
machine A, start the first Master
nohup ./bin/mqbroker -n 192.168. 36.189:9876 -c ./conf/2m-noslave/broker-a.properties >/dev/null 2>&1 &
# On machine B, start the second Master
nohup ./bin/mqbroker -n 192.168.36.189:9876 -c ./conf/2m-noslave/broker-b.properties >/dev/null 2>&1 &
(3) Multi-Master multi-slave mode, asynchronous replication
Each Master is configured with one Slave, with multiple pairs of Master-Slave, HA adopts asynchronous replication mode, and the master and backup have short message delay, millisecond level.
 Advantages: Even if the disk is damaged, very few messages are lost, and the real-time performance of messages will not be affected, because after the Master is down, consumers can still consume from the Slave, and this process is transparent to the application. No human intervention is required. The performance is almost the same as the multi-Master mode.
 Disadvantages: The Master is down, the disk is damaged, and a small amount of messages will be lost.
#Start NameServer first, for example, the machine IP is: 192.168.36.189:9876
nohup ./bin/mqnamesrv >/dev/null 2>&1 & #On
machine A, start the first Master
nohup ./bin/mqbroker -n 192.168. 36.189:9876 -c ./conf/2m-2s-async/broker-a.properties >/dev/null 2>&1 &
# On machine B, start the second Master
nohup ./bin/mqbroker -n 192.168.36.189 :9876 -c ./conf/2m-2s-async/broker-b.properties >/dev/null 2>&1 &
# On machine C, start the first slave
nohup ./bin/mqbroker -n 192.168.36.189: 9876 -c ./conf/2m-2s-async/broker-as.properties >
# On machine D, start the second slave
nohup ./bin/mqbroker -n 192.168.36.189:9876 -c ./conf/2m-2s-async/broker-bs.properties >/dev/null 2>&1 &
( 4) Multi-Master multi-slave mode, synchronous double write
Each Master is configured with one Slave, there are multiple pairs of Master-Slave, HA adopts synchronous double write mode, the master and backup are successfully written, and the success is returned to the application.
 Advantages: There is no single point for data and services. In the case of Master downtime, messages are not delayed, and service availability and data availability are very high
 . Disadvantages: The performance is slightly lower than the asynchronous replication mode, about 10% lower, and the RT that sends a single message will slightly higher. At present, after the main machine is down, the standby machine cannot automatically switch to the main machine, and the automatic switch function will be supported in the future.
#Start NameServer first, for example, the machine IP is: 192.168.36.189:9876
nohup ./bin/mqnamesrv >/dev/null 2>&1 & #On
machine A, start the first Master
nohup ./bin/mqbroker -n 192.168. 36.189:9876 -c ./conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 &
# On machine B, start the second Master
nohup ./bin/mqbroker -n 192.168.36.189 :9876 -c ./conf/2m-2s-sync/broker-b.properties >/dev/null 2>&
#On machine C, start the first Slave
nohup ./bin/mqbroker -n 192.168.36.189:9876 -c ./conf/2m-2s-sync/broker-as.properties >/dev/null 2>&1 &
# On machine D, start the second slave
nohup ./bin/mqbroker -n 192.168.36.189:9876 -c ./conf/2m-2s-sync/broker-bs.properties >/dev/null 2>&1 &
above Broker Pairing with the Slave is done by specifying the same brokerName parameter. The BrokerId of the Master must be 0, and the BrokerId of the Slave must be a number greater than or equal to 0. Another Master can mount multiple slaves, and multiple slaves under the same Master can be distinguished by specifying different BrokerIds.
 In addition, the nameserver also needs to be clustered.
The following takes the configuration of one master and one backup (synchronization) and two nameservers as an example to test.
1. Two machines in the environment:
192.168.36.54 is the main machine and
192.168.36.189 is the backup machine.
At the same time, start a nameserver on two machines. To install RocketMq, please refer to: http://blog.csdn.net/zhu_tianwei/article/details/40948447
2. Modify the configuration
(1) Create a directory
mkdir /home/rocket/alibaba-rocketmq/logs #Create log directory
mkdir -p /home/rocket/alibaba-rocketmq/data/store/commitlog #Create data storage directory
Change log directory
cd /home/rocket/alibaba-rocketmq/ conf
sed -i 's#${user.home}#${user.home}/alibaba-rocketmq#g' *.xml
(2) Modify the main configuration
vi ./conf/2m-2s-sync/broker-a .properties
[plain] view plain copy print? View snippet on CODE derived to my snippet
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=0
namesrvAddr=192.168.36.189:9876;192.168.36.54:9876
defaultTopicQueueNums=4
autoCreateTopicEnable =true
autoCreateSubscriptionGroup=true
listenPort=10911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=50000000
destroyMapedFileIntervalForcibly=120000
redeleteHangedFileInterval=120000
diskMaxUsedSpaceRatio=88

storePathRootDir=/home/rocket/alibaba-rocketmq/data/store
storePathCommitLog=/home/rocket/alibaba-rocketmq/data/store/commitlog
maxMessageSize=65536
flushCommitLogLeastPages=4
flushConsumeQueueLeastPages=2
flushCommitLogThoroughInterval=10000
flushConsumeQueueThoroughInterval=60000

checkTransactionMessageEnable=false
sendMessageThreadPoolNums=128
pullMessageThreadPoolNums=128

brokerRole=SYNC_MASTER
flushDiskType=ASYNC_FLUSH
(3) Modify the standby configuration
[plain] view plain copy print? View the code slice on CODE derived from my code slice
brokerClusterName=DefaultCluster
brokerName=broker-a
brokerId=1
namesrvAddr=192.168.36.189:9876;192.168.36.54:9876
defaultTopicQueueNums=4
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
deleteWhen=04
fileReservedTime=120
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=50000000
destroyMapedFileIntervalForcibly=120000
redeleteHangedFileInterval=120000
diskMaxUsedSpaceRatio=88

storePath
storePathCommitLog=/home/rocket/alibaba-rocketmq/data/store/commitlog
maxMessageSize=65536
flushCommitLogLeastPages=4
flushConsumeQueueLeastPages=2
flushCommitLogThoroughInterval=10000
flushConsumeQueueThoroughInterval=60000

checkTransactionMessageEnable=false
sendMessageThreadPoolNums=128
pullMessageThreadPoolNums=128

brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH
3.启动两台nameserver
nohup ./bin/mqnamesrv >/dev/null 2>&1 &
4.启动2台broker
主：nohup ./bin/mqbroker -c ./conf/2m-2s-sync/broker-a.properties >/dev/null 2>&1 &
备：nohup ./bin/mqbroker -c ./conf/2m-2s-sync/broker-a-s.properties >/dev/null 2>&1 &
ok, the configuration startup is complete.
Example:
1. Consumer Consumer.java, which is consumed by active pull.
[java] view plain copy print? View code slice on CODE derived to my code slice
package cn.slimsmart.rocketmq.demo.cluster;

import java.util.HashMap;
import java.util.List;
import java.util. Map;
import java.util.Set;

import com.alibaba.rocketmq.client.consumer.DefaultMQPullConsumer;
import com.alibaba.rocketmq.client.consumer.PullResult;
import com.alibaba.rocketmq.client.exception.MQClientException;
import com .alibaba.rocketmq.common.message.MessageExt;
import com.alibaba.rocketmq.common.message.MessageQueue;

//Consumer pull
public class Consumer {

 // Java cache
 private static final Map<MessageQueue, Long> offsetTable = new HashMap<MessageQueue, Long>();

 /**
 * Active pull consumption
 *
 * @throws MQClientException
 */
 public static void main(String[] args) throws MQClientException {
 /**
 * An application creates a Consumer, which is maintained by the application, which can be set as a global object or a singleton 
 * Note: ConsumerGroupName needs to be guaranteed by the application to be unique, it is best to use the package name of the service to distinguish the same service, The name of a type of Consumer collection,
 * This type of Consumer usually consumes a type of message, and the consumption logic is consistent
 * PullConsumer: a type of Consumer, the application usually actively calls the Consumer's pull message method to pull messages from the Broker, and the initiative is controlled by the application
 * /
 DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("ConsumerGroupName");
 // //nameserver service
 consumer.setNamesrvAddr("192.168.36.189:9876;192.168.36.54:9876");
 consumer.setInstanceName("Consumber");
 consumer.start();
 // 拉取订阅主题的队列，默认队列大小是4
 Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("TopicTest1");
 for (MessageQueue mq : mqs) {
 System.out.println("Consume from the queue: " + mq);
 SINGLE_MQ: while (true) {
 try {
 PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
 List<MessageExt> list = pullResult.getMsgFoundList();
 if (list != null && list.size() < 100) {
 for (MessageExt msg : list) {
 System.out.println(new String(msg.getBody()));
 }
 }
 System.out.println(pullResult.getNextBeginOffset());
 putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
 switch (pullResult.getPullStatus()) {
 case FOUND:
 break;
 case NO_MATCHED_MSG:
 break;
 case NO_NEW_MSG:
 break SINGLE_MQ;
 case OFFSET_ILLEGAL:
 break;
 default:
 break;
 }
 } catch (Exception e) {
 e.printStackTrace();
 }
 }
 }
 consumer.shutdown();
 }

 private static void putMessageQueueOffset(MessageQueue mq, long offset) {
 offseTable.put(mq, offset);
 }

 private static long getMessageQueueOffset(MessageQueue mq) {
 Long offset = offseTable.get(mq);
 if (offset != null) {
 System.out.println(offset);
 return offset;
 }
 return 0;
 }
}

2. Producer Producer.java , TransactionMQProducer uses
[java] view plain copy print? View the code slice on CODE Derived to my code slice
package cn.slimsmart.rocketmq.demo.cluster;

import java.util.concurrent.TimeUnit;

import com.alibaba.rocketmq.client.exception.MQClientException;
import com.alibaba.rocketmq.client.producer.LocalTransactionExecuter;
import com.alibaba.rocketmq.client.producer.LocalTransactionState;
import com .alibaba.rocketmq.client.producer.SendResult;
import com.alibaba.rocketmq.client.producer.TransactionCheckListener;
import com.alibaba.rocketmq.client.producer.TransactionMQProducer;
import com.alibaba.rocketmq.common.message.Message;
import com.alibaba.rocketmq.common.message.MessageExt;

//Producer
public class Producer {

 public static void main(String[] args) throws MQClientException,
 InterruptedException {
 / **
 * An application creates a Producer, and the application maintains this object, which can be set as a global object or a singleton 
 * Note: ProducerGroupName needs to be guaranteed by the application to be unique, the name of a type of Producer collection, this type of Producer usually sends A type of message, and the sending logic is consistent 
 * The concept of ProducerGroup is not very useful when sending ordinary messages, but it is more critical when sending distributed transaction messages,
 * Because the server will check back any Producer under this group
 * /
 final TransactionMQProducer producer = new TransactionMQProducer("ProducerGroupName");
 //nameserver service
 producer.setNamesrvAddr("192.168.36.189:9876;192.168.36.54:9876");
 producer.setInstanceName("Producer");

 /**
 * The Producer object must be initialized by calling start before it is used, and it can be initialized once 
 * Note: Remember not to call the start method every time a message is sent
 */
 producer.start();
 //The server calls back the Producer to check the success or failure of the local transaction branch
 producer.setTransactionCheckListener( new TransactionCheckListener() {

 public LocalTransactionState checkLocalTransactionState (MessageExt msg) {
 System.out.println("checkLocalTransactionState --"+new String(msg.getBody()));
 return LocalTransactionState.COMMIT_MESSAGE;
 }
 });

 /**
 * The following code indicates that a Producer object can send messages of multiple topics and tags.
 * Note: The send method is called synchronously, as long as no exception is thrown, it will indicate success. However, there may be various states of successful sending. 
 * For example, the message is successfully written to the Master, but the Slave is unsuccessful. In this case, the message is successful, but for individual applications, if the reliability of the message is extremely high, 
 * This situation needs to be handled. In addition, the message may fail to be sent, and the failure retry is handled by the application.
 */

 for (int i = 0; i < 10; i++) {
 try {
 {
 Message msg = new Message("TopicTest1",// topic
 "TagA",// tag
 "OrderID001",// key message keyword, Multiple Keys are separated by KEY_SEPARATOR (used in query messages)
 ("Hello MetaQA").getBytes());// body
 SendResult sendResult = producer.sendMessageInTransaction(msg,new Producer().new MyTransactionExecuter(),"
$$$");
 System.out.println(sendResult);
 }

 {
 Message msg = new Message("TopicTest2",// topic
 "TagB",// tag
 "OrderID0034",// key 消息关键词，多个Key用KEY_SEPARATOR隔开（查询消息使用）
 ("Hello MetaQB").getBytes());// body
 SendResult sendResult = producer.sendMessageInTransaction(msg,new Producer().new MyTransactionExecuter(),"
$$$");
 System.out.println(sendResult);
 }

 {
 Message msg = new Message("TopicTest3",// topic
 "TagC",// tag
 "OrderID061",// key
 ("Hello MetaQC").getBytes());// body
 SendResult sendResult = producer.sendMessageInTransaction(msg,new Producer().new MyTransactionExecuter(),"
$$$");
 System.out.println(sendResult);
 }
 } catch (Exception e) {
 e.printStackTrace();
 }
 TimeUnit.MILLISECONDS.sleep(1000);
 }

 /**
 * When the application exits, call shutdown to clean up resources, close the network connection, and log out from the MetaQ server
 * Note: We recommend that the application call the shutdown method in the exit hook of JBOSS, Tomcat and other containers
 */
 // producer.shutdown() ;
 Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
 public void run() {
 producer.shutdown();
 }
 }));
 System.exit(0);
 }

 //Execute local transaction, by client End callback
 public class MyTransactionExecuter implements LocalTransactionExecuter{

 public LocalTransactionState executeLocalTransactionBranch(Message msg, Object arg) {
 System.out.println("executeLocalTransactionBranch--msg="+new String(msg.getBody()));
 System.out.println("executeLocalTransactionBranch--arg="+arg);
 return LocalTransactionState.COMMIT_MESSAGE;
 }

 }
}
实例代码：http://download.csdn.net/detail/tianwei7518/8138705

rocketMq cluster (source: http://blog.csdn.net/zhu_tianwei/article/details/40949523/)

Guess you like