Table of contents
2. Producer and consumer model
This article focuses on the programming model of RocketMQ. The download, installation and concept can be moved to the other two blog posts of the blogger:
Basic Concepts of RocketMQ__BugMan's Blog-CSDN Blog
RocketMQ download and installation, cluster building nanny-level tutorial_BugMan's Blog-CSDN Blog
1. Environment
RocketMQ version: 4.7.1
Maven depends on:
<dependency>
<groupId>org.apache.rocketmq</groupId>
<artifactId>rocketmq-client</artifactId>
<version>4.7.1</version>
</dependency>
2. Producer and consumer model
The producer has three production modes:
-
Synchronously, the producer waits for the broker's response before going down.
-
Asynchronous, the producer does not wait for the broker's response, and continues to go down. The broker's response triggers a callback function through event monitoring to notify the producer.
-
One-way, the producer just sends, regardless of the broker's response, this mode can't do the remedy after the message is lost
Consumers have two consumption modes:
-
Active pull
-
waiting to push
Producer example:
public static void main(String[] args) throws MQClientException, InterruptedException {
//创建消费者，创建的时候可以指定该消费者属于哪个消费者组
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
//指定name server的地址
producer.setNamesrvAddr("name-server1-ip:9876;name-server2-ip:9876");
producer.start();
//发送一千条信息
for (int i = 0; i < 1000; i++) {
try {
//消息，topic为TopicTest，后面跟的一串是tag
Message msg = new Message("TopicTest" /* Topic */,
"TagA" /* Tag */,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET) /* Message body */
);
//同步发送
SendResult sendResult = producer.send(msg);
/**异步发送，通过自定义回调函数的方式来触发响应
producer.send(msg, new SendCallback() {
@Override
public void onSuccess(SendResult sendResult) {
countDownLatch.countDown();
System.out.printf("%-10d OK %s %n", index, sendResult.getMsgId());
}
@Override
public void onException(Throwable e) {
countDownLatch.countDown();
System.out.printf("%-10d Exception %s %n", index, e);
e.printStackTrace();
}
}**/
System.out.printf("%s%n", sendResult);
} catch (Exception e) {
e.printStackTrace();
Thread.sleep(1000);
}
}
producer.shutdown();
}Consumer example:
push mode:
public static void main(String[] args) throws InterruptedException, MQClientException {
//创建消费者
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_4");
//设置name server
consumer.setNamesrvAddr("name-server1-ip:9876;name-server2-ip:9876");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
consumer.subscribe("TopicTest", "*");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.printf("Consumer Started.%n");
}Pull mode:
private static final Map<MessageQueue, Long> OFFSE_TABLE = new HashMap<MessageQueue, Long>();
public static void main(String[] args) throws MQClientException {
DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("please_rename_unique_group_name_5");
consumer.setNamesrvAddr("127.0.0.1:9876");
consumer.start();
Set<MessageQueue> mqs = consumer.fetchSubscribeMessageQueues("broker-a");
for (MessageQueue mq : mqs) {
System.out.printf("Consume from the queue: %s%n", mq);
SINGLE_MQ:
while (true) {
try {
PullResult pullResult =
consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
System.out.printf("%s%n", pullResult);
putMessageQueueOffset(mq, pullResult.getNextBeginOffset());
switch (pullResult.getPullStatus()) {
case FOUND:
break;
case NO_MATCHED_MSG:
break;
case NO_NEW_MSG:
break SINGLE_MQ;
case OFFSET_ILLEGAL:
break;
default:
break;
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
consumer.shutdown();
}
private static long getMessageQueueOffset(MessageQueue mq) {
Long offset = OFFSE_TABLE.get(mq);
if (offset != null)
return offset;
return 0;
}
private static void putMessageQueueOffset(MessageQueue mq, long offset) {
OFFSE_TABLE.put(mq, offset);
}
}3. Sequential messages
In some scenarios, we need to ensure the order of messages, such as the following scenarios:
1. Register as a member and initialize the points to 10 points
2. Complete the operation of earning points, +5 points
3. Complete the illegal operation, -10 points
5 points left
If the message out of order is 321, the final score is 10 points.
The producer sends the messages to a MessageQueue sequentially, and then the consumer goes to a MessageQueue to get the messages, so that the order of the messages can be guaranteed.
RocketMQ supports producers and consumers to specify MessageQueue during production and consumption, but how to use this to achieve sequential consumption requires developers to write by hand.
Producer:
for (int i = 0; i < 100; i++) {
for(int j=0;i<5;i++) {
//每一个订单用同一个orderId
int orderId = i;
Message msg =
new Message("TopicTest","order_"+orderId, "KEY" + orderId,
("order_" + orderId+"step"+j).getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.send(msg, new MessageQueueSelector() {
/**
* 回调函数
* @param mqs broker中的所有MessageQueue
* @param msg 发送的消息
* @param arg 就是传过来的orderId
* @return
*/
@Override
public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) {
Integer id = (Integer) arg;
//每个订单的message用同一个orderId，必然会选中同一个MessageQueue，自然会顺序存进去
int index = id % mqs.size();
return mqs.get(index);
}
}, orderId);
System.out.printf("%s%n", sendResult);
}
}consumer:
//MessageListenerOrderly这种类型的监听器，会让consumer一直去读一个messagequeue的内容，一直读完为止
consumer.registerMessageListener(new MessageListenerOrderly() {
@Override
public ConsumeOrderlyStatus consumeMessage(List<MessageExt> msgs, ConsumeOrderlyContext context) {
context.setAutoCommit(true);
for (MessageExt msg : msgs) {
System.out.println("收到的消息内容:"+new String(msg.getBody()));
}
return ConsumeOrderlyStatus.SUCCESS;
}
});4. Broadcast message
By default, a message in a topic will only be consumed by one consumer. In broadcast mode, a message in a topic can be consumed by all consumers subscribed to the topic.
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name_1");
consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
//设置为广播模式
consumer.setMessageModel(MessageModel.BROADCASTING);
consumer.subscribe("TopicTest", "TagA || TagC || TagD");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.printf("Broadcast Consumer Started.%n");
}5. Delayed messages
Delay message, that is, specify a residence time of a message in MQ, after how long, after the residence time, consumers can consume the message.
Delayed messages can be used for timing tasks.
In terms of API, on the producer side, just specify a delay level for the message:
//messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
//3即第三个等级，10s
msg.setDelayTimeLevel(3);Obviously, the above delay levels are fixed, if we want to use it for timing tasks, we need to customize the delay level. The open source version of RocketMQ does not support custom latency levels, and only the commercial version (RocketMQ deployed on Alibaba Cloud) supports it.
The open source version of RocketMQ can only change the source code by itself, so this is also the focus of transformation when companies make their own customized RocketMQ
6. Batch messages
RocketMQ supports producers to send messages in batches to reduce the network IO of producers. Batch messages are only related to producers, and consumers still consume one by one.
public class SimpleBatchProducer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("BatchProducerGroupName");
producer.start();
//If you just send messages of no more than 1MiB at a time, it is easy to use batch
//Messages of the same batch should have: same topic, same waitStoreMsgOK and no schedule support
String topic = "BatchTest";
List<Message> messages = new ArrayList<>();
messages.add(new Message(topic, "Tag", "OrderID001", "Hello world 0".getBytes()));
messages.add(new Message(topic, "Tag", "OrderID002", "Hello world 1".getBytes()));
messages.add(new Message(topic, "Tag", "OrderID003", "Hello world 2".getBytes()));
producer.send(messages);
}
}7. Filter messages
Through tags, consumers can consume the news they are interested in under the same topic
producer:
public class TagFilterProducer {
public static void main(String[] args) throws Exception {
DefaultMQProducer producer = new DefaultMQProducer("please_rename_unique_group_name");
producer.start();
String[] tags = new String[] {"TagA", "TagB", "TagC"};
for (int i = 0; i < 60; i++) {
Message msg = new Message("TagFilterTest",
tags[i % tags.length],
"Hello world".getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.send(msg);
System.out.printf("%s%n", sendResult);
}
producer.shutdown();
}consumer:
public class TagFilterConsumer {
public static void main(String[] args) throws InterruptedException, MQClientException, IOException {
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("please_rename_unique_group_name");
consumer.subscribe("TagFilterTest", "TagA || TagC");
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs,
ConsumeConcurrentlyContext context) {
System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
consumer.start();
System.out.printf("Consumer Started.%n");
}
}8. Transactional messages
Transactional messages, that is, messages sent to MQ can be rolled back together with local transactions.
In fact, there will be consistency problems similar to distributed transactions between MQ messages and businesses. For example:
Taking the e-commerce transaction scenario as an example, the core operation of user payment for orders will also involve changes in multiple subsystems such as downstream logistics delivery, point changes, and shopping cart status clearing. Obviously, the calls of these steps should be in a transaction, and these calls can be made by MQ, so the problem arises. Once one of these businesses fails, the other businesses should fail at the same time, but what if the message has already been sent?
This is where transactional messages come in handy.
When the producer of a transaction message sends a message, it will convert the message into a half (half message) and store it in a Topic RMQ_SYS_TRANS_HALF_TOPIC inside RocketMQ. This Topic is invisible to consumers. When the local transaction is executed successfully, it will Send a commit signal to MQ, and MQ will convert the message to the original Topic. When the execution of the local transaction fails, it will send a roll_back signal to MQ, and MQ will discard the corresponding message. If the local transaction is not completed, you can send an unknown signal to MQ. After receiving the unknown signal, MQ will make the corresponding message wait, and periodically check back the message whose status is unknown.
Customize the commit logic and check logic of local transactions:
public class TransactionListenerImpl implements TransactionListener {
private AtomicInteger transactionIndex = new AtomicInteger(0);
private ConcurrentHashMap<String, Integer> localTrans = new ConcurrentHashMap<>();
@Override
public LocalTransactionState executeLocalTransaction(Message msg, Object arg) {
int value = transactionIndex.getAndIncrement();
int status = value % 3;
localTrans.put(msg.getTransactionId(), status);
return LocalTransactionState.UNKNOW;
}
@Override
public LocalTransactionState checkLocalTransaction(MessageExt msg) {
Integer status = localTrans.get(msg.getTransactionId());
if (null != status) {
switch (status) {
case 0:
return LocalTransactionState.UNKNOW;
case 1:
return LocalTransactionState.COMMIT_MESSAGE;
case 2:
return LocalTransactionState.ROLLBACK_MESSAGE;
default:
return LocalTransactionState.COMMIT_MESSAGE;
}
}
return LocalTransactionState.COMMIT_MESSAGE;
}
}Producer:
public class TransactionProducer {
public static void main(String[] args) throws MQClientException, InterruptedException {
TransactionListener transactionListener = new TransactionListenerImpl();
TransactionMQProducer producer = new TransactionMQProducer("please_rename_unique_group_name");
ExecutorService executorService = new ThreadPoolExecutor(2, 5, 100, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(2000), new ThreadFactory() {
@Override
public Thread newThread(Runnable r) {
Thread thread = new Thread(r);
thread.setName("client-transaction-msg-check-thread");
return thread;
}
});
producer.setExecutorService(executorService);
producer.setTransactionListener(transactionListener);
producer.start();
String[] tags = new String[] {"TagA", "TagB", "TagC", "TagD", "TagE"};
for (int i = 0; i < 10; i++) {
try {
Message msg =
new Message("TopicTest1234", tags[i % tags.length], "KEY" + i,
("Hello RocketMQ " + i).getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = producer.sendMessageInTransaction(msg, null);
System.out.printf("%s%n", sendResult);
Thread.sleep(10);
} catch (MQClientException | UnsupportedEncodingException e) {
e.printStackTrace();
}
}
for (int i = 0; i < 100000; i++) {
Thread.sleep(1000);
}
producer.shutdown();
}
}