Source of some content https://www.jianshu.com/p/453c6e7ff81c
There are 4 default teams in rocketmq. When sending messages, the messages of the same group need to be sent to the corresponding mq in order. The messages of the same group are consumed in order, and messages of different groups can be consumed in parallel.
Let's take a look at the producer's code:
package com.alibaba.rocketmq.example.message.order; import com.alibaba.rocketmq.client.exception.MQBrokerException; import com.alibaba.rocketmq.client.exception.MQClientException; import com.alibaba.rocketmq.client.producer.DefaultMQProducer; import com.alibaba.rocketmq.client.producer.MessageQueueSelector; import com.alibaba.rocketmq.client.producer.SendResult; import com.alibaba.rocketmq.common.message.Message; import com.alibaba.rocketmq.common.message.MessageQueue; import com.alibaba.rocketmq.remoting.exception.RemotingException; import java.text.SimpleDateFormat; import java.util.Date; import java.util.List; /** * @author : Jixiaohu * @Date : 2018-04-19. * @Time : 9:20. * @Description : */ public class Producer { public static void main(String[] args) throws MQClientException, InterruptedException, MQBrokerException { String groupName = "order_producer"; DefaultMQProducer producer = new DefaultMQProducer(groupName); producer.setNamesrvAddr("192.168.1.114:9876;192.168.2.2:9876"); producer.start(); SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); String dateStr = sdf.format(new Date()); try { for (int i = 1; i <= 3; i++) { String body = dateStr + "Hello RoctetMq : " + i; Message msg = new Message("Topic1", "Tag1", "Key" + i, body.getBytes()); SendResult sendResult = producer.send(msg, new MessageQueueSelector() { @Override public MessageQueue select(List<MessageQueue> list, Message message, Object o) { Integer id = (Integer) o; return list.get(id); } }, 0); // 0 is the subscript of the queue System.out.println(sendResult); } for (int i = 1; i <= 3; i++) { String body = dateStr + "Hello RoctetMq : " + i; Message msg = new Message("Topic1", "Tag1", "Key" + i, body.getBytes()); SendResult sendResult = producer.send(msg, new MessageQueueSelector() { @Override public MessageQueue select(List<MessageQueue> list, Message message, Object o) { Integer id = (Integer) o; return list.get(id); } }, 1); // 1 is the subscript of the queue System.out.println(sendResult); } for (int i = 1; i <= 3; i++) { String body = dateStr + "Hello RoctetMq : " + i; Message msg = new Message("Topic1", "Tag1", "Key" + i, body.getBytes()); SendResult sendResult = producer.send(msg, new MessageQueueSelector() { @Override public MessageQueue select(List<MessageQueue> list, Message message, Object o) { Integer id = (Integer) o; return list.get(id); } }, 2); // 2 is the subscript of the queue System.out.println(sendResult); } for (int i = 1; i <= 3; i++) { String body = dateStr + "Hello RoctetMq : " + i; Message msg = new Message("Topic1", "Tag1", "Key" + i, body.getBytes()); SendResult sendResult = producer.send(msg, new MessageQueueSelector() { @Override public MessageQueue select(List<MessageQueue> list, Message message, Object o) { Integer id = (Integer) o; return list.get(id); } }, 3); //3 is the subscript of the queue System.out.println(sendResult); } for (int i = 1; i <= 3; i++) { String body = dateStr + "Hello RoctetMq : " + i; Message msg = new Message("Topic1", "Tag1", "Key" + i, body.getBytes()); SendResult sendResult = producer.send(msg, new MessageQueueSelector() { @Override public MessageQueue select(List<MessageQueue> list, Message message, Object o) { Integer id = (Integer) o; return list.get(id); } }, 4); //4 is the subscript of the queue System.out.println(sendResult); } for (int i = 1; i <= 3; i++) { String body = dateStr + "Hello RoctetMq : " + i; Message msg = new Message("Topic1", "Tag1", "Key" + i, body.getBytes()); SendResult sendResult = producer.send(msg, new MessageQueueSelector() { @Override public MessageQueue select(List<MessageQueue> list, Message message, Object o) { Integer id = (Integer) o; return list.get(id); } }, 5); //5 is the subscript of the queue System.out.println(sendResult); } } catch (RemotingException e) { e.printStackTrace (); Thread.sleep(1000); } producer.shutdown(); } }
Multiple groups of messages are sent here, and the order of each group of messages is 1, 2, and 3, respectively.
Let's look at consumer1 and consumer2. Because of the sequential consumption, it should be noted that the listeners of these two consumers are MessageListenerOrderly. The codes of the two are the same. I will only show the code of one consumer here.
package com.alibaba.rocketmq.example.message.order; import com.alibaba.rocketmq.client.consumer.DefaultMQPushConsumer; import com.alibaba.rocketmq.client.consumer.listener.ConsumeOrderlyContext; import com.alibaba.rocketmq.client.consumer.listener.ConsumeOrderlyStatus; import com.alibaba.rocketmq.client.consumer.listener.MessageListenerOrderly; import com.alibaba.rocketmq.common.consumer.ConsumeFromWhere; import com.alibaba.rocketmq.common.message.MessageExt; import java.util.List; import java.util.Random; import java.util.concurrent.TimeUnit; /** * @author : Jixiaohu * @Date : 2018-04-23. * @Time : 9:35. * @Description : sequential message consumption */ public class Consumer1 { public Consumer1() throws Exception { String groupName = "order_producer"; DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(groupName); consumer.setNamesrvAddr("192.168.1.114:9876;192.168.2.2:9876"); /** * Set whether the Consumer starts to consume from the head of the queue or the tail of the queue when the Consumer is started for the first time * If it is not the first time to start, then continue to consume according to the position of the last consumption */ consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET); // Subscribed topics, and filtered label content consumer.subscribe("Topic1", "*" ); // Register listeners consumer.registerMessageListener( new Listener()); consumer.start(); System.out.println("Consumer1 Started."); } class Listener implements MessageListenerOrderly { private Random random = new Random(); @Override public ConsumeOrderlyStatus consumeMessage(List<MessageExt> list, ConsumeOrderlyContext context) { // 设置自动提交 context.setAutoCommit(true); for (MessageExt msg : list) { System.out.println(msg + "，context" + new String(msg.getBody())); } try { TimeUnit.SECONDS.sleep(random.nextInt(1)); } catch (InterruptedException e) { e.printStackTrace (); } return ConsumeOrderlyStatus.SUCCESS; } } public static void main(String[] args) throws Exception { new Consumer1(); } }
It is still in the order of starting the consumer first, and the order of starting the producer.
Here's a look at the console information
There are 6 groups of messages in total, 4 groups of messages are received on broker-a, 2 groups of messages are received on broker-b, and 3 messages of the same group will be sent to the same queue of the same broker, so as to ensure sequential consumption,
Let's take a look at the print results of consumer1 and consumer2
Since sequential consumption can only be single-threaded,
It can be seen that the queueids here are all 3 and 3 prints, and there will be no alternation. Let's take a look at the consumption order of a group of messages.
It can be seen that messages are consumed in the order in which they are sent, and the print result of consumer2 is similar, but consumer2 consumes 6 messages.
In this way, the sequential consumption of rocket is realized. Although sequential consumption is realized, there will be the problem of duplicate data due to network communication.
For the problem of duplicate data, rocket does not provide a solution, and let the business side solve it by itself. There are two main methods:
- The business logic for processing messages on the consumer side remains idempotent
- Ensure that each message has a unique number and ensure that the message processing is successful and the log of the deduplication table appears at the same time
Item 1 is easy to understand, as long as idempotency is maintained, no matter how many duplicate messages come, the final result will be the same. The second principle is to use a log table to record the ID of the message that has been processed successfully. If the newly arrived message ID is already in the log table, then this message will not be processed.
The first solution, obviously, should be implemented on the consumer side, and it does not belong to the function to be implemented by the message system. Item 2 can be implemented by the message system or by the business side. Under normal circumstances, the probability of duplicate messages is actually very small. If it is implemented by the message system, it will definitely affect the throughput and high availability of the message system. Therefore, it is better for the business side to handle the problem of message duplication by itself. This is also The reason why RocketMQ doesn't solve the problem of duplicate messages.
RocketMQ does not guarantee that messages are not repeated. If your business needs to ensure strict non-duplication of messages, you need to deduplicate them on the business side.