Source code analysis, why does RocketMQ have strange phenomena when setting different tags in the same consumer group?

1. The problem reappears

1. Description

Consumers of two identical Consumer Groups subscribe to the same topic, but different tags, Consumer1 subscribes to Topic tag1, Consumer2 subscribes to Topic tag2, and then start them separately. At this time, 10 pieces of data are sent to tag1 of Topic, and 10 pieces of data are sent to tag2 of Topic. Visual inspection should be that Consumer1 and Consumer2 have received 10 corresponding messages respectively. The result is that only Consumer2 received the message, and only 4-6 messages were received, which is not fixed.

2. Code

2.1、Consumer

public class Consumer {
    public static void main(String[] args) throws MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("test-consumer");
        consumer.setNamesrvAddr("124.57.180.156:9876");
        consumer.subscribe("TopicTest2","tag1");
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            public ConsumeConcurrentlyStatus consumeMessage(
                    List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                MessageExt msg = msgs.get(0);
                System.out.println(msg.getTags());
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.out.println("ConsumerStarted.");
    }
}

Start the Consumer subscribed to the tag1 tag of TopicTest2, and then change tag1 to tag2 to start the Consumer again. This is equivalent to starting two Consumer processes, one subscribes to the tag1 tag of TopicTest2, and the other subscribes to the tag2 tag of TopicTest2.

2.2、Producer

public class Producer {
    public static void main(String[] args) throws MQClientException {
        final DefaultMQProducer producer = new DefaultMQProducer("test-producer");
        producer.setNamesrvAddr("124.57.180.156:9876");
        producer.start();

        for (int i = 0; i < 10; i++){
            try {
                Message msg = new Message("TopicTest2", "tag1", ("Hello tag1 - "+i).getBytes());
                SendResult sendResult = producer.send(msg);
                System.out.println(sendResult);
            }catch(Exception e) {
                e.printStackTrace();
            }
        }
    }
}

Start the Producer and send 10 messages to tag1 of TopicTest2. Change tag1 to tag2 again, and then start the Producer again to send, so that there are 10 messages under tag1 of TopicTest2, and 10 messages under tag2 of TopicTest2.

3. Results

After both the Consumer and Producer are started, the following are found:

• Producer sent 20 messages normally.
• Consumer1 did not consume the data under tag1
• Consumer2 consumes half (not necessarily a few, sometimes 5, sometimes 6) messages.

Second, the answer to the question

• First of all, this is decided by Broker, not by Consumer

I have read an article before and it is justified. It was written on the Consumer side. I also posted the debug source code, saying that the latter covers the former, but I want to say: You started two independent Consumers, that is two An independent process, there is no problem of covering or not covering at all, it is independent. Just one JVM. It's not a shared JVM, how can it be overwritten?

• The consumer sends a heartbeat to the broker, and the broker stores it in the consumerTable (that is, a Map) after receiving it. The key is GroupName and the value is ConsumerGroupInfo.
• ConsumerGroupInfo contains topic and other information, but the problem lies in the previous step. The key is groupName. If you are the same as GroupName, the last Consumer received by the Broker heartbeat will overwrite the former. It is equivalent to the following code:

map.put(groupName, ConsumerGroupInfo);

In this way, the same key must be overwritten. So Consumer1 will not receive any messages, but why Consumer2 has only received half (not fixed) messages?

That's because: you are consuming in cluster mode, it will load balance and distribute to each node to consume, so half of the messages (not a fixed number) ran to Consumer1, as a result Consumer1 subscribes to tag1, so there will be no output. If you change to BROADCASTING, the latter will definitely receive all the messages instead of half, because the broadcast is to broadcast all the Consumers.

Three, source code verification

1. Call chain

# 核心在于如下这个方法
org.apache.rocketmq.broker.client.ConsumerManager#registerConsumer()
    
# 关键调用链如下
# 入口是Broker启动的时候
org.apache.rocketmq.broker.BrokerStartup#start()
org.apache.rocketmq.broker.BrokerController#start()
org.apache.rocketmq.remoting.netty.NettyRemotingServer#start() 
org.apache.rocketmq.remoting.netty.NettyRemotingServer#prepareSharableHandlers()
org.apache.rocketmq.remoting.netty.NettyRemotingServer.NettyServerHandler#channelRead0()
org.apache.rocketmq.remoting.netty.NettyRemotingAbstract#processMessageReceived()
org.apache.rocketmq.remoting.netty.NettyRemotingAbstract#processRequestCommand()
org.apache.rocketmq.broker.processor.ClientManageProcessor#processRequest()
org.apache.rocketmq.broker.processor.ClientManageProcessor#heartBeat()
org.apache.rocketmq.broker.client.ConsumerManager#registerConsumer()

2. Source code

2.1、registerConsumer

/**
 * Consumer信息
 */
public class ConsumerGroupInfo {
    // 组名
    private final String groupName;
    // topic信息，比如topic、tag等
    private final ConcurrentMap<String/* Topic */, SubscriptionData> subscriptionTable =
        new ConcurrentHashMap<String, SubscriptionData>();
    // 客户端信息，比如clientId等
    private final ConcurrentMap<Channel, ClientChannelInfo> channelInfoTable =
        new ConcurrentHashMap<Channel, ClientChannelInfo>(16);
    // PULL/PUSH
    private volatile ConsumeType consumeType;
    // 消费模式：BROADCASTING/CLUSTERING
    private volatile MessageModel messageModel;
    // 消费到哪了
    private volatile ConsumeFromWhere consumeFromWhere;
}

/**
 * 通过心跳将Consumer信息注册到Broker端。
 */
public boolean registerConsumer(final String group, final ClientChannelInfo clientChannelInfo,
        ConsumeType consumeType, MessageModel messageModel, ConsumeFromWhere consumeFromWhere,
        final Set<SubscriptionData> subList, boolean isNotifyConsumerIdsChangedEnable) {

    // consumerTable：维护所有的Consumer
    ConsumerGroupInfo consumerGroupInfo = this.consumerTable.get(group);
    // 如果没有Consumer，则put到map里
    if (null == consumerGroupInfo) {
        ConsumerGroupInfo tmp = new ConsumerGroupInfo(group, consumeType, messageModel, consumeFromWhere);
        // put到map里
        ConsumerGroupInfo prev = this.consumerTable.putIfAbsent(group, tmp);
        consumerGroupInfo = prev != null ? prev : tmp;
    }

    // 更新Consumer信息，客户端信息
    boolean r1 =
        consumerGroupInfo.updateChannel(clientChannelInfo, consumeType, messageModel,
                                        consumeFromWhere);
    // 更新订阅Topic信息
    boolean r2 = consumerGroupInfo.updateSubscription(subList);

    if (r1 || r2) {
        if (isNotifyConsumerIdsChangedEnable) {
            this.consumerIdsChangeListener.handle(ConsumerGroupEvent.CHANGE, group, consumerGroupInfo.getAllChannel());
        }
    }

    this.consumerIdsChangeListener.handle(ConsumerGroupEvent.REGISTER, group, subList);

    return r1 || r2;
}

From this step, it can be seen that the consumer information is stored in the map (consumerTable) with groupName as the key and ConsumerGroupInfo as the value. It is obvious that the latter will definitely overwrite the former because the keys are the same. The latter’s tag is tag2, which must cover the former’s tag1, which is stored in the subscriptionTable of ConsumerGroupInfo.

private final ConcurrentMap<String/* Topic */, SubscriptionData> subscriptionTable =
 new ConcurrentHashMap<String, SubscriptionData>();

• 1
• 2

SubscriptionData contains topic and other information

public class SubscriptionData implements Comparable<SubscriptionData> {
 // topic
 private String topic;
 private String subString;
 // tags
 private Set<String> tagsSet = new HashSet<String>();
 private Set<Integer> codeSet = new HashSet<Integer>();
}

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

2.2 Two problems

1. How are topics, tags and other information covered?

boolean r1 = consumerGroupInfo.updateChannel(clientChannelInfo, consumeType, messageModel,consumeFromWhere);

/**
 * 其实很简单，就是以topic为key，SubscriptionData为value。而SubscriptionData里包含了tags信息，所以直接覆盖掉
 */
public boolean updateSubscription(final Set<SubscriptionData> subList) {
    for (SubscriptionData sub : subList) {
        SubscriptionData old = this.subscriptionTable.get(sub.getTopic());
        if (old == null) {
            SubscriptionData prev = this.subscriptionTable.putIfAbsent(sub.getTopic(), sub);
        } else if (sub.getSubVersion() > old.getSubVersion()) {
            this.subscriptionTable.put(sub.getTopic(), sub);
        }
    }
}

Wait, there seems to be a new discovery hereConsumerGroupInfo#subscriptionTable

// {@link org.apache.rocketmq.broker.client.ConsumerGroupInfo#subscriptionTable}
private final ConcurrentMap<String/* Topic */, SubscriptionData> subscriptionTable =
        new ConcurrentHashMap<String, SubscriptionData>();

There can be a windfall that topic is the key of the map. Isn't it possible that one Consumer can subscribe to multiple topics? Yes, it can be found that there is nothing wrong with this source code, and I have also tested it.

2. If you look at it this way, there will only be one process on the Consumer side, because the same group, registration will be overwritten?

Brother, pay attention to the channelInfoTable in ConsumerGroupInfo

// 客户端信息，比如clientId等
private final ConcurrentMap<Channel, ClientChannelInfo> channelInfoTable =
    new ConcurrentHashMap<Channel, ClientChannelInfo>(16);

ClientChannelInfo contains information such as clientId and represents a Consumer. The registration method is:

boolean r2 = consumerGroupInfo.updateSubscription(subList);

/**
 * 下面是删减后的代码，其实就是以Channel作为key，每个Consumer的Channel是不一样的。所以能存多个Consumer客户端
 */
public boolean updateChannel(final ClientChannelInfo infoNew, ConsumeType consumeType,
        MessageModel messageModel, ConsumeFromWhere consumeFromWhere) {
    ClientChannelInfo infoOld = this.channelInfoTable.get(infoNew.getChannel());
    if (null == infoOld) {
        ClientChannelInfo prev = this.channelInfoTable.put(infoNew.getChannel(), infoNew);
    }
}

•