RocketMQ retry queue creation

Get into the habit of writing together! This is the 8th day of my participation in the "Nuggets Daily New Plan·April Update Challenge", click to view the details of the event

RocketMQ retry queue creation

consumerWhen the consumption fails, the message can be re-delivered back to the RocketMQspecial one topic: retry topic. If the consumption fails for many times, then the message will be delivered dead topic.

retry topic

Retry topicIt is consumer groupnamed according to the specific value: %RETRY%${group_name}, that is, group nameadding a fixed prefix. Therefore, for a group, it retry topicis a fixed topic, and messages that the group subscribes to other topics and cause consumption failure will be delivered to this topic. Then, there is some questions:

When was the Retry topic created?
How many queues are there for Retry topic?
Rocketmq has master-slave, will there be retryTopic on slave?

Users create topics on Broker.

The command line supports the creation topicmethod : specify brokeror specify the cluster. 2. Registerbroker your own specific implementation in 3. Nameserver updates routing information 4. Consumer reads topic subscription information 5. Consumer pulls messages through the broker of the routing table.nameservertopicBrockerController#registerBrokerAll

registerBrokerAll code implementation

public synchronized void registerBrokerAll(final boolean checkOrderConfig, boolean oneway, boolean forceRegister) {
        TopicConfigSerializeWrapper topicConfigWrapper = this.getTopicConfigManager().buildTopicConfigSerializeWrapper();

        if (!PermName.isWriteable(this.getBrokerConfig().getBrokerPermission())
            || !PermName.isReadable(this.getBrokerConfig().getBrokerPermission())) {
            ConcurrentHashMap<String, TopicConfig> topicConfigTable = new ConcurrentHashMap<String, TopicConfig>();
            for (TopicConfig topicConfig : topicConfigWrapper.getTopicConfigTable().values()) {
                TopicConfig tmp =
                    new TopicConfig(topicConfig.getTopicName(), topicConfig.getReadQueueNums(), topicConfig.getWriteQueueNums(),
                        this.brokerConfig.getBrokerPermission());
                topicConfigTable.put(topicConfig.getTopicName(), tmp);
            }
            topicConfigWrapper.setTopicConfigTable(topicConfigTable);
        }

        if (forceRegister || needRegister(this.brokerConfig.getBrokerClusterName(),
            this.getBrokerAddr(),
            this.brokerConfig.getBrokerName(),
            this.brokerConfig.getBrokerId(),
            this.brokerConfig.getRegisterBrokerTimeoutMills())) {
            doRegisterBrokerAll(checkOrderConfig, oneway, topicConfigWrapper);
        }
    }
复制代码

It can be seen that there are two conditions to judge whether registration is required:

Whether to enable forceRegister (default enabled)
Whether dataVersion has changed

Code:

 if (forceRegister || needRegister(this.brokerConfig.getBrokerClusterName(),
            this.getBrokerAddr(),
            this.brokerConfig.getBrokerName(),
            this.brokerConfig.getBrokerId(),
            this.brokerConfig.getRegisterBrokerTimeoutMills())) {
            doRegisterBrokerAll(checkOrderConfig, oneway, topicConfigWrapper);
        }
复制代码

needRegister contains the following code:

 nameServerDataVersion = DataVersion.decode(body, DataVersion.class);
                                        if (!topicConfigWrapper.getDataVersion().equals(nameServerDataVersion)) {
                                            changed = true;
                                        }
复制代码

dataVerson

dataVersion represents the version of the local data, which is composed of timestamps and counters, and is updated when the data changes (create, delete, topic).

public class DataVersion extends RemotingSerializable {
    private long timestamp = System.currentTimeMillis();
    private AtomicLong counter = new AtomicLong(0);

    public void assignNewOne(final DataVersion dataVersion) {
        this.timestamp = dataVersion.timestamp;
        this.counter.set(dataVersion.counter.get());
    }

    public void nextVersion() {
        this.timestamp = System.currentTimeMillis();
        this.counter.incrementAndGet();
    }
 ...
}
复制代码

The required composition DataVesioncan be seen .timestamp+counter

When to create a retry topic?

Consumer clientand brokerthe heartbeat mechanism of
Mechanism for sending retry messages

heartbeat

ClientManageProcessor#heartBeatThe heartbeat function is as follows:

public RemotingCommand heartBeat(ChannelHandlerContext ctx, RemotingCommand request) {
        RemotingCommand response = RemotingCommand.createResponseCommand(null);
        HeartbeatData heartbeatData = HeartbeatData.decode(request.getBody(), HeartbeatData.class);
        ClientChannelInfo clientChannelInfo = new ClientChannelInfo(
            ctx.channel(),
            heartbeatData.getClientID(),
            request.getLanguage(),
            request.getVersion()
        );

        for (ConsumerData data : heartbeatData.getConsumerDataSet()) {
            SubscriptionGroupConfig subscriptionGroupConfig =
                this.brokerController.getSubscriptionGroupManager().findSubscriptionGroupConfig(
                    data.getGroupName());
            boolean isNotifyConsumerIdsChangedEnable = true;
            if (null != subscriptionGroupConfig) {
                isNotifyConsumerIdsChangedEnable = subscriptionGroupConfig.isNotifyConsumerIdsChangedEnable();
                int topicSysFlag = 0;
                if (data.isUnitMode()) {
                    topicSysFlag = TopicSysFlag.buildSysFlag(false, true);
                }
                String newTopic = MixAll.getRetryTopic(data.getGroupName());
                // topic 更新到 topic Config 这个内存对象，然后还会触发
                this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(
                    newTopic,
                    subscriptionGroupConfig.getRetryQueueNums(),
                    PermName.PERM_WRITE | PermName.PERM_READ, topicSysFlag);
            }

            boolean changed = this.brokerController.getConsumerManager().registerConsumer(
                data.getGroupName(),
                clientChannelInfo,
                data.getConsumeType(),
                data.getMessageModel(),
                data.getConsumeFromWhere(),
                data.getSubscriptionDataSet(),
                isNotifyConsumerIdsChangedEnable
            );

            if (changed) {
                log.info("registerConsumer info changed {} {}",
                    data.toString(),
                    RemotingHelper.parseChannelRemoteAddr(ctx.channel())
                );
            }
        }

        for (ProducerData data : heartbeatData.getProducerDataSet()) {
            this.brokerController.getProducerManager().registerProducer(data.getGroupName(),
                clientChannelInfo);
        }
        response.setCode(ResponseCode.SUCCESS);
        response.setRemark(null);
        return response;
    }
复制代码

Contains retry topicinformation , getRetryTopicinformation, and then registers the retry topicinformation to nameserver.

   String newTopic = MixAll.getRetryTopic(data.getGroupName());
                this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(
                    newTopic,
                    subscriptionGroupConfig.getRetryQueueNums(),
                    PermName.PERM_WRITE | PermName.PERM_READ, topicSysFlag);
复制代码

consumer clientRe-request nameserver, you can get retry topicthe information. retryQueueNumsThe default is 1 , which retry topicmeans brokerthere is one on each by default queue. The retry topic provides the implementation of the heartbeat mechanism. On the client side, the client and all brokers perform long heartbeat links. Therefore broker masterand broker slavehave both retry topic.

delivery retry message

SendMessageProcessor#asyncConsumerSendMsgBack. Similar to the heartbeat mechanism, it is createTopicInSendMessageBackMethodimplemented , so if the retry topic is not created, it will also trigger brokerthe nameserverregistration. But as far as the client implementation is concerned, the heartbeat will be sent first, and then the message will be pulled for delivery. asyncConsumerSendMsgBackmethod to see the retry queue.

String newTopic = MixAll.getRetryTopic(requestHeader.getGroup());
        int queueIdInt = ThreadLocalRandom.current().nextInt(99999999) % subscriptionGroupConfig.getRetryQueueNums();
        int topicSysFlag = 0;
        if (requestHeader.isUnitMode()) {
            topicSysFlag = TopicSysFlag.buildSysFlag(false, true);
        }

        TopicConfig topicConfig = this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(
            newTopic,
            subscriptionGroupConfig.getRetryQueueNums(),
            PermName.PERM_WRITE | PermName.PERM_READ, topicSysFlag);
复制代码