Achieve rocketMQ retry message

consumer consumption fails, the message will be sent to re-% RETRY% + consumerGroup, this news will retry after a certain time, true to retry topic.

broker processing a message sent to the retry topic:

org.apache.rocketmq.broker.processor.SendMessageProcessor#consumerSendMsgBack

News consumption exceeds the maximum number of client configuration or sent directly to the dead-letter queue, send messages to put the dead letter queue, otherwise the message retry topic, although the news seems to be written directly% RETRY% + consumerGroup

But in fact putMessage time, will write messages SCHEDULE_TOPIC_XXXX

// org.apache.rocketmq.store.CommitLog#putMessage
if (msg.getDelayTimeLevel() > 0) {
    if (msg.getDelayTimeLevel() > this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel()) {
        msg.setDelayTimeLevel(this.defaultMessageStore.getScheduleMessageService().getMaxDelayLevel());
    }

    topic = ScheduleMessageService.SCHEDULE_TOPIC;
    queueId = ScheduleMessageService.delayLevel2QueueId(msg.getDelayTimeLevel());

    // Backup real topic, queueId
    MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_TOPIC, msg.getTopic());
    MessageAccessor.putProperty(msg, MessageConst.PROPERTY_REAL_QUEUE_ID, String.valueOf(msg.getQueueId()));
    msg.setPropertiesString(MessageDecoder.messageProperties2String(msg.getProperties()));

    msg.setTopic(topic);
    msg.setQueueId(queueId);
}

This very interesting topic SCHEDULE_TOPIC_XXXX, broker does not explicitly create this topic, i.e. broker nameserver and metadata is not saved in the broker, topic of data is written normally commitLog, a level corresponding to a delay queue, queueId = delayLevel - 1, so SCHEDULE_TOPIC_XXXX have up to 18 queue.

// org.apache.rocketmq.store.config.MessageStoreConfig#messageDelayLevel
private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";

A total of 18 delayLevel

// org.apache.rocketmq.common.subscription.SubscriptionGroupConfig#retryMaxTimes
private int retryMaxTimes = 16;

This parameter consumer can configure the default 16

ScheduleMessageService initialization delayLevelTable, the key is delayLevel, the delay value is the number of milliseconds, from 1 to 18

// org.apache.rocketmq.store.schedule.ScheduleMessageService#load
public boolean load() {
    boolean result = super.load();
    result = result && this.parseDelayLevel();
    return result;
}
public boolean parseDelayLevel() {
    HashMap<String, Long> timeUnitTable = new HashMap<String, Long>();
    timeUnitTable.put("s", 1000L);
    timeUnitTable.put("m", 1000L * 60);
    timeUnitTable.put("h", 1000L * 60 * 60);
    timeUnitTable.put("d", 1000L * 60 * 60 * 24);

    String levelString = this.defaultMessageStore.getMessageStoreConfig().getMessageDelayLevel();
    try {
        String[] levelArray = levelString.split(" ");
        for (int i = 0; i < levelArray.length; i++) {
            String value = levelArray[i];
            String ch = value.substring(value.length() - 1);
            Long tu = timeUnitTable.get(ch);

            int level = i + 1;
            if (level > this.maxDelayLevel) {
                this.maxDelayLevel = level;
            }
            long num = Long.parseLong(value.substring(0, value.length() - 1));
            long delayTimeMillis = tu * num;
            this.delayLevelTable.put(level, delayTimeMillis);
        }
    } catch (Exception e) {
        log.error("parseDelayLevel exception", e);
        log.info("levelString String = {}", levelString);
        return false;
    }

    return true;
}

ScheduleMessageService created for each level a timed task, traversing consume queue, to determine whether the message expiration, expired messages are written to put the real topic of commitLog

// org.apache.rocketmq.store.schedule.ScheduleMessageService#start
public void start() {
    for (Map.Entry<Integer, Long> entry : this.delayLevelTable.entrySet()) {
        Integer level = entry.getKey();
        Long timeDelay = entry.getValue();
        Long offset = this.offsetTable.get(level);
        if (null == offset) {
            offset = 0L;
        }

        if (timeDelay != null) {
            this.timer.schedule(new DeliverDelayedMessageTimerTask(level, offset), FIRST_DELAY_TIME);
        }
    }

    // 定期持久化处理完的 queue offset 到 delayOffset.json 文件中
    this.timer.scheduleAtFixedRate(new TimerTask() {

        @Override
        public void run() {
            try {
                ScheduleMessageService.this.persist();
            } catch (Throwable e) {
                log.error("scheduleAtFixedRate flush exception", e);
            }
        }
    }, 10000, this.defaultMessageStore.getMessageStoreConfig().getFlushDelayOffsetInterval());
}