一起养成写作习惯！这是我参与「掘金日新计划 · 4 月更文挑战」的第8天，点击查看活动详情
RocketMQ 重试队列创建
consumer 在消费失败的时候, 可以将消息重新投递回 RocketMQ 特殊的topic: retry topic, 如果消费多次，依旧消费失败, 那么, 这条消息将会被投递到dead topic.
retry topic
Retry topic 是根据consumer group 命名的, 具体的值: %RETRY%${group_name}, 也就是在group name添加一个固定前缀. 因此，对于一个group 而言, retry topic 就是固定的topic, group 订阅其他topic导致消费失败的消息都会投递到这个topic中. 那么, 就有几个问题:
- Retry topic 是什么时候创建的？
- Retry topic 的queue有多少个？
- rocketmq是有master-slave的，slave上也会有retryTopic吗？
- 用户在 Broker 创建 topic。
命令行支持创建 topic 方式：指定 broker 或者指定集群。 2. broker 向 nameserver注册自己持有的 topic 具体实现在 BrockerController#registerBrokerAll 3. nameserver 更新路由信息 4. consumer 读取 topic 订阅信息 5. consumer 通过路由表的 broker ，拉取消息。
registerBrokerAll 代码实现
public synchronized void registerBrokerAll(final boolean checkOrderConfig, boolean oneway, boolean forceRegister) {
TopicConfigSerializeWrapper topicConfigWrapper = this.getTopicConfigManager().buildTopicConfigSerializeWrapper();
if (!PermName.isWriteable(this.getBrokerConfig().getBrokerPermission())
|| !PermName.isReadable(this.getBrokerConfig().getBrokerPermission())) {
ConcurrentHashMap<String, TopicConfig> topicConfigTable = new ConcurrentHashMap<String, TopicConfig>();
for (TopicConfig topicConfig : topicConfigWrapper.getTopicConfigTable().values()) {
TopicConfig tmp =
new TopicConfig(topicConfig.getTopicName(), topicConfig.getReadQueueNums(), topicConfig.getWriteQueueNums(),
this.brokerConfig.getBrokerPermission());
topicConfigTable.put(topicConfig.getTopicName(), tmp);
}
topicConfigWrapper.setTopicConfigTable(topicConfigTable);
}
if (forceRegister || needRegister(this.brokerConfig.getBrokerClusterName(),
this.getBrokerAddr(),
this.brokerConfig.getBrokerName(),
this.brokerConfig.getBrokerId(),
this.brokerConfig.getRegisterBrokerTimeoutMills())) {
doRegisterBrokerAll(checkOrderConfig, oneway, topicConfigWrapper);
}
}
复制代码可以看到，是否需要注册有两个条件判断:
- 是否开启 forceRegister （默认开启）
- dataVersion 是否变更
代码:
if (forceRegister || needRegister(this.brokerConfig.getBrokerClusterName(),
this.getBrokerAddr(),
this.brokerConfig.getBrokerName(),
this.brokerConfig.getBrokerId(),
this.brokerConfig.getRegisterBrokerTimeoutMills())) {
doRegisterBrokerAll(checkOrderConfig, oneway, topicConfigWrapper);
}
复制代码needRegister 包含代码如下：
nameServerDataVersion = DataVersion.decode(body, DataVersion.class);
if (!topicConfigWrapper.getDataVersion().equals(nameServerDataVersion)) {
changed = true;
}
复制代码dataVerson
dataVersion 表示本地数据的版本，由时间戳和计数器组层，当数据发生变更（创建，删除，topic）就会更新。
public class DataVersion extends RemotingSerializable {
private long timestamp = System.currentTimeMillis();
private AtomicLong counter = new AtomicLong(0);
public void assignNewOne(final DataVersion dataVersion) {
this.timestamp = dataVersion.timestamp;
this.counter.set(dataVersion.counter.get());
}
public void nextVersion() {
this.timestamp = System.currentTimeMillis();
this.counter.incrementAndGet();
}
...
}
复制代码可以看到 DataVesion 需要 timestamp+counter的组成。
什么时候创建retry topic？
Consumer client和broker的心跳机制- 发送重试消息的机制
心跳
ClientManageProcessor#heartBeat 心跳函数具体如下：
public RemotingCommand heartBeat(ChannelHandlerContext ctx, RemotingCommand request) {
RemotingCommand response = RemotingCommand.createResponseCommand(null);
HeartbeatData heartbeatData = HeartbeatData.decode(request.getBody(), HeartbeatData.class);
ClientChannelInfo clientChannelInfo = new ClientChannelInfo(
ctx.channel(),
heartbeatData.getClientID(),
request.getLanguage(),
request.getVersion()
);
for (ConsumerData data : heartbeatData.getConsumerDataSet()) {
SubscriptionGroupConfig subscriptionGroupConfig =
this.brokerController.getSubscriptionGroupManager().findSubscriptionGroupConfig(
data.getGroupName());
boolean isNotifyConsumerIdsChangedEnable = true;
if (null != subscriptionGroupConfig) {
isNotifyConsumerIdsChangedEnable = subscriptionGroupConfig.isNotifyConsumerIdsChangedEnable();
int topicSysFlag = 0;
if (data.isUnitMode()) {
topicSysFlag = TopicSysFlag.buildSysFlag(false, true);
}
String newTopic = MixAll.getRetryTopic(data.getGroupName());
// topic 更新到 topic Config 这个内存对象，然后还会触发
this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(
newTopic,
subscriptionGroupConfig.getRetryQueueNums(),
PermName.PERM_WRITE | PermName.PERM_READ, topicSysFlag);
}
boolean changed = this.brokerController.getConsumerManager().registerConsumer(
data.getGroupName(),
clientChannelInfo,
data.getConsumeType(),
data.getMessageModel(),
data.getConsumeFromWhere(),
data.getSubscriptionDataSet(),
isNotifyConsumerIdsChangedEnable
);
if (changed) {
log.info("registerConsumer info changed {} {}",
data.toString(),
RemotingHelper.parseChannelRemoteAddr(ctx.channel())
);
}
}
for (ProducerData data : heartbeatData.getProducerDataSet()) {
this.brokerController.getProducerManager().registerProducer(data.getGroupName(),
clientChannelInfo);
}
response.setCode(ResponseCode.SUCCESS);
response.setRemark(null);
return response;
}
复制代码包含 retry topic 信息, getRetryTopic 信息，然后将 retry topic 信息注册到 nameserver。
String newTopic = MixAll.getRetryTopic(data.getGroupName());
this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(
newTopic,
subscriptionGroupConfig.getRetryQueueNums(),
PermName.PERM_WRITE | PermName.PERM_READ, topicSysFlag);
复制代码consumer client 再请求 nameserver ，就可以获取到 retry topic 的信息。 retryQueueNums 默认是1 , 也就是说 retry topic 默认在每个 broker 上有一个 queue。 retry topic 给予心跳机制实现，在客户端实现上，客户端和所有 broker 进行心跳长链接。因此 broker master 和 broker slave 都有 retry topic。
投递重试消息
SendMessageProcessor#asyncConsumerSendMsgBack . 和心跳机制类似, 都是调用 createTopicInSendMessageBackMethod实现, 所以如果retry topic没有创建, 也会触发broker向nameserver的注册. 但是就客户端实现而言, 会先发送心跳, 再拉取消息进行投递的。asyncConsumerSendMsgBack 方法可以看到重试队列。
String newTopic = MixAll.getRetryTopic(requestHeader.getGroup());
int queueIdInt = ThreadLocalRandom.current().nextInt(99999999) % subscriptionGroupConfig.getRetryQueueNums();
int topicSysFlag = 0;
if (requestHeader.isUnitMode()) {
topicSysFlag = TopicSysFlag.buildSysFlag(false, true);
}
TopicConfig topicConfig = this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(
newTopic,
subscriptionGroupConfig.getRetryQueueNums(),
PermName.PERM_WRITE | PermName.PERM_READ, topicSysFlag);
复制代码