introduction:
Why is there multiple Broker servers in the cluster, and autoCreateTopicEnable is set to true, which means that the topic is automatically created, but the routing information of the newly created topic is only contained on one of the Broker servers. Why?
Expected value : For the high availability of message sending, it is hoped that the newly created Topic will create a corresponding queue on each Broker in the cluster to avoid a single node failure of the Broker.
text:
I still remember the first time I used rocketmq , I needed to go to the console to create topics in advance . I wondered why I wanted to design this way, so I decided to pick up a wave of source code and take everyone to understand the creation mechanism of rocketmq topic from the root.
In the design idea of Rocketmq, topic is used as the organization form of the same business logic message. It is only a logical concept , and a topic contains several logical queues, namely message queues . The message content is actually stored in In the queue, and the queue is stored in the broker , below I use a diagram to illustrate the topic storage model:
In fact, there are two different topic creation methods in rocketmq, one is the pre-creation that I just mentioned , and the other is the automatic creation . Let me drive you to explain these two creation mechanisms in detail from the perspective of source code.
1. Automatic creation
1.1 The process of automatically creating a theme
By default, the topic does not need to be created manually. When the producer sends a message, it will pull the topic routing information from the nameserver. If the topic routing information does not exist, it will be pulled by default when the broker is started. The default name is " TBW102 " Topic:
org.apache.rocketmq.common.MixAll：
// Will be created at broker when isAutoCreateTopicEnablepublic static final String AUTO_CREATE_TOPIC_KEY_TOPIC = "TBW102";
public static final String AUTO_CREATE_TOPIC_KEY_TOPIC = "TBW102";
The automatically created switch is configured in BrokerConfig and controlled by the autoCreateTopicEnable field,
org.apache.rocketmq.common.BrokerConfig：
@ImportantField
private boolean autoCreateTopicEnable = true;
When the broker starts , will call TopicConfigManager constructor, after the (true) autoCreateTopicEnable open, will "TBW102" Save to topicConfigTable in :
org.apache.rocketmq.broker.topic.TopicConfigManager # TopicConfigManager ：
// MixAll.AUTO_CREATE_TOPIC_KEY_TOPICif (this.brokerController.getBrokerConfig().isAutoCreateTopicEnable()) { String topic = MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC; TopicConfig topicConfig = new TopicConfig(topic); this.systemTopicList.add(topic); topicConfig.setReadQueueNums(this.brokerController.getBrokerConfig() .getDefaultTopicQueueNums()); topicConfig.setWriteQueueNums(this.brokerController.getBrokerConfig() .getDefaultTopicQueueNums()); int perm = PermName.PERM_INHERIT | PermName.PERM_READ | PermName.PERM_WRITE; topicConfig.setPerm(perm); this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);}
if (this.brokerController.getBrokerConfig().isAutoCreateTopicEnable()) {
String topic = MixAll.AUTO_CREATE_TOPIC_KEY_TOPIC;
TopicConfig topicConfig = new TopicConfig(topic);
this.systemTopicList.add(topic);
topicConfig.setReadQueueNums(this.brokerController.getBrokerConfig()
.getDefaultTopicQueueNums());
topicConfig.setWriteQueueNums(this.brokerController.getBrokerConfig()
.getDefaultTopicQueueNums());
int perm = PermName.PERM_INHERIT | PermName.PERM_READ | PermName.PERM_WRITE;
topicConfig.setPerm(perm);
this.topicConfigTable.put(topicConfig.getTopicName(), topicConfig);
}
The broker will send the topic information of the topicConfigTable to the nameserver by sending a heartbeat packet , and the nameserver will register the topic information in the RouteInfoManager .
Continue to see how the topic routing information is obtained from the nameserver when the message is sent :
org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl#tryToFindTopicPublishInfo：
private TopicPublishInfo tryToFindTopicPublishInfo(final String topic) { TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic); if (null == topicPublishInfo || !topicPublishInfo.ok()) { this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo()); // 生产者第一次发送消息，topic在nameserver中并不存在 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic); topicPublishInfo = this.topicPublishInfoTable.get(topic); } if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) { return topicPublishInfo; } else { // 第二次请求会将isDefault=true，开启默认“TBW102”从namerserver获取路由信息 this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer); topicPublishInfo = this.topicPublishInfoTable.get(topic); return topicPublishInfo; }}
TopicPublishInfo topicPublishInfo = this.topicPublishInfoTable.get(topic);
if (null == topicPublishInfo || !topicPublishInfo.ok()) {
this.topicPublishInfoTable.putIfAbsent(topic, new TopicPublishInfo());
// 生产者第一次发送消息，topic在nameserver中并不存在
this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic);
topicPublishInfo = this.topicPublishInfoTable.get(topic);
}
if (topicPublishInfo.isHaveTopicRouterInfo() || topicPublishInfo.ok()) {
return topicPublishInfo;
} else {
// 第二次请求会将isDefault=true，开启默认“TBW102”从namerserver获取路由信息
this.mQClientFactory.updateTopicRouteInfoFromNameServer(topic, true, this.defaultMQProducer);
topicPublishInfo = this.topicPublishInfoTable.get(topic);
return topicPublishInfo;
}
}
In the above method, the topic sends a message for the first time , and the routing information of the topic cannot be obtained from the namserver at this time. If topicPublishInfo=null enters the first if, then a second request to the namserver will be made , and then isDefault=true . Enable the default "TBW102" topic to obtain routing information from the namerserver . At this time, the "TBW102" topic has been registered to the nameserver by the broker by default (the above has been registered when the broker is started):
org.apache.rocketmq.client.impl.factory.MQClientInstance#updateTopicRouteInfoFromNameServer：
if (isDefault && defaultMQProducer != null) {
// 使用默认的“TBW102”topic获取路由信息
topicRouteData = this.mQClientAPIImpl.getDefaultTopicRouteInfoFromNameServer(defaultMQProducer.getCreateTopicKey(),1000 * 3);
if (topicRouteData != null) {
for (QueueData data : topicRouteData.getQueueDatas()) {
int queueNums = Math.min(defaultMQProducer.getDefaultTopicQueueNums(), data.getReadQueueNums());
data.setReadQueueNums(queueNums);
data.setWriteQueueNums(queueNums);
}
}
}
If isDefault=true and defaultMQProducer is not empty , the default routing information is obtained from the nameserver . At this time , the default "TBW102" topic routing information of all brokers with the automatic creation switch will be obtained , and the default topic message queue number (minimum value) ) .
org.apache.rocketmq.client.impl.factory.MQClientInstance#updateTopicRouteInfoFromNameServer：
TopicRouteData old = this.topicRouteTable.get(topic);
boolean changed = topicRouteDataIsChange(old, topicRouteData);
if (!changed) {
changed = this.isNeedUpdateTopicRouteInfo(topic);
} else {
log.info("the topic[{}] route info changed, old[{}] ,new[{}]", topic, old, topicRouteData);
}
Retrieve the topic routing information from the local cache. Since the topic is sending a message for the first time, there is no routing information for the topic at this time. Therefore, when comparing the topic routing information to "TBW102", changed to true, that is, there is a change, enter The following logic:
org.apache.rocketmq.client.impl.factory.MQClientInstance#updateTopicRouteInfoFromNameServer：
// Update sub info{
{
Set<MessageQueue> subscribeInfo = topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
Iterator<Entry<String, MQConsumerInner>> it = this.consumerTable.entrySet().iterator();
while (it.hasNext()) {
Entry<String, MQConsumerInner> entry = it.next();
MQConsumerInner impl = entry.getValue();
if (impl != null) {
impl.updateTopicSubscribeInfo(topic, subscribeInfo);
}
}
}
Use the "TBW102" topic routing information to build TopicPublishInfo , and use topic as the key and TopicPublishInfo as the value to update the broker's local cache . Here you can understand that the brokers have worked so hard to create the "TBW102" topic and register its routing information to The nameserver, immediately after being acquired by the new topic, uses the routing information of the "TBW102" topic to construct a new TopicPublishInfo and use it as your own. Since the routing information of TopicPublishInfo is the routing information of the default "TBW102" topic, you really want to send the message The topic will also be loaded and sent to the broker where the topic "TBW102" is located . Here we can call it a hacking method .
When the broker receives the message, it will call the createTopicInSendMessageMethod method in the msgCheck method to stuff the topic information into the topicConfigTable cache, and the broker will periodically send a heartbeat to send the topicConfigTable to the nameserver for registration.
A sequence diagram for automatically creating and obtaining topic information when sending messages:
1.2 Producer message process
Let's take a look at the message sending process. It is roughly not very complicated. It is nothing more than finding the Topic to send the message on which Broker, and then sending the message. 【reference】
Now I know what TBW102 is used for, that is, the broker startup that accepts the automatic creation of the topic will register this default topic to the NameServer, so that when the Producer sends a new topic message, it will know which Broker can automatically create the topic, and then send it to that Broker .
When the Broker receives this message, it finds that it does not find the corresponding topic, but it accepts to create a new topic, which will create the corresponding Topic routing information.
1.3 Disadvantages of automatic theme creation
Automatically create a topic then it is possible that the message of the topic will only be sent to one Broker , and it will not play the role of load balancing .
Because after the request to create a new Topic arrives at the Broker, the Broker creates the corresponding routing information, but the heartbeat is sent every 30s , so it takes up to 30s for the NameServer to know the routing information of this new Topic.
Assuming that the sender is still sending messages continuously and quickly , there is actually no routing information about this topic on the NameServer, so there is a chance that other brokers that allow automatic creation can also create corresponding topic routing information, so that the Brokers in the cluster You can accept this topic's information and achieve the purpose of load balancing, but there are some brokers who may not receive it .
If the sender does not send one within 30s after sending this time , the previous Broker updates the routing information to the NameServer with the heartbeat, then the Producer that sends the topic message afterwards can only know the topic message from the NameServer Send to the previous Broker, which is not balanced. If there are a lot of news about this new topic, the load on that Broker will be very high. Moreover, high availability cannot be achieved. If this broker fails, it cannot be loaded to another, resulting in a single point of failure.
Therefore, it is not recommended to enable the automatic creation of topics online, that is, the autoCreateTopicEnable parameter.
Second, create in advance
In fact, it seems more appropriate to call pre-creation, that is, to create topic-related information in the broker and register it in the nameserver, and then directly obtain the topic routing information from the nameserver when the client sends a message , but manual creation will be more action- oriented The image is easy to understand, telling you directly, your topic information needs to be manually created on the console .
Pre-creation needs to be created through topic related commands provided by mqadmin, and execute:
./mqadmin updateTopicThe official parameters are as follows:
usage: mqadmin updateTopic
-b,--brokerAddr <arg> create topic to which broker
-c,--clusterName <arg> create topic to which cluster
-h,--help Print help
-n,--namesrvAddr <arg> Name server address list, eg: 192.168.0.1:9876;192.168.0.2:9876
-o,--order <arg> set topic's order(true|false
-p,--perm <arg> set topic's permission(2|4|6), intro[2:W 4:R; 6:RW]
-r,--readQueueNums <arg> set read queue nums
-s,--hasUnitSub <arg> has unit sub (true|false
-t,--topic <arg> topic name
-u,--unit <arg> is unit topic (true|false
-w,--writeQueueNums <arg> set write queue nums
We directly locate the method of executing commands in its implementation class:
2.1 Created by broker mode:
org.apache.rocketmq.tools.command.topic.UpdateTopicSubCommand#execute：
// -b,--brokerAddr <arg> create topic to which broker
if (commandLine.hasOption('b')) {
String addr = commandLine.getOptionValue('b').trim();
defaultMQAdminExt.start();
defaultMQAdminExt.createAndUpdateTopicConfig(addr, topicConfig);
return;
}
Obtain the address of the broker with the -b parameter at runtime from the commandLine command line tool. DefaultMQAdminExt is the API executed by the default rocketmq console . At this time, call the start method, which creates a mqClientInstance, which encapsulates the details of netty communication, and then This is the most important step. Call createAndUpdateTopicConfig to send the topic configuration information to the specified broker to complete the creation of the topic.
2.2 Create by cluster mode:
org.apache.rocketmq.tools.command.topic.UpdateTopicSubCommand#execute：
// -c,--clusterName <arg> create topic to which cluster
else if (commandLine.hasOption('c')) {
String clusterName = commandLine.getOptionValue('c').trim();
defaultMQAdminExt.start();
Set<String> masterSet =
CommandUtil.fetchMasterAddrByClusterName(defaultMQAdminExt, clusterName);
for (String addr : masterSet) {
defaultMQAdminExt.createAndUpdateTopicConfig(addr, topicConfig);
System.out.printf("create topic to %s success.%n", addr);
}
return;
}
The logic of creating through cluster mode is roughly the same as that of creating through broker mode. There is a step of obtaining the master address of all brokers in the cluster from the nameserver according to the cluster , and then sending topic information to each broker in the cluster in a loop . This logic is the same as the specified A single broker is consistent.
This also shows that when the cluster mode is used to create a topic, the number of queues of each broker in the cluster is the same. When a single broker mode is used to create a topic, the number of queues of each broker can be inconsistent.
Create a sequence diagram in advance:
3. When do I need to create a topic in advance?
It is recommended to turn on offline and turn off online , not what I said, but an official recommendation:
Why is RocketMQ designed like this? After a wave of deep analysis of the source code, I got the answer I wanted:
According to the above source code analysis, we conclude that rocketmq will first obtain topic routing information when sending a message . If topic is the first time to send a message , since the nameserver does not have topic routing information, it will use " TBW102 " again. The default topic obtains routing information. Assuming that the brokers have turned on the automatic creation switch , then the routing information of all brokers will be obtained at this time . The message sending will select one of the brokers to send messages according to the load algorithm (only one of them can be sent, Otherwise, it will be sent repeatedly) . After the message arrives at the broker, it is found that there is no such topic locally, and the information for creating the topic will be stuffed into the local cache, and the topic routing information will be registered in the nameserver, then this will cause a consequence: all future Messages of this topic will all be sent to this broker. If the topic message volume is very large, it will cause too much load on a certain broker. Such message storage will not achieve the effect of load balancing . Similarly, if this Broker fails and hangs up, sending messages will fail, and Broker's single node fails.
●The strongest Tomcat8 performance optimization in history
● Why can Alibaba resist 10 billion in 90 seconds? --The evolution of server-side high-concurrency distributed architecture
● B2B e-commerce platform--ChinaPay UnionPay electronic payment function
● Learn Zookeeper distributed lock, let interviewers look at you with admiration
● SpringCloud e-commerce spike microservice-Redisson distributed lock solution
Check out more good articles, enter the official account--please me--excellent in the past
A deep and soulful public account 0.0