Prerequisite knowledge: RocketMQ's topic has multiple queues, and multiple topics are allocated in the same consumer group. There are multiple consumers in the consumer group. When a consumer is injected into the consumer group, communication between the consumer and multiple queues must be done. Allocation, and this allocation is called the Rebalance mechanism. The original intention of this mechanism is to improve parallel consumption capabilities.
The guaranteed mechanism in RocketMQ is that a queue can be allocated to at most one consumer, and one consumer can consume multiple queues. When the number of consumers under a certain consumer group is greater than the number of queues, there will be consumers. No queues can be allocated.
The reason why a queue can be consumed by at most one consumer is to ensure the order and reliability of messages. At the same time, a consumer can consume multiple queues to increase the consumer's concurrent consumption capacity and load. Balance.
Execute RebalanceService for redistribution
After DefaultMQPushConsumerImpl executes the run method, it will execute the rebalanceImmediately() method to actively rebalance.
// 新的Consumer服务启动的时候，主动调用rebalanceImmediately唤醒
// 负载均衡服务rebalanceService，进行重分配。
public void rebalanceImmediately() {
this.rebalanceService.wakeup();
}
When DefaultMQPushConsumerImpl executes the run method, it will start the CreateMQClientInstance client communication instance. At this time, it will execute the this.rebalanceService.start() method to start the redistribution service, and then execute the run method of the rebalanceService service, and perform redistribution every 20 seconds. .
@Override
public void run() {
log.info(this.getServiceName() + " service started");
// 服务没停止一直运行
while (!this.isStopped()) {
// 等待运行，默认休息20s，可以通过rocketmq.client.rebalance.waitInterval来配置
this.waitForRunning(waitInterval);
// 执行该方法进行负载均衡
this.mqClientFactory.doRebalance();
}
log.info(this.getServiceName() + " service end");
}
Broker heartbeat processing or topic addition and deletion, when a new Consumer is registered, the service inside MQClientInstance will also send heartbeat information to the broker regularly for 30 seconds. When sent to the Broker, the processing code is HEART_BEAT, and the processor is registered according to the time when the Broker starts. Method registerProcessor(), the final processing logic is handled by the processRequest method of ClientManageProcessor, and the final loop traverses the consumerDataSet collection. If the consumer information changes, (two conditions determine whether the change isNotifyConsumerIdsChangedEnable is true, there is an updateChannel update connection, and there is an updateSubscription update subscription) broker A request with Code NOTIFY_CONSUMER_IDS_CHANGED will be sent to all consumer clients in the same group to request redistribution.
public RemotingCommand processRequest(ChannelHandlerContext ctx, RemotingCommand request)
throws RemotingCommandException {
switch (request.getCode()) {
// 心跳处理
case RequestCode.HEART_BEAT:
return this.heartBeat(ctx, request);
case RequestCode.UNREGISTER_CLIENT:
return this.unregisterClient(ctx, request);
case RequestCode.CHECK_CLIENT_CONFIG:
return this.checkClientConfig(ctx, request);
default:
break;
}
return null;
}
No matter which method is used, the run method of RebalanceService will be executed to implement the redistribution logic.
public void run() {
log.info(this.getServiceName() + " service started");
while (!this.isStopped()) {
this.waitForRunning(waitInterval);
// 执行该方法进行负载均衡
this.mqClientFactory.doRebalance();
}
log.info(this.getServiceName() + " service end");
}
The doRebalance() method performs redistribution.
public void doRebalance(final boolean isOrder) {
// 获取当前消费者的订阅信息集合
Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
if (subTable != null) {
// 不为空，编辑订阅信息
for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
final String topic = entry.getKey();
try {
// 根据topic重新分配
this.rebalanceByTopic(topic, isOrder);
} catch (Throwable e) {
if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
log.warn("rebalanceByTopic Exception", e);
}
}
}
}
this.truncateMessageQueueNotMyTopic();
}
Obtain the subscription information collection of the current consumer, then traverse the subscription information collection, obtain the subscribed topic, and call the rebalanceByTopic method to redistribute the topic.
rebalanceByTopic method
private void rebalanceByTopic(final String topic, final boolean isOrder) {
switch (messageModel) {
// 广播模式
case BROADCASTING: {
// 根据topic获取MessageQueue集合
Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
if (mqSet != null) {
boolean changed = this.updateProcessQueueTableInRebalance(topic, mqSet, isOrder);
if (changed) {
this.messageQueueChanged(topic, mqSet, mqSet);
log.info("messageQueueChanged {} {} {} {}",
consumerGroup,
topic,
mqSet,
mqSet);
}
} else {
log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
}
break;
}
// 集群模式
case CLUSTERING: {
// 获取Topic下的所有的队列
Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
// 获取同一个消费者组的所有实例
List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
if (null == mqSet) {
if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
}
}
if (null == cidAll) {
log.warn("doRebalance, {} {}, get consumer id list failed", consumerGroup, topic);
}
// 负载均衡分配MessageQueue
if (mqSet != null && cidAll != null) {
List<MessageQueue> mqAll = new ArrayList<MessageQueue>();
// 备份所有队列
mqAll.addAll(mqSet);
// 两个队列进行排序
Collections.sort(mqAll);
Collections.sort(cidAll);
// 获取分配策略
AllocateMessageQueueStrategy strategy = this.allocateMessageQueueStrategy;
List<MessageQueue> allocateResult = null;
try {
// 进行分配，存在6种不同的策略
allocateResult = strategy.allocate(
this.consumerGroup,
this.mQClientFactory.getClientId(),
mqAll,
cidAll);
} catch (Throwable e) {
log.error("AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}", strategy.getName(),
e);
return;
}
Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
if (allocateResult != null) {
allocateResultSet.addAll(allocateResult);
}
boolean changed = this.updateProcessQueueTableInRebalance(topic, allocateResultSet, isOrder);
if (changed) {
log.info(
"rebalanced result changed. allocateMessageQueueStrategyName={}, group={}, topic={}, clientId={}, mqAllSize={}, cidAllSize={}, rebalanceResultSize={}, rebalanceResultSet={}",
strategy.getName(), consumerGroup, topic, this.mQClientFactory.getClientId(), mqSet.size(), cidAll.size(),
allocateResultSet.size(), allocateResultSet);
this.messageQueueChanged(topic, mqSet, allocateResultSet);
}
}
break;
}
default:
break;
}
}
There is no redistribution in broadcast mode, and each Consumer will consume all messages.
In cluster mode, obtain consumers under the same consumer group, find the allocation strategy AllocateMessageQueueStrategy, execute the allocate method, and redistribute.
AllocateMessageQueueStrategy is a strategy algorithm interface for message distribution between RocketMQ consumers. RocketMQ itself provides 6 built-in implementations. At the same time, we can also define our own strategy by implementing this interface.
AllocateMachineRoomNearby indicates that the computer room is allocated nearby
public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
List<String> cidAll) {
List<MessageQueue> result = new ArrayList<MessageQueue>();
// 参数校验
if (!check(consumerGroup, currentCID, mqAll, cidAll)) {
return result;
}
// 将消息队列根据机房分组
Map<String/*machine room */, List<MessageQueue>> mr2Mq = new TreeMap<String, List<MessageQueue>>();
for (MessageQueue mq : mqAll) {
String brokerMachineRoom = machineRoomResolver.brokerDeployIn(mq);
if (StringUtils.isNoneEmpty(brokerMachineRoom)) {
if (mr2Mq.get(brokerMachineRoom) == null) {
mr2Mq.put(brokerMachineRoom, new ArrayList<MessageQueue>());
}
mr2Mq.get(brokerMachineRoom).add(mq);
} else {
throw new IllegalArgumentException("Machine room is null for mq " + mq);
}
}
// 将消息者根据机房分组
Map<String/*machine room */, List<String/*clientId*/>> mr2c = new TreeMap<String, List<String>>();
for (String cid : cidAll) {
String consumerMachineRoom = machineRoomResolver.consumerDeployIn(cid);
if (StringUtils.isNoneEmpty(consumerMachineRoom)) {
if (mr2c.get(consumerMachineRoom) == null) {
mr2c.put(consumerMachineRoom, new ArrayList<String>());
}
mr2c.get(consumerMachineRoom).add(cid);
} else {
throw new IllegalArgumentException("Machine room is null for consumer id " + cid);
}
}
List<MessageQueue> allocateResults = new ArrayList<MessageQueue>();
// 获取当前消费者的机房，然后就当前消费者分配到就近机房
String currentMachineRoom = machineRoomResolver.consumerDeployIn(currentCID);
List<MessageQueue> mqInThisMachineRoom = mr2Mq.remove(currentMachineRoom);
List<String> consumerInThisMachineRoom = mr2c.get(currentMachineRoom);
if (mqInThisMachineRoom != null && !mqInThisMachineRoom.isEmpty()) {
allocateResults.addAll(allocateMessageQueueStrategy.allocate(consumerGroup, currentCID, mqInThisMachineRoom, consumerInThisMachineRoom));
}
//2.allocate the rest mq to each machine room if there are no consumer alive in that machine room
for (Entry<String, List<MessageQueue>> machineRoomEntry : mr2Mq.entrySet()) {
if (!mr2c.containsKey(machineRoomEntry.getKey())) {
// no alive consumer in the corresponding machine room, so all consumers share these queues
allocateResults.addAll(allocateMessageQueueStrategy.allocate(consumerGroup, currentCID, machineRoomEntry.getValue(), cidAll));
}
}
return allocateResults;
}
First, consumers and queues are grouped according to the computer room, and then the computer room information of the current consumer is obtained. If the consumer and the queue belong to the same computer room, they will be allocated. The specific strategy to be used is determined based on the allocatedMessageQueueStrategy passed in. If there is no corresponding consumers on the consumer queue, then the consumer queue has all consumer allocations, and the specific strategy is also determined by the incoming allocateMessageQueueStrategy.
AllcateMessageQueueAveragely means even distribution
public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
List<String> cidAll) {
List<MessageQueue> result = new ArrayList<MessageQueue>();
// 检验参数
if (!check(consumerGroup, currentCID, mqAll, cidAll)) {
return result;
}
// currentCID在cidAll中的索引位置
int index = cidAll.indexOf(currentCID);
// 计算平均分配后的余数，大于0表示不能被整除
int mod = mqAll.size() % cidAll.size();
// 计算当前消费者分配的队列数
int averageSize =
mqAll.size() <= cidAll.size() ? 1 : (mod > 0 && index < mod ? mqAll.size() / cidAll.size()
+ 1 : mqAll.size() / cidAll.size());
int startIndex = (mod > 0 && index < mod) ? index * averageSize : index * averageSize + mod;
int range = Math.min(averageSize, mqAll.size() - startIndex);
// 按顺序分配
for (int i = 0; i < range; i++) {
result.add(mqAll.get((startIndex + i) % mqAll.size()));
}
return result;
}
A simple understanding is that the number of message queues is divided by the number of consumers. If the remainder is 0, it will be distributed evenly. If the remainder is not 0, each consumer will be allocated at least the number of divisors, and the remainder will only be ranked first. consumers can be allocated.
AllocateMessageQueueAveragelyByCircle means circular average allocation
public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
List<String> cidAll) {
List<MessageQueue> result = new ArrayList<MessageQueue>();
//检查参数
if (!check(consumerGroup, currentCID, mqAll, cidAll)) {
return result;
}
// 获取下标
int index = cidAll.indexOf(currentCID);
// 依次分配（轮询）
for (int i = index; i < mqAll.size(); i++) {
if (i % cidAll.size() == index) {
result.add(mqAll.get(i));
}
}
return result;
}
The simple understanding is to allocate them in sequence until all message queues are allocated (in the final analysis, it is polling).
AllocateMessageQueueAveragelyByCircle表示环形平均分配
public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
List<String> cidAll) {
return this.messageQueueList;
}
Just to give us some expansion, we can call the setMessageQueueList method to customize the message queue collection that needs to be consumed.
AllocateMessageQueueByMachineRoom表示机房平均分配
public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
List<String> cidAll) {
List<MessageQueue> result = new ArrayList<MessageQueue>();
// 检查配置
if (!check(consumerGroup, currentCID, mqAll, cidAll)) {
return result;
}
// 获取下标
int currentIndex = cidAll.indexOf(currentCID);
if (currentIndex < 0) {
return result;
}
List<MessageQueue> premqAll = new ArrayList<MessageQueue>();
for (MessageQueue mq : mqAll) {
String[] temp = mq.getBrokerName().split("@");
if (temp.length == 2 && consumeridcs.contains(temp[0])) {
premqAll.add(mq);
}
}
// 平均分配的对立
int mod = premqAll.size() / cidAll.size();
int rem = premqAll.size() % cidAll.size();
int startIndex = mod * currentIndex;
int endIndex = startIndex + mod;
for (int i = startIndex; i < endIndex; i++) {
result.add(premqAll.get(i));
}
if (rem > currentIndex) {
result.add(premqAll.get(currentIndex + mod * cidAll.size()));
}
return result;
}
The messager only needs to bind the broker in the corresponding group. This strategy requires that the brokerName must be named according to "computer room name@brokerName". When the consumer allocates the queue, it will first come out all the MesgeQueue according to the computer room name, and then follow the corresponding policy. Make allocations. At the same time, AllocateMessageQueueByMachineRoom focuses more on the division and allocation of computer rooms, while AllocateMachineRoomNearby focuses more on nearby deployment and network delay optimization.
AllocateMessageQueueConsisterntHash represents consistent hash allocation
public List<MessageQueue> allocate(String consumerGroup, String currentCID, List<MessageQueue> mqAll,
List<String> cidAll) {
List<MessageQueue> result = new ArrayList<MessageQueue>();
// 检查配置
if (!check(consumerGroup, currentCID, mqAll, cidAll)) {
return result;
}
Collection<ClientNode> cidNodes = new ArrayList<ClientNode>();
for (String cid : cidAll) {
cidNodes.add(new ClientNode(cid));
}
final ConsistentHashRouter<ClientNode> router; //for building hash ring
if (customHashFunction != null) {
router = new ConsistentHashRouter<ClientNode>(cidNodes, virtualNodeCnt, customHashFunction);
} else {
router = new ConsistentHashRouter<ClientNode>(cidNodes, virtualNodeCnt);
}
List<MessageQueue> results = new ArrayList<MessageQueue>();
for (MessageQueue mq : mqAll) {
ClientNode clientNode = router.routeNode(mq.toString());
if (clientNode != null && currentCID.equals(clientNode.getKey())) {
results.add(mq);
}
}
return results;
}
This queue strategy stores the hash value of the Consumer and the hash value of the Queue as Node nodes on the hash ring. In the counterclockwise direction, the Consumer closest to the Queue is the Consumer to be allocated by the Queue. There is a virtualNodeCnt object, virtualNodeCnt Represents a virtual thread of a physical node.
