Consumer uses DefaultMQPushConsumerImpl by default to consume messages by long polling, which can ensure the same real-time performance as Push. You can also refer to the example project and use DefaultMQPullConsumerImpl, which is controlled by the business to pull messages and update the consumption progress.
2. Cluster mode VS broadcast mode
cluster mode:
- The consumption progress of the message, that is, consumerOffset.json, is saved on the broker.
- All consumers consume messages of the topic on average.
- After the message consumption fails, the consumer will send it back to the broker, and the broker will set a different delayLevel according to the number of consumption failures to resend.
- Different consumerGroups with the same topic form a pseudo-broadcast mode, so that all consumers can receive messages.
Broadcast mode:
- The consumption progress of the message is saved on the consumer's machine.
- All consumers will receive messages under the topic.
- After the message consumption fails, it will be discarded directly and will not be sent back to the broker for re-delivery.
3. ConsumeFromWhere
consumer can set the starting point of consumption, MQ provides three ways:
- CONSUME_FROM_LAST_OFFSET
- CONSUME_FROM_FIRST_OFFSET
- CONSUME_FROM_TIMESTAMP
We understand the meaning of these three different consumption points from the perspective of source code. There is a class RebalancePushImpl inside DefaultMQPushConsumerImpl, which first calculates that the client needs to pull the queue, then go to the broker to get the consumption progress, and get the consumption offset code in the computePullFromWhere method:
public long computePullFromWhere(MessageQueue mq) {
long result = -1;
final ConsumeFromWhere consumeFromWhere = this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeFromWhere();
final OffsetStore offsetStore = this.defaultMQPushConsumerImpl.getOffsetStore();
switch (consumeFromWhere) {
case CONSUME_FROM_LAST_OFFSET_AND_FROM_MIN_WHEN_BOOT_FIRST:
case CONSUME_FROM_MIN_OFFSET:
case CONSUME_FROM_MAX_OFFSET:
case CONSUME_FROM_LAST_OFFSET: {
long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
if (lastOffset >= 0) {
result = lastOffset;
}
// First start,no offset
else if (-1 == lastOffset) {
if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
result = 0L;
} else {
try {
result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
} catch (MQClientException e) {
result = -1;
}
}
} else {
result = -1;
}
break;
}
case CONSUME_FROM_FIRST_OFFSET: {
long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
if (lastOffset >= 0) {
result = lastOffset;
} else if (-1 == lastOffset) {
result = 0L;
} else {
result = -1;
}
break;
}
case CONSUME_FROM_TIMESTAMP: {
long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);
if (lastOffset >= 0) {
result = lastOffset;
} else if (-1 == lastOffset) {
if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
try {
result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);
} catch (MQClientException e) {
result = -1;
}
} else {
try {
long timestamp = UtilAll.parseDate(this.defaultMQPushConsumerImpl.getDefaultMQPushConsumer().getConsumeTimestamp(),
UtilAll.yyyyMMddHHmmss).getTime();
result = this.mQClientFactory.getMQAdminImpl().searchOffset(mq, timestamp);
} catch (MQClientException e) {
result = -1;
}
}
} else {
result = -1;
}
break;
}
default:
break;
}
return result;
}
First look at the logic of CONSUME_FROM_LAST_OFFSET, lastOffset >= 0, which means that there is consumption progress on the broker side, indicating that some messages have been started and consumed before, then start consumption from the returned offset. When -1 == lastOffset, if it is a retry queue, it will consume from the beginning, and if it is a normal queue, it will consume from the maximum offset.
CONSUME_FROM_TIMESTAMP, if it is the first startup, that is, when -1 == lastOffset, if it is a normal queue, it will start consumption from the set time point. If no time point is set, it will start consumption half an hour ago by default.
private String consumeTimestamp = UtilAll.timeMillisToHumanString3(System.currentTimeMillis() - (1000 * 60 * 30));
4. The startup source code of the pull message
Consumer is as follows:
public void start() throws MQClientException {
switch (this.serviceState) {
case CREATE_JUST:
log.info("the consumer [{}] start beginning. messageModel={}, isUnitMode={}", this.defaultMQPushConsumer.getConsumerGroup(),
this.defaultMQPushConsumer.getMessageModel(), this.defaultMQPushConsumer.isUnitMode());
this.serviceState = ServiceState.START_FAILED;
this.checkConfig();
this.copySubscription();
if (this.defaultMQPushConsumer.getMessageModel() == MessageModel.CLUSTERING) {
this.defaultMQPushConsumer.changeInstanceNameToPID();
}
this.mQClientFactory = MQClientManager.getInstance().getAndCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);
this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
this.pullAPIWrapper = new PullAPIWrapper (//
mQClientFactory, //
this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);
if (this.defaultMQPushConsumer.getOffsetStore() != null) {
this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
switch (this.defaultMQPushConsumer.getMessageModel()) {
case BROADCASTING:
this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
case CLUSTERING:
this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
default:
break;
}
}
this.offsetStore.load();
if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
this.consumeOrderly = true;
this.consumeMessageService =
new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
this.consumeOrderly = false;
this.consumeMessageService =
new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}
this.consumeMessageService.start();
boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
this.consumeMessageService.shutdown();
throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
mQClientFactory.start();
log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The PushConsumer service state not OK, maybe started once, "//
+ this.serviceState//
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
this.updateTopicSubscribeInfoWhenSubscriptionChanged();
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
this.mQClientFactory.rebalanceImmediately();
}
this.offsetStore.load();
The first is to load the message consumption progress. The BROADCASTING mode instantiates LocalFileOffsetStore, CLUSTERING instantiates RemoteBrokerOffsetStore, and loads the memory. All subsequent reads and writes are directly manipulated memory data, which is placed on the disk every 5s by the scheduled task.
this.consumeMessageService.start();
In order to consume message services, the default implementation is ConsumeMessageConcurrentlyService, that is, concurrent consumption.
mQClientFactory.start();
The startup of mQClientFactory will establish and broker channels, timed tasks, pull message services, and load balancing services. Since the request to pull the message is initiated by the load balancing, let's talk about the load balancing service first.
The load balancing service is performed by the RebalanceService thread every 20s, and the tracking code will eventually call the RebalanceImpl#rebalanceByTopic method:
case BROADCASTING: {
Set<MessageQueue> mqSet = this.topicSubscribeInfoTable.get(topic);
if (mqSet != null) {
boolean changed = this.updateProcessQueueTableInRebalance(topic, mqSet, isOrder);
if (changed) {
this.messageQueueChanged(topic, mqSet, mqSet);
log.info("messageQueueChanged {} {} {} {}", //
consumerGroup, //
topic, //
mqSet, //
mqSet);
}
} else {
log.warn("doRebalance, {}, but the topic[{}] not exist.", consumerGroup, topic);
}
break;
}
In broadcast mode, since all consumers need to receive messages, there is no load balancing strategy.
List<String> cidAll = this.mQClientFactory.findConsumerIdList(topic, consumerGroup);
In cluster mode, first obtain the list of consumers through topic and consumerGroup, and then allocate and pull message queues. The default is the average allocation strategy AllocateMessageQueueAveragely.
Direct mock data, you can see the allocate method of AllocateMessageQueueAveragely, which is the above conclusion.
After getting the message queue that the current consumer needs to pull, go to RebalanceImpl#updateProcessQueueTableInRebalance to construct a data pull request PullRequest, go to DefaultMQPushConsumerImpl#pullMessage for logical integration before pulling messages, and finally pull data asynchronously to the broker through the mQClientAPIImpl channel inside mQClientFactory.
Go back to the DefaultMQPushConsumerImpl#pullMessage method and take a look at the internal logic:
long size = processQueue.getMsgCount().get();
if (size > this.defaultMQPushConsumer.getPullThresholdForQueue()) {
this.executePullRequestLater(pullRequest, PullTimeDelayMillsWhenFlowControl);
if ((flowControlTimes1++ % 1000) == 0) {
log.warn(
"the consumer message buffer is full, so do flow control, minOffset={}, maxOffset={}, size={}, pullRequest={}, flowControlTimes={}",
processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), size, pullRequest, flowControlTimes1);
}
return;
}
Because the pulled messages will be placed in the local queue ProcessQueue for processing, when the local queue size is found to exceed 1000, it will be delayed for 50ms before pulling.
if (!this.consumeOrderly) {
if (processQueue.getMaxSpan() > this.defaultMQPushConsumer.getConsumeConcurrentlyMaxSpan()) {
this.executePullRequestLater(pullRequest, PullTimeDelayMillsWhenFlowControl);
if ((flowControlTimes2++ % 1000) == 0) {
log.warn(
"the queue's messages, span too long, so do flow control, minOffset={}, maxOffset={}, maxSpan={}, pullRequest={}, flowControlTimes={}",
processQueue.getMsgTreeMap().firstKey(), processQueue.getMsgTreeMap().lastKey(), processQueue.getMaxSpan(),
pullRequest, flowControlTimes2);
}
return;
}
}
Since the message takes offset as the key and is put into the TreeMap of the local queue ProcessQueue, there is a span check here. When the span value (ie this.msgTreeMap.lastKey() - this.msgTreeMap.firstKey()) is greater than 2000, delay the pull. Due to the business relationship, the speed of message consumption cannot be guaranteed. If messages with large offsets are processed quickly, the local queue will accumulate messages with small offsets, so the value of span may become larger and larger.
Since the message is pulled asynchronously, a PullCallback object needs to be constructed here. Inside the onSuccess method:
if (pullResult != null) {
pullResult = DefaultMQPushConsumerImpl.this.pullAPIWrapper.processPullResult(pullRequest.getMessageQueue(), pullResult,
subscriptionData);
Process the pull result, if there is a message, deserialize it, and then perform a tag comparison and deduplication. As mentioned in the previous chapter, the hashcode value of the tag is stored in the broker's ConsumeQueue, so this step of the consumer's deduplication is to ensure the accuracy of the message.
boolean dispathToConsume = processQueue.putMessage(pullResult.getMsgFoundList());
DefaultMQPushConsumerImpl.this.consumeMessageService.submitConsumeRequest(//
pullResult.getMsgFoundList(), //
processQueue, //
pullRequest.getMessageQueue(), //
dispathToConsume);
Then put the message into the local queue, and construct a ConsumeRequest through the submitConsumeRequest method. Since the consumeBatchSize is 1, the task is submitted to the consumeExecutor thread pool (20 threads), and each thread and each message are processed concurrently. ConsumeRequest will call consumeMessageService to trigger MessageListener#consumeMessage to execute the business. deal with.
ConsumeMessageConcurrentlyService.this.processConsumeResult(status, context, this);
After the ConsumeRequest#run business is processed, execute the processConsumeResult method:
switch (this.defaultMQPushConsumer.getMessageModel()) {
case BROADCASTING:
for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
MessageExt msg = consumeRequest.getMsgs().get(i);
log.warn("BROADCASTING, the message consume failed, drop it, {}", msg.toString());
}
break;
case CLUSTERING:
List<MessageExt> msgBackFailed = new ArrayList<MessageExt>(consumeRequest.getMsgs().size());
for (int i = ackIndex + 1; i < consumeRequest.getMsgs().size(); i++) {
MessageExt msg = consumeRequest.getMsgs().get(i);
boolean result = this.sendMessageBack(msg, context);
if (!result) {
msg.setReconsumeTimes(msg.getReconsumeTimes() + 1);
msgBackFailed.add(msg);
}
}
if (!msgBackFailed.isEmpty()) {
consumeRequest.getMsgs().removeAll(msgBackFailed);
this.submitConsumeRequestLater(msgBackFailed, consumeRequest.getProcessQueue(), consumeRequest.getMessageQueue());
}
break;
default:
break;
}
When the business process returns RECONSUME_LATER, the ackIndex is -1, CONSUME_SUCCESS is returned, and the ackIndex is 0. If the consumer is in broadcast mode, the log is printed directly after the message consumption fails. If the consumer is in cluster mode, the message will be sent back to the broker through this.sendMessageBack(msg, context), and the consumer will receive the message again. If sending back to the broker fails, the consumer will try to consume again after 5s.
These consumption failure messages sent back to the broker will set different delayLevels according to the number of consumption failures. SendMessageProcessor#consumerSendMsgBack:
if (msgExt.getReconsumeTimes() >= maxReconsumeTimes//
|| delayLevel < 0) {
newTopic = MixAll.getDLQTopic(requestHeader.getGroup());
queueIdInt = Math.abs(this.random.nextInt() % 99999999) % DLQ_NUMS_PER_GROUP;
topicConfig = this.brokerController.getTopicConfigManager().createTopicInSendMessageBackMethod(newTopic, //
DLQ_NUMS_PER_GROUP, //
PermName.PERM_WRITE, 0
);
if (null == topicConfig) {
response.setCode(ResponseCode.SYSTEM_ERROR);
response.setRemark("topic[" + newTopic + "] not exist");
return response;
}
}
else {
if (0 == delayLevel) {
delayLevel = 3 + msgExt.getReconsumeTimes();
}
msgExt.setDelayTimeLevel(delayLevel);
}
The first consumption fails, msgExt.getReconsumeTimes() is 0, since the original messageDelayLevel is "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h", when the delayLevel is 3, it corresponds to 10s , that is to say, the message that fails to be consumed for the first time will be re-consumed after 10s, and so on. After the first consumption fails, the message is transferred from the original topic to the %RETRY% queue. If the number of message consumption failures is greater than maxReconsumeTimes (16 times), the message will enter the DLQ queue.
long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
this.defaultMQPushConsumerImpl.getOffsetStore().updateOffset(consumeRequest.getMessageQueue(), offset, true);
}
Then delete the data in the local processing queue ProcessQueue. Note that inside this removeMessage method, msgTreeMap.firstKey() is always returned, so the offset of update consumption is always the minimum value, which is exactly what the MQ official document says, MQ duplicate messages need Let the business side filter or use idempotent operations.
5. Long polling
As mentioned earlier, the consumer is a long polling pull message. When the consumer pulls a message, if there is no new message on the broker side, the broker will hold the request through the PullRequestHoldService service:
if (brokerAllowSuspend && hasSuspendFlag) {
long pollingTimeMills = suspendTimeoutMillisLong;
if (!this.brokerController.getBrokerConfig().isLongPollingEnable()) {
pollingTimeMills = this.brokerController.getBrokerConfig().getShortPollingTimeMills();
}
String topic = requestHeader.getTopic();
long offset = requestHeader.getQueueOffset();
int queueId = requestHeader.getQueueId();
PullRequest pullRequest = new PullRequest(request, channel, pollingTimeMills,
this.brokerController.getMessageStore().now(), offset, subscriptionData);
this.brokerController.getPullRequestHoldService().suspendPullRequest(topic, queueId, pullRequest);
response = null;
break;
}
Broker builds ConsumeQueue asynchronously through ReputMessageService and notifies PullRequestHoldService#notifyMessageArriving through the registered MessageArrivingListener that there is a message, and pushes it to the consumer immediately. ReputMessageService#doReput:
if (BrokerRole.SLAVE != DefaultMessageStore.this.getMessageStoreConfig().getBrokerRole()
&& DefaultMessageStore.this.brokerConfig.isLongPollingEnable()) {
DefaultMessageStore.this.messageArrivingListener.arriving(dispatchRequest.getTopic(),
dispatchRequest.getQueueId(), dispatchRequest.getConsumeQueueOffset() + 1,
dispatchRequest.getTagsCode());
}
public void arriving(String topic, int queueId, long logicOffset, long tagsCode) {
this.pullRequestHoldService.notifyMessageArriving(topic, queueId, logicOffset, tagsCode);
}