Prerequisite introduction
There are generally two ways to obtain messages in RocketMQ, one is pull (consumers actively go to the broker to pull), and the other is push (actively pushes to consumers). The relevant information has been introduced in the previous chapter. Push operation, the next chapter will introduce the consumption mechanism system of Pull operation.
DefaultMQPullConsumer
The biggest difference between DefaultMQPullConsumer and DefaultMQPushConsumer is that which queue messages are consumed, which offset to start consuming, and when to submit the consumption offset are all controlled by the program itself. Let's introduce the internal principles of DefaultMQPullConsumer.
overall process execution
DefaultMQPullConsumer usage example
public class MQPullConsumer {
private static final Map OFFSE_TABLE = new HashMap();
public static void main(String[] args) throws MQClientException {
DefaultMQPullConsumer consumer = new DefaultMQPullConsumer("groupName");
consumer.setNamesrvAddr("name-serverl-ip:9876;name-server2-ip:9876");
consumer.start();
Set mqs = consumer.fetchSubscribeMessageQueues("order-topic");
for(MessageQueue mq:mqs){
try {
long offset = consumer.fetchConsumeOffset(mq,true);
while(true){
PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
putMessageQueueOffset(mq,pullResult.getNextBeginOffset());
switch(pullResult.getPullStatus()){
case FOUND:
List messageExtList = pullResult.getMsgFoundList();
for (MessageExt m : messageExtList) {
System.out.println(new String(m.getBody()));
}
break;
case NO_MATCHED_MSG:
break;
case NO_NEW_MSG:
break;
case OFFSET_ILLEGAL:
break;
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
consumer.shutdown();
}
private static void putMessageQueueOffset(MessageQueue mq,
long nextBeginOffset) {
OFFSE_TABLE.put(mq, nextBeginOffset);
}
private static Long getMessageQueueOffset(MessageQueue mq) {
Long offset = OFFSE_TABLE.get(mq);
if(offset != null){
return offset;
}
return 0l;
}
}
- Consumer start: consumer.start();
- Get all the message queues under the topic: here is obtained from the nameserver according to the topic. Here we can modify it to obtain the queue information from other locations.
Set mqs = consumer.fetchSubscribeMessageQueues("topicTest");
for(MessageQueue mq:mqs){
try {
long offset = consumer.fetchConsumeOffset(mq,true);
while(true){
PullResult pullResult = consumer.pullBlockIfNotFound(mq, null, getMessageQueueOffset(mq), 32);
}
The overall process of DefaultMQPullConsumer
Starting DefaultMQPullConsumer is done by calling the start() method
DefaultMQPullConsumer pull source code analysis
Analyze the process of DefaultMQPullConsumer pulling messages
consumer.fetchSubscribeMessageQueues("order-topic")
Pull all message queues from the specified topic
Set mqs = consumer.fetchSubscribeMessageQueues("order-topic");
Core source code analysis
fetchSubscribeMessageQueues()
- The read queue information of the specified topic (GET_ROUTEINTO_BY_TOPIC) can be obtained by calling the fetchSubscribeMessageQueues() method. It sends a GetRouteInfoRequest request to the nameserver, and the request content is GET_ROUTEINTO_BY_TOPIC. The nameserver sends the number of read queues under the topic to the consumer, and then the consumer uses the following code to create a MessageQueue object with the same number of read queues.
- Each MessageQueue object records the topic, broker name and read queue number. Finally, fetchSubscribeMessageQueues() returns the MessageQueue object collection to the caller.
- Send a request to NameServer to obtain the Broker information and topic configuration information corresponding to the topic parameter, that is, the TopicRouteData object.
public Set fetchSubscribeMessageQueues(String topic) throws MQClientException {
try {
TopicRouteData topicRouteData = this.mQClientFactory.getMQClientAPIImpl().getTopicRouteInfoFromNameServer(topic, timeoutMillis);
if (topicRouteData != null) {
Set mqList = MQClientInstance.topicRouteData2TopicSubscribeInfo(topic, topicRouteData);
if (!mqList.isEmpty()) {
return mqList;
} else {
throw new MQClientException("Can not find Message Queue for this topic, " + topic + " Namesrv return empty", null);
}
}
} catch (Exception e) {
throw new MQClientException(
"Can not find Message Queue for this topic, " + topic + FAQUrl.suggestTodo(FAQUrl.MQLIST_NOT_EXIST),
e);
}
throw new MQClientException("Unknow why, Can not find Message Queue for this topic, " + topic, null);
}
Traversal process TopicRouteData
Traverse each QueueData object in the QueueData list of the TopicRouteData object, first determine whether the QueueData object has read permission, if so, create readQueueNums MessageQueue objects based on the readQueueNums value of the QueueData object, and form a MessageQueue collection; finally return to the MessageQueue collection
public static Set topicRouteData2TopicSubscribeInfo(final String topic, final TopicRouteData route) {
Set mqList = new HashSet();
List qds = route.getQueueDatas();
for (QueueData qd : qds) {
if (PermName.isReadable(qd.getPerm())) {
for (int i = 0; i < qd.getReadQueueNums(); i++) {
MessageQueue mq = new MessageQueue(topic, qd.getBrokerName(), i);
mqList.add(mq);
}
}
}
return mqList;
}
consumer.fetchConsumeOffset
This method is used to obtain the message content starting from the offset position under the MessageQueue queue, where maxNums=32 indicates the maximum number of messages obtained, and offset is the starting consumption position of the MessageQueue object.
DefaultMQPullConsumer.fetchConsumeOffset(MessageQueue mq, boolean fromStore)
fetchConsumeOffset() has two input parameters. The first parameter represents the queue, and the second parameter represents whether to obtain the consumption displacement of the queue from the broker. true means to obtain it from the broker, and false means to obtain it from the local record. If it cannot be obtained locally, then Obtained from broker. The local acquisition mentioned here refers to the acquisition from the RemoteBrokerOffsetStore.offsetTable property, which records the consumption displacement of each queue. The offsetTable is updated when the displacement is obtained from the broker.
pullBlockIfNotFound pull information
rocketmq provides multiple pull methods, you can use the pullBlockIfNotFound() method or the pull() method. The difference between the two is that if there is no message in the queue, the timeout time of the two methods is different. pullBlockIfNotFound will wait for 30s to return an empty result, and pull will wait for 10s to return an empty result.
However, the input parameters of the pull method can adjust the timeout, and pullBlockIfNotFound requires modifying the DefaultMQPullConsumer.consumerPullTimeoutMillis parameter. However, the underlying logic of both method calls is the same, which is to call the DefaultMQPullConsumerImpl.pullSyncImpl() method to obtain the message. Let's analyze the pullSyncImpl() method.
public PullResult pullBlockIfNotFound(MessageQueue mq, String subExpression, long offset, int maxNums)
throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
return this.pullSyncImpl(mq, subExpression, offset, maxNums, true, this.getDefaultMQPullConsumer().getConsumerPullTimeoutMillis());
}
Obtain the consumption progress of the MessageQueue queue to set the parameter offset value. This method finally calls pullSyncImpl to obtain relevant result data.
- Parameter 1: Message queue (the required message queue of the corresponding topic can be obtained by calling the consumer's fetchSubscibeMessageQueue(topic));
- Parameter 2: the expression to be used for filtering;
- Parameter 3: The offset is the progress of the consumption queue;
- Parameter 4: Get the maximum value of the message at one time;
DefaultMQPullConsumerImpl.pullSyncImpl(MessageQueue mq, String subExpression, long offset, int maxNums, boolean block)
Implementation process of DefaultMQPullConsumerImpl.pullSyncImpl
private PullResult pullSyncImpl(MessageQueue mq, SubscriptionData subscriptionData, long offset, int maxNums, boolean block,
long timeout)
throws MQClientException, RemotingException, MQBrokerException, InterruptedException {
this.isRunning();
if (null == mq) {
throw new MQClientException("mq is null", null);
}
if (offset < 0) {
throw new MQClientException("offset < 0", null);
}
if (maxNums 0) {
throw new MQClientException("maxNums , null);
}
this.subscriptionAutomatically(mq.getTopic());
int sysFlag = PullSysFlag.buildSysFlag(false, block, true, false);
long timeoutMillis = block ? this.defaultMQPullConsumer.getConsumerTimeoutMillisWhenSuspend() : timeout;
boolean isTagType = ExpressionType.isTagType(subscriptionData.getExpressionType());
PullResult pullResult = this.pullAPIWrapper.pullKernelImpl(
mq,
subscriptionData.getSubString(),
subscriptionData.getExpressionType(),
isTagType ? 0L : subscriptionData.getSubVersion(),
offset,
maxNums,
sysFlag,
0,
this.defaultMQPullConsumer.getBrokerSuspendMaxTimeMillis(),
timeoutMillis,
CommunicationMode.SYNC,
null
);
this.pullAPIWrapper.processPullResult(mq, pullResult, subscriptionData);
this.resetTopic(pullResult.getMsgFoundList());
if (!this.consumeMessageHookList.isEmpty()) {
ConsumeMessageContext consumeMessageContext = null;
consumeMessageContext = new ConsumeMessageContext();
consumeMessageContext.setNamespace(defaultMQPullConsumer.getNamespace());
consumeMessageContext.setConsumerGroup(this.groupName());
consumeMessageContext.setMq(mq);
consumeMessageContext.setMsgList(pullResult.getMsgFoundList());
consumeMessageContext.setSuccess(false);
this.executeHookBefore(consumeMessageContext);
consumeMessageContext.setStatus(ConsumeConcurrentlyStatus.CONSUME_SUCCESS.toString());
consumeMessageContext.setSuccess(true);
this.executeHookAfter(consumeMessageContext);
}
return pullResult;
}
Check whether the topic of the MessageQueue object is in RebalanceImpl.subscriptionInner:ConcurrentHashMap
SubscriptionData subscriptionData;
try {
subscriptionData = FilterAPI.buildSubscriptionData(this.defaultMQPullConsumer.getConsumerGroup(),
mq.getTopic(), subExpression);
} catch (Exception e) {
throw new MQClientException("parse subscription error", e);
}
long timeoutMillis = block ? this.defaultMQPullConsumer.getConsumerTimeoutMillisWhenSuspend() : timeout;
PullResult pullResult = this.pullAPIWrapper.pullKernelImpl(
mq,
subscriptionData.getSubString(),
0L,
offset,
maxNums,
sysFlag,
0,
this.defaultMQPullConsumer.getBrokerSuspendMaxTimeMillis(),
timeoutMillis,
CommunicationMode.SYNC,
null
);
this.pullAPIWrapper.processPullResult(mq, pullResult, subscriptionData);
if (!this.consumeMessageHookList.isEmpty()) {
ConsumeMessageContext consumeMessageContext = null;
consumeMessageContext = new ConsumeMessageContext();
consumeMessageContext.setConsumerGroup(this.groupName());
consumeMessageContext.setMq(mq);
consumeMessageContext.setMsgList(pullResult.getMsgFoundList());
consumeMessageContext.setSuccess(false);
this.executeHookBefore(consumeMessageContext);
consumeMessageContext.setStatus(ConsumeConcurrentlyStatus.CONSUME_SUCCESS.toString());
consumeMessageContext.setSuccess(true);
this.executeHookAfter(consumeMessageContext);
}
return pullResult;
}
Comparison of Push and Pull operations
- push-Advantages: timeliness, unified processing on the server and easy implementation
- push-disadvantages: easy to cause accumulation and uncontrollable load performance
- pull-Advantages: Convenient to obtain message status, controllable load balancing performance
- pull-disadvantages: poor timeliness
Using DefaultMQPullConsumer to pull messages, the submission displacement sent to the broker is always 0, so the broker cannot record the effective displacement, and the program needs to record and control the submission displacement by itself.
share resources
To obtain the above resources, please visit the open source project and click to jump.