As we all know, the RocketMQ consumption mode has PULL mode and PUSH mode, but they are all PULL modes in essence, and in actual use, PUSH mode is generally used.
However, the PUSH mode of RocketMQ has obvious deficiencies, which are mainly reflected in the following aspects:
-
There is a backlog of news, and adding consumers may not necessarily solve it. The PUSH mode is as follows:
In the figure above, each consumer in the consumer group consumes two MessageQueues. In this case, increasing consumers can increase consumption capacity.
But in the picture below, each consumer consumes a MessageQueue, because the same MessageQueue can only be consumed by one consumer in the same consumption group, so adding consumers does not increase consumption capacity.
-
There are many processing logics on the client side, such as load balancing, offset management, and processing after consumption failure (such as sending failure messages back to Broker), all of which are on the client side.
-
If other languages are supported, the client will become heavier and heavier.
-
The consumer machine hangs, which may cause a backlog of messages, as shown in the following figure:
The client is responsible for balancing. The MessageQueue0 queue is assigned to Consumer0 for exclusive consumption. If Consumer0 hangs up, but the service is not hung up, it cannot be offline from the Name Server, because the messages pulled by Consumer0 cannot be consumed. It is impossible to send a request to update the Offset to the Broker, which eventually leads to a backlog of messages. In this case, you can only manually take Consumer0 offline or restart Consumer0.
RocketMQ 5.0 introduces POP Consumer in order to solve the above problems of PUSH Consumer.
1 POP client
The background of the introduction of POP mode client is RocketMQ 5.0. In order to better embrace cloud native, the client needs to be transformed into a stateless lightweight client. The client in RocketMQ 4.x has load balancing, authority management, and consumption management. and other functions have been moved from the client to the Proxy.
The POP consumption mode is as follows:
All four consumers can consume all queues on Broker1 and Broker2, so that even if a certain consumer hangs, other consumers can consume without causing a backlog of messages.
At the same time, as can be seen from the above figure, the POP client has another advantage. Increasing the number of consumers can improve consumption capacity, and it is not limited by the number of MessageQueue and the number of consumers.
Compared with the PUSH mode, after the POP mode pulls the message, it will set a POP_CK attribute, the code is as follows:
//MQClientAPIImpl.java
if (requestHeader instanceof PopMessageRequestHeader) {
if (startOffsetInfo == null) {
// we should set the check point info to extraInfo field , if the command is popMsg
// find pop ck offset
String key = messageExt.getTopic() + messageExt.getQueueId();
if (!map.containsKey(messageExt.getTopic() + messageExt.getQueueId())) {
map.put(key, ExtraInfoUtil.buildExtraInfo(messageExt.getQueueOffset(), responseHeader.getPopTime(), responseHeader.getInvisibleTime(), responseHeader.getReviveQid(),
messageExt.getTopic(), brokerName, messageExt.getQueueId()));
}
messageExt.getProperties().put(MessageConst.PROPERTY_POP_CK, map.get(key) + MessageConst.KEY_SEPARATOR + messageExt.getQueueOffset());
} else {
String queueIdKey = ExtraInfoUtil.getStartOffsetInfoMapKey(messageExt.getTopic(), messageExt.getQueueId());
String queueOffsetKey = ExtraInfoUtil.getQueueOffsetMapKey(messageExt.getTopic(), messageExt.getQueueId(), messageExt.getQueueOffset());
int index = sortMap.get(queueIdKey).indexOf(messageExt.getQueueOffset());
Long msgQueueOffset = msgOffsetInfo.get(queueIdKey).get(index);
messageExt.getProperties().put(MessageConst.PROPERTY_POP_CK,
ExtraInfoUtil.buildExtraInfo(startOffsetInfo.get(queueIdKey), responseHeader.getPopTime(), responseHeader.getInvisibleTime(),
responseHeader.getReviveQid(), messageExt.getTopic(), brokerName, messageExt.getQueueId(), msgQueueOffset)
);
//...
}
}
It can be seen that the POP_CK attribute includes parameters such as brokerName, Topic, QueueId, and offset, through which a message can be uniquely identified.
From the above code, we can also see that there is an invisibleTime attribute in responseHeader. The function of this attribute is that after the consumer pulls a message through the POP mode, the message is invisible on the Broker side during this period (invisibleTime). , the consumer will not pull repeatedly if he pulls again. But if after this period of time, the consumer has not returned an ACK to the Broker, this message will become visible and be pulled by the consumer again .
After the consumption is completed, send an ACK message to the Broker, see the code below:
public void ackMessageAsync(
final String addr,
final long timeOut,
final AckCallback ackCallback,
final AckMessageRequestHeader requestHeader //
) throws RemotingException, MQBrokerException, InterruptedException {
final RemotingCommand request = RemotingCommand.createRequestCommand(RequestCode.ACK_MESSAGE, requestHeader);
this.remotingClient.invokeAsync(addr, request, timeOut, new BaseInvokeCallback(MQClientAPIImpl.this) {
@Override
public void onComplete(ResponseFuture responseFuture) {
RemotingCommand response = responseFuture.getResponseCommand();
if (response != null) {
try {
AckResult ackResult = new AckResult();
if (ResponseCode.SUCCESS == response.getCode()) {
ackResult.setStatus(AckStatus.OK);
} //...
assert ackResult != null;
ackCallback.onSuccess(ackResult);
} //...
} else {
//...
}
}
});
}
2. Broker
From the above introduction, we can see that each consumer can pull messages from all MessageQueues of Broker, so if multiple consumers pull messages from one MessageQueue, is it possible to consume repeatedly?
When the Broker receives the message pull request, when pulling the message from the MessageStore, it will first lock the MessageQueue. After the lock is successful, the message will be pulled. This is because other clients will fail to lock when they pull the message.
//PopMessageProcessor.java
String lockKey = topic + PopAckConstants.SPLIT + requestHeader.getConsumerGroup() + PopAckConstants.SPLIT + queueId;
long offset = getPopOffset(topic, requestHeader, queueId, false, lockKey);
if (!queueLockManager.tryLock(lockKey)) {
restNum = this.brokerController.getMessageStore().getMaxOffsetInQueue(topic, queueId) - offset + restNum;
return restNum;
}
After the Broker pulls the message from the MessageStore, it will define a CheckPoint and put it in the cache. The code is as follows:
//PopMessageProcessor.java
private long popMsgFromQueue(boolean isRetry, GetMessageResult getMessageResult,
PopMessageRequestHeader requestHeader, int queueId, long restNum, int reviveQid,
Channel channel, long popTime,
ExpressionMessageFilter messageFilter, StringBuilder startOffsetInfo,
StringBuilder msgOffsetInfo, StringBuilder orderCountInfo) {
String topic = isRetry ? KeyBuilder.buildPopRetryTopic(requestHeader.getTopic(),
requestHeader.getConsumerGroup()) : requestHeader.getTopic();
String lockKey =
topic + PopAckConstants.SPLIT + requestHeader.getConsumerGroup() + PopAckConstants.SPLIT + queueId;
//...
offset = getPopOffset(topic, requestHeader, queueId, true, lockKey);
GetMessageResult getMessageTmpResult = null;
try {
//...
restNum = getMessageTmpResult.getMaxOffset() - getMessageTmpResult.getNextBeginOffset() + restNum;
if (!getMessageTmpResult.getMessageMapedList().isEmpty()) {
if (isOrder) {
//...
} else {
appendCheckPoint(requestHeader, topic, reviveQid, queueId, offset, getMessageTmpResult, popTime, this.brokerController.getBrokerConfig().getBrokerName());
}
} //...
} //...
return restNum;
}
After the Broker receives the ACK from the consumer, it will remove the CheckPoint from the cache.
If the Broker has not received the ACK, it will remove the CheckPoint from the cache, and at the same time send the CheckPoint to the MessageStore, and the MessageStore will send it to the retry queue. code show as below:
boolean removeCk = !this.serving;
// ck will be timeout
if (point.getReviveTime() - now < brokerController.getBrokerConfig().getPopCkStayBufferTimeOut()) {
removeCk = true;
}
// the time stayed is too long
if (now - point.getPopTime() > brokerController.getBrokerConfig().getPopCkStayBufferTime()) {
removeCk = true;
}
// double check
if (removeCk) {
// put buffer ak to store
if (pointWrapper.getReviveQueueOffset() < 0) {
putCkToStore(pointWrapper, false);
}
}
}
3 Summary
POP clients have many advantages, summarized as follows:
-
Stateless, better embrace cloud native;
-
Calculation-related functions are moved down to Proxy, which is more lightweight;
-
Expansion of consumption capacity is not limited by the number of MessageQueue;
-
Consumers hang up and will not cause a backlog of messages.