文章部分图片来自参考资料，侵删
Outline
We know from the previous process to send a message to a topic messageque broker's, if let's design a message consumer queue process, then how should consumers consume more than a small number of messagequeue it? There are two consumer spending patterns: Broadcast mode and trunked mode, broadcast mode is well understood by the consumer all the news; cluster mode is equivalent to think logically on multiple consumers as a whole, the most popular understanding is there a message in the cluster only a complete consumption even if consumer spending. Then the cluster model of consumption should be in accordance with what strategy it? Since then a cluster mode only allows a consumer spending, how to prevent other consumers consume it? Gets there are two ways of consumption, is a broker push yourself over it, or consumers themselves to pull it? According to these questions we will take questions to see how rocketmq design.
Message Queue Load Balancing
Consumer queue load balancing solution is to go where consumer spending issues, so as to achieve a balanced consumption.
Cluster Mode
the group as long as there is a group for human consumption even if the consumer is successful, a variety of sub-strategies
The average allocation strategy
The average group assignment message, the node 9, for example three messages, using the division, three per message. Pictures from reference materials, invasion deleted.
Consistent hashing strategy
(Distributed Hash consistency) [https://www.cnblogs.com/Benjious/p/11899188.html] can learn through this article consistent hashing
- Allocating polling strategy also mean that the above example, polling assignment, the same strategy, rocketmq will use the same virtual node on preventing uneven hash node ring, the keys on the ring and drop it on a queue broker.
Broadcast mode.
Full consumption, well understood
Consumption patterns implemented in code
Message queue load balancing balancing service by a timed run continuously executed, execution logic RebalanceImpl achieved,
public void doRebalance(final boolean isOrder) {
Map<String, SubscriptionData> subTable = this.getSubscriptionInner();
if (subTable != null) {
for (final Map.Entry<String, SubscriptionData> entry : subTable.entrySet()) {
final String topic = entry.getKey();
try {
this.rebalanceByTopic(topic, isOrder);
} catch (Throwable e) {
if (!topic.startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
log.warn("rebalanceByTopic Exception", e);
}
}
}
}
this.truncateMessageQueueNotMyTopic ();
}
public the ConcurrentMap <String, SubscriptionData> getSubscriptionInner () {
return subscriptionInner;
}
Private void rebalanceByTopic (Final String Topic, Final Boolean isOrder) {
Switch (messageModel) {
// broadcast is stored locally, the cluster is stored in the broker
Case bROADCASTING: {
// broadcast, then there must be a local to find
...
}
Case cLUSTERING: {
// cluster mode consumption schedule stored in the remote broker an end
...
AllocateMessageQueueStrategy at Strategy = this.allocateMessageQueueStrategy;
List <of the MessageQueue> allocateResult = null;
the try {
// call allocation strategy
allocateResult = strategy.allocate(
this.consumerGroup,
this.mQClientFactory.getClientId(),
mqAll,
cidAll);
} catch (Throwable e) {
log.error("AllocateMessageQueueStrategy.allocate Exception. allocateMessageQueueStrategyName={}", strategy.getName(),
e);
return;
}
Set<MessageQueue> allocateResultSet = new HashSet<MessageQueue>();
if (allocateResult != null) {
allocateResultSet.addAll(allocateResult);
}
}
}
}
The default is to use AllocateMessageQueueAveragely allocation strategy, where we think about a problem, we know that a message for the next cluster mode only can a consumer spending, the message is placed in the messagequeue, when the number of consumers is greater than the messagequeue of the time, so how to allocate it? A messagequeue can assign multiple consumers do? Now I wrote a test, the same allocation logic and AllocateMessageQueueAveragely.
public static void main(String[] args) {
Main m = new Main();
m.test();
}
public List<MessageQueue> op( String currentCID, List<MessageQueue> mqAll,
List<String> cidAll) {
List<MessageQueue> result = new ArrayList<>();
int index = cidAll.indexOf(currentCID);
int mod = mqAll.size() % cidAll.size();
int averageSize =
mqAll.size() <= cidAll.size() ? 1 : (mod > 0 && index < mod ? mqAll.size() / cidAll.size()
+ 1 : mqAll.size() / cidAll.size());
int startIndex = (mod > 0 && index < mod) ? index * averageSize : index * averageSize + mod;
int range = Math.min(averageSize, mqAll.size() - startIndex);
for (int i = 0; i < range; i++) {
result.add(mqAll.get((startIndex + i) % mqAll.size()));
}
return result;
}
public void test(){
List<String> cidAll = new ArrayList<>();
for (int i=0; i<8; i++) {
String cid = "192.168.10.86@" + i;
cidAll.add(cid);
}
List<MessageQueue> mqAll = new ArrayList<>();
for (int i=0; i<3; i++) {
MessageQueue mq = new MessageQueue(i);
mqAll.add(mq);
}
for (String cid :
cidAll) {
List <the MessageQueue> OP = OP (CID, mqAll, cidAll);
System.out.println ( "Current CID:" CID + + "");
for (the MessageQueue MQ: OP) {
System.out.println ( "go" + mq.no + "message number to get the message");
}
}
}
Current cid: 192.168.10.86@0 to get the message 0 message current cid: 192.168.10.86@1 to take a message for message No. current cid: 192.168.10.86@2 to take a message for message No. 2 Current cid: 192.168. 10.86@3 current cid: 192.168.10.86@4 current cid: 192.168.10.86@5 current cid: 192.168.10.86@6 current cid: 192.168.10.86@7
(Ignore my non-standard name, ha ha) can be seen in a clustered mode if there is a surplus of consumers so they are definitely hungry, allocate less than messagequeue.
Consumer progress
A consumer consumes a message, how to identify the consumer "has been my consumption of it," that is the problem to save your progress, for the broadcast mode, save your progress is saved in the broker ends, and this cluster mode It is stored locally on the client. Progress is mainly offsetStore storage interface, which subclasses implement LocalFileOffsetStore and RemoteBrokerOffsetStore which correspond, respectively, local storage and remote storage
Clustered mode
The following comes from references, the authors write very
在消费者客户端，RebalanceService 服务会定时地 (默认 20 秒) 从 Broker 服务器获取当前客户端所需要消费的消息队列，并与当前消费者客户端的消费队列进行对比，看是否有变化。对于每个消费队列，会从 Broker 服务器查询这个队列当前的消费偏移量。然后根据这几个消费队列，创建对应的拉取请求 PullRequest 准备从 Broker 服务器拉取消息，如下图所示:
When pulling down messages from Broker server only when a user successfully consumption, will update the local offset table. Local offset table through the service every 5 seconds, the timing synchronization to the server Broker:
public class MQClientInstance {
private void startScheduledTask() {
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
MQClientInstance.this.persistAllConsumerOffset();
}
}, 1000 * 10, this.clientConfig.getPersistConsumerOffsetInterval(), TimeUnit.MILLISECONDS);
}
}
Broker and maintained in the server side offset table is also updated every 5 seconds to disk in sequence:
public class BrokerController {
public boolean initialize() throws CloneNotSupportedException {
this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
BrokerController.this.consumerOffsetManager.persist();
}
}, 1000 * 10, this.brokerConfig.getFlushConsumerOffsetInterval(), TimeUnit.MILLISECONDS);
}
}
拉取消费
rocketmq 的 push 实际都是利用不断地去 pull 来达到 push 的效果。 push 实际是用 pull 实现的，开始的时候内存为空，生成 pullRequest 然后去 broker 请求数据，请求回来后再次生成 pullRequest再次去请求,去broker拉取消费进行的消费的服务 ： PullMessageService ，它接受 PullRequest
public class PullRequest {
private MessageQueue messageQueue;
private ProcessQueue processQueue;
}
public class ProcessQueue {
private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();
}
PullRequest 关联MessageQueue 和 ProcessQueue ，ProcessQueue 是指某个MessageQueue的消费进度抽象
/**
* Queue consumption snapshot
*
*/
public class ProcessQueue {
...
private final Logger log = ClientLogger.getLog();
//读写锁
private final ReadWriteLock lockTreeMap = new ReentrantReadWriteLock();
// TreeMap 是可以排序的 map(红黑树实现)
private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();
private final AtomicLong msgCount = new AtomicLong();
private final AtomicLong msgSize = new AtomicLong();
private final Lock lockConsume = new ReentrantLock();
/**
* A subset of msgTreeMap, will only be used when orderly consume
*/
private final TreeMap<Long, MessageExt> consumingMsgOrderlyTreeMap = new TreeMap<Long, MessageExt>();
private final AtomicLong tryUnlockTimes = new AtomicLong(0);
}
可以看到ProcessQueue维护两个消息树为了就是记录消费的进度，这在后面会介，我们也可以大概地猜测到 PullRequest 实际应该的含义是某个指定的 MessageQueue 进度发生了变化，就会生成一个 PullRequest 去远程拉取消费进行消费。 服务器在收到客户端的请求之后，会根据话题和队列 ID 定位到对应的消费队列。然后根据这条请求传入的 offset 消费队列偏移量，定位到对应的消费队列文件。偏移量指定的是消费队列文件的消费下限。
public class DefaultMessageStore implements MessageStore {
public GetMessageResult getMessage(final String group, final String topic, final int queueId, final long offset,
final int maxMsgNums,
final MessageFilter messageFilter) {
// ...
ConsumeQueue consumeQueue = findConsumeQueue(topic, queueId);
if (consumeQueue != null) {
// 首先根据消费队列的偏移量定位消费队列
SelectMappedBufferResult bufferConsumeQueue = consumeQueue.getIndexBuffer(offset);
if (bufferConsumeQueue != null) {
try {
status = GetMessageStatus.NO_MATCHED_MESSAGE;
// 最大消息长度
final int maxFilterMessageCount = Math.max(16000, maxMsgNums * ConsumeQueue.CQ_STORE_UNIT_SIZE);
// 取消息
for (; i < bufferConsumeQueue.getSize() && i < maxFilterMessageCount; i += ConsumeQueue.CQ_STORE_UNIT_SIZE) {
long offsetPy = bufferConsumeQueue.getByteBuffer().getLong();
int sizePy = bufferConsumeQueue.getByteBuffer().getInt();
// 根据消息的偏移量和消息的大小从 CommitLog 文件中取出一条消息
SelectMappedBufferResult selectResult = this.commitLog.getMessage(offsetPy, sizePy);
getResult.addMessage(selectResult);
status = GetMessageStatus.FOUND;
}
// 增加下次开始的偏移量
nextBeginOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE);
} finally {
bufferConsumeQueue.release();
}
}
}
// ...
}
}
客户端和 Broker 服务器端完整拉取消息的流程图如下所示：
消费消费
顺序消费和并发消费，顺序消费指的是消费同一个 messagequeue 里的消息，从而达到顺序消费的目的。
broker 中记录的信息
consumerFilter.json
消费者过滤相关
consumerOffset.json
消费者消费broker各个队列到了哪个位置
{
"offsetTable":{
"%RETRY%generalCallbackGroup@generalCallbackGroup":{0:0
},
"Jodie_topic_1023@CID_JODIE_1":{0:10,1:11,2:10,3:9
},
"PayTransactionTopic@mq_test_callback":{0:0,1:0,2:1,3:1
},
}
}
可以看到是有json表示的是“topic + group ”中的四个队列的消费情况
delayOffset.json 、
延时相关
subscriptionGroup.json
订阅相关，group相关的配置, 消费者订阅了哪些topic
{
"dataVersion":{
"counter":1,
"timestamp":1572054949837
},
"subscriptionGroupTable":{
"CID_ONSAPI_OWNER":{
"brokerId":0,
"consumeBroadcastEnable":true,
"consumeEnable":true,
"consumeFromMinEnable":true,
"groupName":"CID_ONSAPI_OWNER",
"notifyConsumerIdsChangedEnable":true,
"retryMaxTimes":16,
"retryQueueNums":1,
"whichBrokerWhenConsumeSlowly":1
},
"CID_ONSAPI_PERMISSION":{
"brokerId":0,
"consumeBroadcastEnable":true,
"consumeEnable":true,
"consumeFromMinEnable":true,
"groupName":"CID_ONSAPI_PERMISSION",
"notifyConsumerIdsChangedEnable":true,
"retryMaxTimes":16,
"retryQueueNums":1,
"whichBrokerWhenConsumeSlowly":1
},
...
}
}
topics.json
topic相关配置，broker 中拥有那些topic
{
"dataVersion":{
"counter":5,
"timestamp":1573745719274
},
"topicConfigTable":{
"TopicTest":{
"order":false,
"perm":6,
"readQueueNums":4,
"topicFilterType":"SINGLE_TAG",
"topicName":"TopicTest",
"topicSysFlag":0,
"writeQueueNums":4
},
"%RETRY%please_rename_unique_group_name_4":{
"order":false,
"perm":6,
"readQueueNums":1,
"topicFilterType":"SINGLE_TAG",
"topicName":"%RETRY%please_rename_unique_group_name_4",
"topicSysFlag":0,
"writeQueueNums":1
}
...
}
}
consumerQueue 图例
消费消息
消费消息有并发消费和顺序消费两种，主要的核心实现就是 ConsumeMessageConcurrentlyService 和 ConsumeMessageOrderlyService ，又它们继承的接口看的出来他们都是持有了一个线程池，并在线程池内进行消费。
public class DefaultMQPushConsumerImpl implements MQConsumerInner {
public void pullMessage(final PullRequest pullRequest) {
PullCallback pullCallback = new PullCallback() {
@Override
public void onSuccess(PullResult pullResult) {
if (pullResult != null) {
switch (pullResult.getPullStatus()) {
case FOUND:
// 消息放入处理队列的消息树中
boolean dispathToConsume = processQueue
.putMessage(pullResult.getMsgFoundList());
// 提交一个消息消费请求
DefaultMQPushConsumerImpl.this
.consumeMessageService
.submitConsumeRequest(
pullResult.getMsgFoundList(),
processQueue,
pullRequest.getMessageQueue(),
dispathToConsume);
break;
}
}
}
};
}
}
下面的代码可以看到任务来自自己投到线程池执行。
public class ConsumeMessageConcurrentlyService implements ConsumeMessageService {
class ConsumeRequest implements Runnable {
@Override
public void run() {
// ...
status = listener.consumeMessage(Collections.unmodifiableList(msgs), context);
// ...
}
}
}
消费完后
public class ConsumeMessageConcurrentlyService implements ConsumeMessageService {
public void processConsumeResult(final ConsumeConcurrentlyStatus status, /** 其它参数 **/) {
// 从消息树中删除消息
long offset = consumeRequest.getProcessQueue().removeMessage(consumeRequest.getMsgs());
//假如某个队列在这个时候刚好给移除了，不提交进度，这可能会存在重复消费的情况，所有客户端还是要自己做幂等处理
if (offset >= 0 && !consumeRequest.getProcessQueue().isDropped()) {
this.defaultMQPushConsumerImpl.getOffsetStore()
.updateOffset(consumeRequest.getMessageQueue(), offset, true);
}
}
}
而有序消费就有趣多了，我们先思考一下，顺序消费是在线程池执行了，那么如何保证有序呢，加锁。假如在执行的时候刚好进行rebalance,移除了该队列的消费，那么有序消费就不能进行了，什么意思呢？假设 Consumer-1 消费者客户端一开始需要消费 3 个消费队列，这个时候又加入了 Consumer-2 消费者客户端，并且分配到了 MessageQueue-2 消费队列。当 Consumer-1 内部的均衡服务检测到当前消费队列需要移除 MessageQueue-2 队列，
可以看到要是2号messagequeue 此时正在执行有序消费，然后却被另一个消费者进行消费，那么就不能保证有序消费了，于是在 broker 端应该也要有把锁，保证messagequeue在被有序消费时只有一个消费者持有，而上面的场景也一样，当消费者不再从messagequeue 消费的时候，也会向broker申请释放锁。
public abstract class RebalanceImpl {
private boolean updateProcessQueueTableInRebalance(final String topic,
final Set<MessageQueue> mqSet,
final boolean isOrder) {
while (it.hasNext()) {
// ...
if (mq.getTopic().equals(topic)) {
// 当前客户端不需要处理这个消息队列了
if (!mqSet.contains(mq)) {
pq.setDropped(true);
// 解锁
if (this.removeUnnecessaryMessageQueue(mq, pq)) {
// ...
}
}
// ...
}
}
}
}
class ConsumeRequest implements Runnable {
private final ProcessQueue processQueue;
private final MessageQueue messageQueue;
public ConsumeRequest(ProcessQueue processQueue, MessageQueue messageQueue) {
this.processQueue = processQueue;
this.messageQueue = messageQueue;
}
public ProcessQueue getProcessQueue() {
return processQueue;
}
public MessageQueue getMessageQueue() {
return messageQueue;
}
@Override
public void run() {
if (this.processQueue.isDropped()) {
log.warn("run, the message queue not be able to consume, because it's dropped. {}", this.messageQueue);
return;
}
final Object objLock = messageQueueLock.fetchLockObject(this.messageQueue);
//获取锁，保证线程池内只有一个线程可以对该messageQueue 进行消费
synchronized (objLock) {
//广播消费 ，或是processQueue.isLocked()已经锁住了，或是锁没过期
//那么 processQueue.isLocked() 什么时候返回true 呢？ConsumeMessageOrderlyService内有个定时任务，周期去broker 中锁住这个 messagequeue
//上文已经讲了 processQueue 和 messagequeue 是一一对应的。
if (MessageModel.BROADCASTING.equals(ConsumeMessageOrderlyService.this.defaultMQPushConsumerImpl.messageModel())
|| (this.processQueue.isLocked() && !this.processQueue.isLockExpired())) {
...
try {
//再次获取锁，这里的锁有什么用呢？我们通过查找processQueue上锁的地方，发现就是在 Rebalance重新分配消费队列的时候会上锁
//为了保证此刻不被其他消费者占用于是上锁
this.processQueue.getLockConsume().lock();
if (this.processQueue.isDropped()) {
log.warn("consumeMessage, the message queue not be able to consume, because it's dropped. {}",
this.messageQueue);
break;
}
//业务逻辑回调
status = messageListener.consumeMessage(Collections.unmodifiableList(msgs), context);
} catch (Throwable e) {
log.warn("consumeMessage exception: {} Group: {} Msgs: {} MQ: {}",
RemotingHelper.exceptionSimpleDesc(e),
ConsumeMessageOrderlyService.this.consumerGroup,
msgs,
messageQueue);
hasException = true;
} finally {
this.processQueue.getLockConsume().unlock();
}
.....
}
}
}
}
补充
service构建源码分析
Rocketmq 中创建一个service部分都继承了 ServiceThread，让我们开看一下源码
public abstract class ServiceThread implements Runnable {
private static final Logger log = LoggerFactory.getLogger(LoggerName.COMMON_LOGGER_NAME);
private static final long JOIN_TIME = 90 * 1000;
//保存了一个线程
protected final Thread thread;
/**
* 主要的阻塞方法是 await 方法，然后通过 countdown 来唤醒正在 await 的线程，每次多个线程调用进行 waitForRunning 的时候
* waitPoint 的 栅栏数量都会重置（阻塞，stop后被唤醒，又再次阻塞的情况）。
*/
protected final CountDownLatch2 waitPoint = new CountDownLatch2(1);
protected volatile AtomicBoolean hasNotified = new AtomicBoolean(false);
protected volatile boolean stopped = false;
//初始化的时候就创建一个线程
public ServiceThread() {
this.thread = new Thread(this, this.getServiceName());
}
public abstract String getServiceName();
public void start() {
this.thread.start();
}
public void shutdown() {
this.shutdown(false);
}
public void shutdown(final boolean interrupt) {
this.stopped = true;
log.info("shutdown thread " + this.getServiceName() + " interrupt " + interrupt);
if (hasNotified.compareAndSet(false, true)) {
waitPoint.countDown(); // notify
}
try {
if (interrupt) {
this.thread.interrupt();
}
long beginTime = System.currentTimeMillis();
if (!this.thread.isDaemon()) {
this.thread.join(this.getJointime());
}
long eclipseTime = System.currentTimeMillis() - beginTime;
log.info("join thread " + this.getServiceName() + " eclipse time(ms) " + eclipseTime + " "
+ this.getJointime());
} catch (InterruptedException e) {
log.error("Interrupted", e);
}
}
public long getJointime() {
return JOIN_TIME;
}
public void stop() {
this.stop(false);
}
public void stop(final boolean interrupt) {
this.stopped = true;
log.info("stop thread " + this.getServiceName() + " interrupt " + interrupt);
if (hasNotified.compareAndSet(false, true)) {
waitPoint.countDown(); // notify
}
if (interrupt) {
this.thread.interrupt();
}
}
public void makeStop() {
this.stopped = true;
log.info("makestop thread " + this.getServiceName());
}
public void wakeup() {
if (hasNotified.compareAndSet(false, true)) {
waitPoint.countDown(); // notify
}
}
/**
* 当只有一个线程的时候直接案通过CAS 成功后执行，多个线程则需要等待一段时间间隔后执行
*
* @param interval 等待的时间
*/
protected void waitForRunning(long interval) {
if (hasNotified.compareAndSet(true, false)) {
this.onWaitEnd();
return;
}
//entry to wait
waitPoint.reset();
try {
waitPoint.await(interval, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
log.error("Interrupted", e);
} finally {
hasNotified.set(false);
this.onWaitEnd();
}
}
protected void onWaitEnd() {
}
public boolean isStopped() {
return stopped;
}
}
He may know the structure creates a thread run method implemented for performing sub-class, and their countDownLatch and Atomic atomic class is mainly used to deal with the case of simultaneous operation of multiple threads.
Reference material
- http://silence.work/2019/03/03/RocketMQ%20%E6%B6%88%E8%B4%B9%E6%B6%88%E6%81%AF%E8%BF%87%E7%A8%8B%E5%88%86%E6%9E%90/
- https://www.kunzhao.org/blog/2018/04/08/rocketmq-message-index-flow/
- https://cloud.tencent.com/developer/article/1554950