How to obtain and maintain consumer RocketMQ Consumer progress?

background

Cosumer news consumption process is more complex, more important are the following modules: Maintenance consumption schedule for messages, message filtering, load balancing, message handling, postback confirmation. Due to space limitations, this article describes how to obtain and maintain consumer Consumer is progress. Because of these steps are closely linked, interpenetrating situation may occur.

Consumer progress file

Our previous article RocketMQ how to build ConsumerQueue of? In spoken, written by acquiring already CommitLog message from a service thread asynchronous, then the key information messages location, size, tagsCode and so saved to choose the right ConsumerQueue in. ConsumerQueue is an index file, save the information on a topic the following message in the location in CommitLog. We consume messages, start reading a message in this ConsumerQueue position, and then go CommitLog fetch news, which is typical of the practice space for time. At the same time, we need to record where we read, and read the next time to continue reading from where you left off forward, so we also need to file a consumer to save progress. In RocketMQ in this document is ConsumserOffset.json, at a store / config directory as follows:

the content of consumerOffset.json like this:

wherein test_url @ sub_localtest is the primary key, rule topic @ consumerGroup, each content is in ConsumerQueue consumer progress. The next example No. 0 Queue Number 2599,1 Queue should consume next should consume No. 2602,6 Queue 102 should consume next. This schedule is accumulated by 1, corresponding to ConsumerQueue fixed 20 bytes.
Now an experiment, first sends a message via Producer (code is very simple, not posted), the transmission result is as follows:

SendResult [sendStatus=SEND_OK, msgId=C0A84D05091058644D4664E1427C0000, offsetMsgId=C0A84D0000002A9F00000000001ED9BA, messageQueue=MessageQueue [topic=test_url, brokerName=broker-a, queueId=6], queueOffset=102]

Can be seen from the results, the message stored in the 102 position of broker-a MesssageQueue-6. Then we start the Consumer, consumption of this message:

2020-01-20 14:08:04.230 INFO  [MessageThread_3] c.c.r.example.simplest.Consumer - 收到消息:topic=test_url,msgId=C0A84D05091058644D4664E1427C0000

The message was successfully consumed, then we could look ConsumerOffset.json file contents:

See? The last shot, the No. 6 Queue consumption schedule is 102, this turned into a 103, has been successfully consume a message.

Now we get to the bottom to begin: when consumption message in Consumer, consumption of this progress is how to obtain and maintain it?

Where to start spending?

Consumer by calling

consumer.start();

Start time, consumption is loaded schedule, as follows:

//DefaultMQPushConsumerImpl.start()方法
if (this.defaultMQPushConsumer.getOffsetStore() != null) {
     this.offsetStore = this.defaultMQPushConsumer.getOffsetStore(); // 如果本地有直接加载
} else {
	switch (this.defaultMQPushConsumer.getMessageModel()) {
                        case BROADCASTING:
                            this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
                            break;
                        case CLUSTERING:
                            this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup()); // 集群模式下生成一个RemoteBrokerOffsetSotre，消费进度就保存在broker端
                            break;
                        default:
                            break;
                    }
                    this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
                }
                this.offsetStore.load(); // 加载本地进度

OffsetStore above-mentioned object, which is an interface, as shown below:

public interface OffsetStore {
    void load() throws MQClientException;
    void updateOffset(final MessageQueue mq, final long offset, final boolean increaseOnly);
    long readOffset(final MessageQueue mq, final ReadOffsetType type);
    void persistAll(final Set<MessageQueue> mqs);
    void persist(final MessageQueue mq);
    void removeOffset(MessageQueue mq);
    Map<MessageQueue, Long> cloneOffsetTable(String topic);
    void updateConsumeOffsetToBroker(MessageQueue mq, long offset, boolean isOneway) throws RemotingException,
        MQBrokerException, InterruptedException, MQClientException;
}

OffsetStore a method of operating consumption of progress, such as: load consumption schedule, read consumer progress, update the progress of consumption and so on. In the cluster consumption patterns, consumption does not progress and persistence in the Consumer side, but stored in a remote Broker end, for example used above is RemoteBrokerOffsetStore categories:

public class RemoteBrokerOffsetStore implements OffsetStore {
    private final static InternalLogger log = ClientLogger.getLog();
    private final MQClientInstance mQClientFactory; // 客户端实例
    private final String groupName; // 集群名称
    private ConcurrentMap<MessageQueue, AtomicLong> offsetTable =
        new ConcurrentHashMap<MessageQueue, AtomicLong>(); // 每个Queue的消费进度
}

Because consumption saving progress in the broker terminals for local consumption Load method to load the schedule is empty in RemoteBrokerOffsetStore in, do not do anything, really read consumption progress is achieved through readOffset method.
So far we know, the progress of consumption is a consumerOffset.json file exists broker in the end, read this file by readOffset method to know where to start spending.

Load Balancing brief

Before understanding readOffset method of reading the progress of the consumer, need to know when it will simply call this method.
We mentioned at the beginning of the article module when the Consumer consume messages, including load balancing this module. A topic multiple consumer queues, and the same group the following group may have multiple Consumer subscribed to this topic, so these queues need to follow certain policies assigned to a group with the following Consumer spending, for example, the following average distribution:

on the map in, TOPIC_A there are five queues, there are two consumer subscribed, according to the average distribution, it is that Consumer1 consumption of which three queues, Consumer2 consumption of which two queues. This is load balancing.
Consumer assigned a message queue to several, will be used to create the respective processing queue (ProcessQueue, when the consumer will use the message), and the request will be generated, a pull message (PullRequest, request message several messages ), the request sent to the broker is not a true end of the message acquiring request, but stored in a queue inside the obstruction, then it is read by a dedicated service thread pull message and assembled acquisition request message, sent to the broker side (so do of course is to get the benefits of asynchronous). This request PullRequest which holds the next interim progress consumption, as follows:

will generate only a PullRequest Here, the subsequent pulling consumerGroup all the messages are repeated using the Queue PullRequest, wherein only update the like nextOffset parameter. PullRequest class is shown below:

public class PullRequest {
    private String consumerGroup; // 消费分组
    private MessageQueue messageQueue; // 消费队列
    private ProcessQueue processQueue; // 消息处理队列
    private long nextOffset; // 消费进度
    private boolean lockedFirst = false; // 是否锁住
}

Calculated consumption schedule

When generating PullRequest, it will calculate where to begin consumption (computPullFromWhere). RocketMQ consumer default starting point is CONSUME_FROM_LAST_OFFSET (start spending from the last offset), so computPullFromWhere approach, come that this case:

 case CONSUME_FROM_LAST_OFFSET: {
                long lastOffset = offsetStore.readOffset(mq, ReadOffsetType.READ_FROM_STORE);// 读取偏移
                if (lastOffset >= 0) { // 正常偏移
                    result = lastOffset;
                }
                // First start,no offset
                else if (-1 == lastOffset) { // 初次启动，broker没有偏移，readOffset返回-1
                    if (mq.getTopic().startsWith(MixAll.RETRY_GROUP_TOPIC_PREFIX)) {
                        result = 0L; // 消息重试
                    } else {
                        try {
                            result = this.mQClientFactory.getMQAdminImpl().maxOffset(mq);// 获取该消息队列消费的最大偏移
                        } catch (MQClientException e) {
                            result = -1;
                        }
                    }
                } else {
                    result = -1; // 异常情况readOffset返回-2，这里再返-1
                }
                break;
            }

Here come readOffset the method we mentioned above!

readOffset

readOffse方法t的入参是当前分配到的messageQueue和 固定的ReadOffsetType.READ_FROM_STORE，意思就从远程Broker读取该MessageQueue的消费进度。因此走到的是READ_FROM_STORE这个分支，如下所示：

case READ_FROM_STORE: {
                    try {
                        long brokerOffset = this.fetchConsumeOffsetFromBroker(mq);// 从broker获取消费进度
                        AtomicLong offset = new AtomicLong(brokerOffset);
                        this.updateOffset(mq, offset.get(), false); // 更新消费进度，更新的是RemoteBrokerOffsetStore.offsetTable这个表
                        return brokerOffset;
                    }
                    // No offset in broker
                    catch (MQBrokerException e) {
                        return -1;
                    }
                    //Other exceptions
                    catch (Exception e) {
                        log.warn("fetchConsumeOffsetFromBroker exception, " + mq, e);
                        return -2;
                    }
                }

fetchConsumerOffsetFromBroker就是往该MessageQueue所在的broker发送获取消费进度的请求了，底层通讯之前的文章已经讲过了，这里就不在赘述了。在Broker端，消费进度保存在ConsumerOffsetManager里面：

key是Topic@ConsumerGroup，value存的是queueId和offset的映射关系。查找消费进度的代码如下：

long offset =
            this.brokerController.getConsumerOffsetManager().queryOffset(
                requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId());

从key来看，topic的消费进度是按照ConsumerGroup来区分的，不同的ConsumerGroup下该MessageQueue的消费进度互不影响，这一点也很好理解。

消息获取与更新消费进度

到目前为止，我们知道了每个MessageQueue的消费进度存在对应的Broker端，在负载均衡服务对每个Topic做负载均衡的时候，创建了PullRequest，并读取了消费进度offset。然后将PullRequest放入了一个阻塞队列中(pullRequestQueue)，供专门的拉取消息线程服务(PullMessageService)读取，然后发起真正的拉取消息请求。这里更新消费进度与获取消息密切相关，因此会涉及一些获取消息的内容。

PullMessageSevice是一个服务线程，专门用于拉取消息，其run方法如下所示：

@Override
    public void run() {
        log.info(this.getServiceName() + " service started");

        while (!this.isStopped()) {
            try {
                PullRequest pullRequest = this.pullRequestQueue.take(); // 从阻塞队列中获取一个PullsRequest
                this.pullMessage(pullRequest); // 拉取消息
            } catch (InterruptedException ignored) {
            } catch (Exception e) {
                log.error("Pull Message Service Run Method exception", e);
            }
        }

        log.info(this.getServiceName() + " service end");
    }

首先从阻塞队列pullRequestQueue中取出PullRequest，然后就是开始调用pullMessage方法获取消息了。调用最终会走到DefaultMQPushConsumerImpl的pullMessage方法中来，代码很多并且如何拉取消息不是本次重点内容，我们这里只贴最后的发送请求部分，理解的其中的参数，也就明白了消息获取请求。

try {
            this.pullAPIWrapper.pullKernelImpl(
                pullRequest.getMessageQueue(), //消息队列
                subExpression,// 订阅表达式，例如"TAG_A"
                subscriptionData.getExpressionType(), // 表达式类型，例如"TAG"
                subscriptionData.getSubVersion(), // 版本号
                pullRequest.getNextOffset(), // 下个消息进度
                this.defaultMQPushConsumer.getPullBatchSize(), // 一次性拉取多少条，默认32
                sysFlag,// 一些标志位集合，暂时不关心
                commitOffsetValue, //
                BROKER_SUSPEND_MAX_TIME_MILLIS,// 长轮询时被hold住时间，默认15秒
                CONSUMER_TIMEOUT_MILLIS_WHEN_SUSPEND,// 调用超时时间，默认30秒
                CommunicationMode.ASYNC, // 异步通讯
                pullCallback// 回调
            );
        } catch (Exception e) {
            log.error("pullKernelImpl exception", e);
            this.executePullRequestLater(pullRequest, PULL_TIME_DELAY_MILLS_WHEN_EXCEPTION);
        }

Consumer端更新消费进度

由于发送消息获取请求是异步操作，返回处理在pullCallback里面，因此我们可以大胆猜测，Consumer端消费进度的更新也肯定在这里面。确实，在pullCallback里面会将PullRequest的nextOffset更新：

broker端更新消费进度

由于消息处理不是本次重点内容，所以Broker端对获取消息的处理，我们不打算深入，仅仅需要知道Broker端获取消息后，会计算出一个nextBeginOffset，它就是下个消费进度，然后会返回到Consumer端去供Consumer端更新进度，如下所示：

nextBeginOffset = offset + (i / ConsumeQueue.CQ_STORE_UNIT_SIZE);

其次，获取完消息后，broker端也会更新消费进度，如下所示：

boolean storeOffsetEnable = brokerAllowSuspend; // brokerAllowSuspend默认是true，如果没有消息就会hold住请求
        storeOffsetEnable = storeOffsetEnable && hasCommitOffsetFlag; // 拉取消息时如果允许提交消费进度，commitOffsetFlag就有
        storeOffsetEnable = storeOffsetEnable 
            && this.brokerController.getMessageStoreConfig().getBrokerRole() != BrokerRole.SLAVE;// 所以Broker master节点默认情况下这个是true
        if (storeOffsetEnable) {
            this.brokerController.getConsumerOffsetManager().commitOffset(RemotingHelper.parseChannelRemoteAddr(channel),
                requestHeader.getConsumerGroup(), requestHeader.getTopic(), requestHeader.getQueueId(), requestHeader.getCommitOffset()); // 保存消费进度，写入offsetTable
        }

消费进度持久化

Broker更新消费进度，仅仅是更新了offsetTable这个表，并没有涉及到ConsumerOffset.json这个文件。其实，在Broker初始化时，会启动一项定时任务，定期保存tableOffset到ConsumerOffset.json文件中，如下所示：

this.scheduledExecutorService.scheduleAtFixedRate(new Runnable() {
                @Override
                public void run() {
                    try {
                        BrokerController.this.consumerOffsetManager.persist(); // 保存文件
                    } catch (Throwable e) {
                        log.error("schedule persist consumerOffset error.", e);
                    }
                }
            }, 1000 * 10, this.brokerConfig.getFlushConsumerOffsetInterval(), TimeUnit.MILLISECONDS);

flushConsumerOffsetIntervval默认是5s，也就是每隔5保存一次消费进度到文件中。保存的过程是先将原来的文件存到ConsumerOffset.json.bak文件中，然后将新的内容存入ConsumerOffset.json文件。
至此，ConsumerQueue的消费进度维护就算完成了。

小结

Topic RocketMQ each message in each ConsumerGroup, each MessageQueue consumption schedule is the presence of a broker end consumerOffset.json file. Consumer end when activated, creates PullRequest request, then sends a request Broker Gets the next consumer progress, read the next consumption Broker progress and returned to the Consumer side. Consumer then read by a separate service PullRequest thread and pull message accordingly, after the end Broker to get a message, will update the progress of consumption. There is also a separate timed tasks on a regular basis the progress of the file saving consumption, and back up the original file.