RocketMQ - MessageOffset

ConsumeFromWhere

https://blog.csdn.net/prestigeding/article/details/96576932
For a new consumer group, neither the cluster mode nor the broadcast mode will store the consumption progress of the consumer group, which can be understood as -1

key	remark
CONSUME_FROM_LAST_OFFSET	The default strategy, start consumption from the end of the queue, that is, skip historical messages
CONSUME_FROM_FIRST_OFFSET	Start to consume from the very beginning of the queue, that is, set to start reading from the smallest Offset
CONSUME_FROM_TIMESTAMP	Start consuming from a certain point in time, used in conjunction with setConsumeTimestamp(), the default is to start consuming 30 minutes before the consumer starts

注意设置读取位置不是每次都有效，它的优先级默认在 Offset Store后面 ， 比如在 DefaultMQPushConsumer 的 BROADCASTING 方式下 ，默认是从 Broker 里读取某个 Topic 对应 ConsumerGroup 的 Offset， 当读取不到 Offset 的时候， ConsumeFromWhere 的设置才生效 。

In most cases, this setting is effective when the Consumer Group is first started. If the Consumer is stopped after normal operation and then restarted, it will start to consume from the last Offset, and the setting of ConsumeFromWhere will be effective.

Message offset Offset

https://www.cnblogs.com/jwen1994/p/12369913.html

/**
 * Offset store interface
 */
public interface OffsetStore {
    
    
    /**
     * Load
     */
    // LocalFileOffsetStore  实现 MixAll.file2String(this.storePath) || this.readLocalOffsetBak()
    // RemoteBrokerOffsetStore 实现为空方法
    void load() throws MQClientException;

    /**
     * Update the offset,store it in memory
     */
    void updateOffset(final MessageQueue mq, final long offset, final boolean increaseOnly);

    /**
     * Get offset from local storage
     * @return The fetched offset
     */
    long readOffset(final MessageQueue mq, final ReadOffsetType type);

    /**
     * Persist all offsets,may be in local storage or remote name server
     */
    void persistAll(final Set<MessageQueue> mqs);

    /**
     * Persist the offset,may be in local storage or remote name server
     */
    void persist(final MessageQueue mq);

    /**
     * Remove offset
     */
    void removeOffset(MessageQueue mq);

    /**
     * @return The cloned offset table of given topic
     */
    Map<MessageQueue, Long> cloneOffsetTable(String topic);

    /**
     * @param mq
     * @param offset
     * @param isOneway
     */
    // LocalFileOffsetStore 的实现为空方法
    // RemoteBrokerOffsetStore 同步offset到Broker
    /**
     * Update the Consumer Offset synchronously, once the Master is off, updated to Slave, here need to be optimized.
     */
    void updateConsumeOffsetToBroker(MessageQueue mq, long offset, boolean isOneway) throws RemotingException,
        MQBrokerException, InterruptedException, MQClientException;
}

The BROADCASTING mode of DefaultMQPushConsumer, each Consumer does not interfere with each other, use LocalFileOffsetStore to store the Offset locally

// package org.apache.rocketmq.client.consumer.store;
public class LocalFileOffsetStore implements OffsetStore {
    
    
    public final static String LOCAL_OFFSET_STORE_DIR = System.getProperty(
        "rocketmq.client.localOffsetStoreDir",
        System.getProperty("user.home") + File.separator + ".rocketmq_offsets");
    private final static InternalLogger log = ClientLogger.getLog();
    private final MQClientInstance mQClientFactory;
    private final String groupName;
    private final String storePath;
    private ConcurrentMap<MessageQueue, AtomicLong> offsetTable =
        new ConcurrentHashMap<MessageQueue, AtomicLong>();
    // ...

The CLUSTERING mode of DefaultMQPushConsumer, the Offset value is stored and controlled by the Broker end, using RemoteBrokerOffsetStore

// package org.apache.rocketmq.client.consumer.store;
public class RemoteBrokerOffsetStore implements OffsetStore {
    
    
    private final static InternalLogger log = ClientLogger.getLog();
    private final MQClientInstance mQClientFactory;
    private final String groupName;
    private ConcurrentMap<MessageQueue, AtomicLong> offsetTable =
        new ConcurrentHashMap<MessageQueue, AtomicLong>();

    //...
}

It is recommended to use pushConsumer. RocketMQ automatically maintains the OffsetStore. If you use another pullConsumer, you need to maintain the OffsetStore yourself.

// this.defaultLitePullConsumerImpl.start()
private void initOffsetStore() throws MQClientException {
    
    
        if (this.defaultLitePullConsumer.getOffsetStore() != null) {
    
    
            this.offsetStore = this.defaultLitePullConsumer.getOffsetStore();
        } else {
    
    
            switch (this.defaultLitePullConsumer.getMessageModel()) {
    
    
                case BROADCASTING:
                    this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultLitePullConsumer.getConsumerGroup());
                    break;
                case CLUSTERING:
                    this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultLitePullConsumer.getConsumerGroup());
                    break;
                default:
                    break;
            }
            this.defaultLitePullConsumer.setOffsetStore(this.offsetStore);
        }
        this.offsetStore.load();
    }

The message queue is an infinite array. When a message comes in, the subscript will increase by 1. The subscript is offset. The position of the message in a MessageQueue can be located through the value of the offset, or instruct the Consumer to start from the message. Start the backward processing.

The maxOffset in the message queue represents the maximum offset of the message, maxOffset is not the offset of the latest message, but the offset+1 of the latest message, and minOffset is the minimum offset that exists.

fileReserveTime=48 After the default message is stored for 48 hours, the consumption will be physically deleted from the disk, and the minOffset of the message queue will increase accordingly. So those messages smaller than minOffset are no longer on the broker and cannot be consumed