Call the MQClientInstance.persistAllConsumerOffset() method every 5 seconds to synchronize the consumption progress to the Broker. Traverse MQClientInstance.consumerTable: ConcurrentHashMap<String/*group */, MQConsumerInner> variables. For the PushConsumer side and the PullConsumer side, the processing logic is the same. Take DefaultMQPushConsumerImpl as an example, call the DefaultMQPushConsumerImpl.persistConsumerOffset() method.
1. Obtain the processQueueTable: ConcurrentHashMap<MessageQueue, ProcessQueue> variable value of the DefaultMQPushConsumerImpl.rebalanceImpl variable, and take the set of key values of the variable, that is, the MessageQueue set; call the OffsetStore.persistAll(Set<MessageQueue> mqs) method with the set as a parameter;
2. If the message mode is BROADCASTING, that is, the DefaultMQPushConsumerImpl.offsetStore variable is initialized to the LocalFileOffsetStore object, and the LocalFileOffsetStore.persistAll(Set<MessageQueue> mqs) method is called here, and the LocalFileOffsetStore.offsetTable:ConcurrentHashMap<MessageQueue,AtomicLong is traversed in this method > variable to persist the consumption progress of the MessageQueue objects contained in the MessageQueue collection of step 1 to the consumerOffset.json physical file;
3. If the message mode is cluster mode, that is, the DefaultMQPushConsumerImpl.offsetStore variable is initialized to the RemoteBrokerOffsetStore object, and the RemoteBrokerOffsetStore.persistAll(Set<MessageQueue> mqs) method is called here, and the RemoteBrokerOffsetStore.offsetTable:ConcurrentHashMap <MessageQueue,AtomicLong> variable is traversed in this method. ; For the MessageQueue objects included in the MessageQueue collection of step 1, call the updateConsumeOffsetToBroker(MessageQueuemq, long offset) method to send the consumption progress information of the UPDATE_CONSUMER_OFFSET request code to the Broker;
Consumption progress (offset)
Consumption progress means that when a consumer in a consumer group consumes a message in a queue, the equeue knows where the current consumption is by recording the consumption position (offset). So that the consumer can continue to consume from this location after restarting. For example, a topic has 4 queues, and a consumer group has 4 consumers, each consumer is assigned to a queue, and then each consumer consumes messages in its own queue. equeue will record the consumption progress of each consumer on its queue separately, so as to ensure that each consumer knows where to start consumption next time after restarting. In fact, maybe after the next restart, the queue will not be consumed by the consumer, but by other consumers in the group. It doesn't matter, because we have already recorded the consumption position of the queue. So it can be seen that the consumption position has nothing to do with the consumer. The consumption position is completely an attribute of the queue, which is used to record where the current consumption is. Another important point is that a topic can be subscribed by consumers in multiple consumer groups. Even if consumers in different consumer groups consume the same queue under the same topic, the consumption progress is stored separately. That is to say, the consumption of consumers in different consumer groups is completely isolated and not affected by each other. Another point is that for cluster consumption and broadcast consumption, the consumption progress is different in the place where the consumption progress is persisted. The consumption progress of cluster consumption is placed on the broker, that is, the message queue server, while the consumption progress of broadcast consumption is stored in the consumer. on the local disk. The reason for this design is that, for cluster consumption, the consumer due to a queue may be replaced, because the consumer The number of consumers under the group may increase or decrease, and then recalculate the queues that each consumer should consume. Is this understandable? So, when the consumer of a queue changes, how does the new consumer know where to start consuming the queue? If the consumption progress of this queue is stored on the previous consumer server, it will be difficult to get the consumption progress, because it is possible that the server has been hung up or taken off the shelf. And because the broker is always serving all consumers, in the case of cluster consumption, the consumption position of the queue of the subscribed topic is stored on the broker, and the storage is isolated according to different consumer groups to Ensure that the consumption progress of consumers under different consumer groups has complementary effects. Then, for broadcast consumption, since there will not be a situation where the consumer of a queue will change, there is no need for the broker to save the consumption position, so it is saved on the consumer's own server.
http://blog.csdn.net/meilong_whpu/article/details/77065587
https://www.cc362.com/content/Npz3glwzPQ.html