Article directory
Consumer starts the process
DefaultMQPushConsumerImpl class is the core class
consumer model
cluster consumption
Each Consumer instance in a Consumer Group shares the consumption of messages, that is, a message will only be delivered to one instance under a Consumer Group.
In fact, each Consumer shares the Message Queue cancellation fee equally. For example, if a Topic has 3 Queues, and one of the Consumer Groups has 3 instances (maybe 3 processes, or 3 machines), then each instance only consumes 1 Queue message.
When the Producer sends a message, it polls all Queues, so the messages will be evenly scattered on different Queues. It can be considered that the messages on the Queues are average. Then the instances will consume messages evenly.
In this mode, the storage of consumption progress (Consumer Offset) will be persisted to the Broker .
radio consumption
The message will be delivered to each Consumer instance under a Consumer Group. Even if these Consumers belong to the same Consumer Group, the message will be consumed once by each Consumer in the Consumer Group.
In this mode, the consumption progress (Consumer Offset) will be stored and persisted locally to the instance .
Consumer load balancing
cluster mode
In cluster consumption mode, each message only needs to be delivered to one instance under the Consumer Group that subscribes to this topic. RocketMQ uses active pulling to pull and consume messages. When pulling, you need to clearly specify which message queue to pull.
Whenever the number of instances changes, load balancing of all instances will be triggered. At this time, the queue will be evenly distributed to each instance according to the number of queues and the number of instances.
The default allocation algorithm is AllocateMessageQueueAveragely, and there is another average algorithm is AllocateMessageQueueAveragelyByCircle, which also allocates each queue equally, but in the form of a circular distribution of queues.
As shown below:
It should be noted that in cluster mode, queue only allows one instance to be allocated. This is because if multiple instances consume messages from a queue at the same time, the consumer actively controls which messages are pulled, which will result in the same message. It is consumed multiple times under different instances, so algorithmically one queue is only assigned to one consumer instance, and one consumer instance can be assigned to different queues at the same time.
By adding consumer instances to share the consumption of the queue, it can play a role in horizontally expanding consumption capabilities. When an instance goes offline, load balancing will be triggered again. At this time, the queue originally allocated will be allocated to other instances to continue consumption.
However, if the number of consumer instances is greater than the total number of message queues, the extra consumer instances will not be allocated to the queue, and messages will not be consumed, and the load will not be shared. Therefore, it is necessary to control the total number of queues to be greater than or equal to the number of consumers.
broadcast mode
Since broadcast mode requires that a message be delivered to all consumer instances under a consumer group, there is no such thing as shared consumption of messages.
In terms of implementation, one of the differences is that when consumers allocate queues, all consumers are assigned to all queues.
Concurrent consumption process
Generally, we use callback functions when consuming. The most commonly used method is concurrent consumption. The consumer client code is as follows:
When consuming RocketMQ, the overall process is as follows:
Get topic configuration information
After the consumer starts, the first step is to obtain Topic related information from the NameServer. This step is designed to interact between components, and RocketMQ uses function numbers to design it.
Finally the call is completed in the MQClientAPIImpl class
Get Group's ConsumerList
Before consuming messages, you need to obtain relevant information that the current group has consumed: ConsumerList
If it is a broadcast consumption mode, there is no need to obtain the consumption progress from the server (the broadcast consumption mode stores the progress locally on the consumer side)
In the broadcast consumption mode, you need to obtain consumption progress related information from the server, as follows:
Get the consumption Offset of the Queue
After allocating the Queue corresponding to the consumer, if it is a cluster mode, you need to obtain the consumption Offset of the Queue corresponding to the consumer, so as to facilitate subsequent retrieval of unconsumed messages.
Enter the RebalancePushImpl class
Pull messages from Queue
Finally enter the pullMessage method of the DefaultMQPushConsumerImpl class
Update the consumption Offset of the Queue
Because the push mode of RocketMQ is implemented based on the pull mode, because messages are pulled in batches, it is not possible to pull a batch and submit the offset once, so a scheduled task is used here to update the offset.
Sequential consumption process
Sequential consumption code: 
The process of sequential consumption is similar to the process of concurrent consumption. The only difference is the use of a lock mechanism to ensure that a queue can only be consumed by one consumer at the same time, thus ensuring the sequentiality of consumption.
ConsumeMessageOrderlyService类
There is a scheduled task here, which runs every 20 seconds (to renew the lock periodically, the lock validity period is 60S)
Consumption problems
Consumption stuck
In the consumption process, especially for sequential messages, there will be a stuck phenomenon. Since the sequential messages need to be locked in the Broker, if one of the consumers hangs up, it will take 60 seconds for the Broker layer to maintain the lock before it can be released. , so during this period of time, consumers cannot consume and are waiting for the lock.
In addition, if the Broker layer is also down, if it is a master-slave structure, the Master node is used to obtain the lock. If the Master node is down, the Slave consumption is used, but there is no lock on the slave node, so if the sequential message occurs like this In some cases, there may also be jamming.
It takes a long time to consume after startup
During concurrent consumption, when we start a lot of consumers, maintain a lot of topics, or there are a lot of queues, we can see that the interaction of the consumption process requires starting multiple threads and doing a lot of work. There are so many things to do, so it feels like it takes a long time to start before you can consume them.