Consumer startup process
After the Producer sends the message to the Broker, the Broker makes the persistence. Then you can create a consumer to consume the message. Today we analyze how to start a consumer. producer code, the principle is similar to starting the producer.
Initialization of DefaultMQPushConsumer class
public DefaultMQPushConsumer(final String namespace, final String consumerGroup, RPCHook rpcHook,
AllocateMessageQueueStrategy allocateMessageQueueStrategy) {
// 消费者组
this.consumerGroup = consumerGroup;
// 命名空间
this.namespace = namespace;
// 分配策略
this.allocateMessageQueueStrategy = allocateMessageQueueStrategy;
// 创建DefaultMQPushConsumerImpl实例
defaultMQPushConsumerImpl = new DefaultMQPushConsumerImpl(this, rpcHook);
}
DefaultMQPushConsumerImpl, as the specific implementation of DefaultMQPushConsumer, is basically similar to the Producer analyzed before. This method initializes and registers the hook function, registers the messageListener message listener, and assigns some parameters such as namesevAddr.
subscribesubscribe
When the Producer is created, you can specify the Topic to send, and the consumer only needs to subscribe to the relevant topics. It can also filter messages based on tags and Sql.
/**
* 订阅topic支持消息过滤表达式
* @param topic 订阅的topic
* @param subExpression 只支持或（“||”）操作，如果是null或者“*”则表示订阅全部
*/
public void subscribe(String topic, String subExpression) throws MQClientException {
this.defaultMQPushConsumerImpl.subscribe(withNamespace(topic), subExpression);
}
/**
*
*/
public void subscribe(String topic, String subExpression) throws MQClientException {
try {
// 解析订阅表达式生成SubscriptionData
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(topic, subExpression);
// 将topic与SubscriptionData的关系维护到RebalanceImpl内部的subscriptionInner
this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
if (this.mQClientFactory != null) {
// 如果mQClientFactory不为null就发送心跳给所有broker
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
}
} catch (Exception e) {
throw new MQClientException("subscription exception", e);
}
}
Start method of DefaultMQPushConsumer
正式开始启动消费者，调用DefaultMQPushConsumer的start启动。
public void start() throws MQClientException {
// 根据namespace和consumerGroup设置消费者组
setConsumerGroup(NamespaceUtil.wrapNamespace(this.getNamespace(), this.consumerGroup));
// 启动具体
this.defaultMQPushConsumerImpl.start();
// 消费者轨迹追踪服务
if (null != traceDispatcher) {
try {
traceDispatcher.start(this.getNamesrvAddr(), this.getAccessChannel());
} catch (MQClientException e) {
log.warn("trace dispatcher start failed ", e);
}
}
}
The start logic of DefaultMQPushConsumer is mainly to set up consumer groups based on namespace and ConsumerGroup. The main implementation is still in its short-sighted implementation class DefaultMQPushConsumerImpl (facade mode).
public synchronized void start() throws MQClientException {
// 根据服务状态实现不同的逻辑，只有当为CREATE_JUST才会正常启动，其他类型的都报错
switch (this.serviceState) {
case CREATE_JUST:
log.info("the consumer [{}] start beginning. messageModel={}, isUnitMode={}", this.defaultMQPushConsumer.getConsumerGroup(),
this.defaultMQPushConsumer.getMessageModel(), this.defaultMQPushConsumer.isUnitMode());
this.serviceState = ServiceState.START_FAILED;
// 检查配置
this.checkConfig();
// 拷贝订阅关系，为集群模式的消费者配置其对应的重试队列
this.copySubscription();
if (this.defaultMQPushConsumer.getMessageModel() == MessageModel.CLUSTERING) {
this.defaultMQPushConsumer.changeInstanceNameToPID();
}
this.mQClientFactory = MQClientManager.getInstance().getOrCreateMQClientInstance(this.defaultMQPushConsumer, this.rpcHook);
// 消费者组
this.rebalanceImpl.setConsumerGroup(this.defaultMQPushConsumer.getConsumerGroup());
// 消费者模式（广播/集群）
this.rebalanceImpl.setMessageModel(this.defaultMQPushConsumer.getMessageModel());
//Consumer负载均衡策略（集群模式才存在负载均衡）
this.rebalanceImpl.setAllocateMessageQueueStrategy(this.defaultMQPushConsumer.getAllocateMessageQueueStrategy());
this.rebalanceImpl.setmQClientFactory(this.mQClientFactory);
this.pullAPIWrapper = new PullAPIWrapper(
mQClientFactory,
this.defaultMQPushConsumer.getConsumerGroup(), isUnitMode());
this.pullAPIWrapper.registerFilterMessageHook(filterMessageHookList);
if (this.defaultMQPushConsumer.getOffsetStore() != null) {
this.offsetStore = this.defaultMQPushConsumer.getOffsetStore();
} else {
switch (this.defaultMQPushConsumer.getMessageModel()) {
// 广播模式（本地的Offset文件）
case BROADCASTING:
this.offsetStore = new LocalFileOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
// 集群模式 （Broker里面的offset）
case CLUSTERING:
this.offsetStore = new RemoteBrokerOffsetStore(this.mQClientFactory, this.defaultMQPushConsumer.getConsumerGroup());
break;
default:
break;
}
this.defaultMQPushConsumer.setOffsetStore(this.offsetStore);
}
// 加载消息偏移量
this.offsetStore.load();
//Consumer中自行指定的回调函数。
if (this.getMessageListenerInner() instanceof MessageListenerOrderly) {
this.consumeOrderly = true;
// MessageListenerOrderly服务
this.consumeMessageService =
new ConsumeMessageOrderlyService(this, (MessageListenerOrderly) this.getMessageListenerInner());
} else if (this.getMessageListenerInner() instanceof MessageListenerConcurrently) {
this.consumeOrderly = false;
// MessageListenerConcurrently服务
this.consumeMessageService =
new ConsumeMessageConcurrentlyService(this, (MessageListenerConcurrently) this.getMessageListenerInner());
}
// 启动消息消费服务
this.consumeMessageService.start();
// 注册
boolean registerOK = mQClientFactory.registerConsumer(this.defaultMQPushConsumer.getConsumerGroup(), this);
if (!registerOK) {
this.serviceState = ServiceState.CREATE_JUST;
this.consumeMessageService.shutdown(defaultMQPushConsumer.getAwaitTerminationMillisWhenShutdown());
throw new MQClientException("The consumer group[" + this.defaultMQPushConsumer.getConsumerGroup()
+ "] has been created before, specify another name please." + FAQUrl.suggestTodo(FAQUrl.GROUP_NAME_DUPLICATE_URL),
null);
}
// 启动CreateMQClientInstance客户端通信实例
// 包括netty服务，各种定时任务，拉取消息服务，rebalanceService服务。
mQClientFactory.start();
log.info("the consumer [{}] start OK.", this.defaultMQPushConsumer.getConsumerGroup());
this.serviceState = ServiceState.RUNNING;
break;
case RUNNING:
case START_FAILED:
case SHUTDOWN_ALREADY:
throw new MQClientException("The PushConsumer service state not OK, maybe started once, "
+ this.serviceState
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_SERVICE_NOT_OK),
null);
default:
break;
}
// 向NameServer拉取并更新当前消费者订阅的topic路由信息
this.updateTopicSubscribeInfoWhenSubscriptionChanged();
// 随机选择broker检查tags
this.mQClientFactory.checkClientInBroker();
// 发送心跳给所有Broker
this.mQClientFactory.sendHeartbeatToAllBrokerWithLock();
// 唤醒负载均衡服务rebalanceService，主动进行一次MessageQueue的重平衡。
this.mQClientFactory.rebalanceImmediately();
}
Related logic implementation
Check configuration
Call the checkConfig() method to verify a series of parameters.
private void checkConfig() throws MQClientException {
// 检查消费组（不能为空，长度不能大于255，只能包含数字和字母）
Validators.checkGroup(this.defaultMQPushConsumer.getConsumerGroup());
if (null == this.defaultMQPushConsumer.getConsumerGroup()) {
throw new MQClientException(
"consumerGroup is null"
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_PARAMETER_CHECK_URL),
null);
}
// 检查是否是默认值名，是则报错
if (this.defaultMQPushConsumer.getConsumerGroup().equals(MixAll.DEFAULT_CONSUMER_GROUP)) {
throw new MQClientException(
"consumerGroup can not equal "
+ MixAll.DEFAULT_CONSUMER_GROUP
+ ", please specify another one."
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_PARAMETER_CHECK_URL),
null);
}
// 检查消费模式（广播或集群）
if (null == this.defaultMQPushConsumer.getMessageModel()) {
throw new MQClientException(
"messageModel is null"
+ FAQUrl.suggestTodo(FAQUrl.CLIENT_PARAMETER_CHECK_URL),
null);
}
// ....
}
First check that the consumer group cannot be empty, the length cannot be greater than 255, and it can only contain numbers and letters. At the same time, the name of the consumer group cannot be the same as the name of the default consumer group (DEFAULT_CONSUMER), otherwise an exception will be thrown.
Checking the consumption mode can only exist in cluster or broadcast mode, otherwise an exception will be thrown.
Check whether ConsumeFromWhere is empty. ConsumeFromWhere represents the consumption strategy. A simple understanding is where the message starts to be consumed. The default is CONSUME_FROM_LAST_OFFSET (the last one in the queue starts consuming). You can also specify CONSUME_FROM_FIRST_OFFSET (the first one in the queue starts consuming) and CONSUME_FROM_TIMESTAMP (specify the timestamp). ).
Verify the start consumption time, load strategy allocateMessageQueueStrategy in cluster mode, subscription relationship, whether to register message monitoring, number of local threads, time interval for pulling messages, the number of messages pulled in a single time, and the number of messages consumed in a single time.
Copy subscription relationship
is to configure the corresponding retry topic retryTopic for consumers in cluster mode and bind it to implement message retry, while broadcast messages will not any news.
private void copySubscription() throws MQClientException {
try {
// 订阅消息进行拷贝存入rebalanceImpl中
Map<String, String> sub = this.defaultMQPushConsumer.getSubscription();
if (sub != null) {
for (final Map.Entry<String, String> entry : sub.entrySet()) {
final String topic = entry.getKey();
final String subString = entry.getValue();
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(topic, subString);
this.rebalanceImpl.getSubscriptionInner().put(topic, subscriptionData);
}
}
if (null == this.messageListenerInner) {
this.messageListenerInner = this.defaultMQPushConsumer.getMessageListener();
}
// 消息类型，如果是集群消息进行绑定进行失败重试，如果是广播消息不做任何操作不重试直接丢弃
switch (this.defaultMQPushConsumer.getMessageModel()) {
case BROADCASTING:
break;
case CLUSTERING:
final String retryTopic = MixAll.getRetryTopic(this.defaultMQPushConsumer.getConsumerGroup());
SubscriptionData subscriptionData = FilterAPI.buildSubscriptionData(retryTopic, SubscriptionData.SUB_ALL);
this.rebalanceImpl.getSubscriptionInner().put(retryTopic, subscriptionData);
break;
default:
break;
}
} catch (Exception e) {
throw new MQClientException("subscription exception", e);
}
}
Note: Cluster messages automatically subscribe to the consumer's retry topic. When a message fails, it will be retried to prevent message loss. Broadcast messages will not perform any operations. When a message fails, it will be discarded directly. The message will be lost.
Other operations
Call the getOrCreateMQClientInstance method to obtain the CreateMQClientInstance instance, set the relevant attributes of the load balancing service rebalanceImpl, and create the core pull message class PullAPIWrapper.
According to different message modes, set different locations of the Offset file, that is, the message offset. In broadcast mode, the Offset file is saved locally. In cluster mode, the Offset file is saved on the remote Broker side, and load is called at the same time to load the offset.
Specify the specified MessageListener to create different ConsumeMessageService message services. There are two classes: MessageListenerOrderly (sequential consumption) and MessageListenerConcurrently (concurrent consumption).
Start the consumer service this.consumeMessageService.start(), register the service, start the CreateMQClientInstance client communication instance, initialize the netty service, various scheduled tasks, pull message services, rebalabceService services, etc., similar to the previous Producer.