Because I recently encountered the problem of spring-kafka consumption thread interrupting consumption, I read the startup process of its consumption thread to summarize.
Lifecycle和SmartLifecycle
If you want to know its loading process, you must first understand these two interfaces:
Lifecycle is the most basic life cycle interface in Spring, which defines the methods for starting and stopping the container.
SmartLifecycle is an extension interface to Lifecycle, which adds the following functions compared to Lifecycle:
- You can call back the start() of the SmartLifecycle interface without calling the start() method explicitly on the container
- If there are multiple SmartLifecycle instances in the container, it is convenient to control the calling sequence.
spring-kafka version
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>1.3.5.RELEASE</version>
</dependency>
Startup process analysis
After spring loads all beans (instantiation + initialization), refresh the container org.springframework.context.support.AbstractApplicationContext#finishRefresh
protected void finishRefresh() {
// Initialize lifecycle processor for this context.
initLifecycleProcessor();
// Propagate refresh to lifecycle processor first.
getLifecycleProcessor().onRefresh();
// Publish the final event.
publishEvent(new ContextRefreshedEvent(this));
// Participate in LiveBeansView MBean, if active.
LiveBeansView.registerApplicationContext(this);
}
The first is the default Lifecycle processor DefaultLifecycleProcessor
The second step: getLifecycleProcessor().onRefresh();
All beans that implement Lifecycle are started, that is, call the following method
org.springframework.context.support.DefaultLifecycleProcessor#startBeans
private void startBeans(boolean autoStartupOnly) {
Map<String, Lifecycle> lifecycleBeans = getLifecycleBeans();
Map<Integer, LifecycleGroup> phases = new HashMap<Integer, LifecycleGroup>();
for (Map.Entry<String, ? extends Lifecycle> entry : lifecycleBeans.entrySet()) {
Lifecycle bean = entry.getValue();
if (!autoStartupOnly || (bean instanceof SmartLifecycle && ((SmartLifecycle) bean).isAutoStartup())) {
int phase = getPhase(bean);
LifecycleGroup group = phases.get(phase);
if (group == null) {
group = new LifecycleGroup(phase, this.timeoutPerShutdownPhase, lifecycleBeans, autoStartupOnly);
phases.put(phase, group);
}
group.add(entry.getKey(), bean);
}
}
if (!phases.isEmpty()) {
List<Integer> keys = new ArrayList<Integer>(phases.keySet());
Collections.sort(keys);
for (Integer key : keys) {
phases.get(key).start();
}
}
}
Sort by phase, the smaller the value, the higher the front, and finally traverse the start method:
public void start() {
if (this.members.isEmpty()) {
return;
}
if (logger.isInfoEnabled()) {
logger.info("Starting beans in phase " + this.phase);
}
Collections.sort(this.members);
for (LifecycleGroupMember member : this.members) {
if (this.lifecycleBeans.containsKey(member.name)) {
doStart(this.lifecycleBeans, member.name, this.autoStartupOnly);
}
}
}
will eventually enter
private void doStart(Map<String, ? extends Lifecycle> lifecycleBeans, String beanName, boolean autoStartupOnly) {
Lifecycle bean = lifecycleBeans.remove(beanName);
if (bean != null && bean != this) {
String[] dependenciesForBean = this.beanFactory.getDependenciesForBean(beanName);
for (String dependency : dependenciesForBean) {
doStart(lifecycleBeans, dependency, autoStartupOnly);
}
if (!bean.isRunning() &&
(!autoStartupOnly || !(bean instanceof SmartLifecycle) || ((SmartLifecycle) bean).isAutoStartup())) {
if (logger.isDebugEnabled()) {
logger.debug("Starting bean '" + beanName + "' of type [" + bean.getClass() + "]");
}
try {
bean.start();
}
catch (Throwable ex) {
throw new ApplicationContextException("Failed to start bean '" + beanName + "'", ex);
}
if (logger.isDebugEnabled()) {
logger.debug("Successfully started bean '" + beanName + "'");
}
}
}
}
Just look at the bean.start() sentence, where a key class in spring-kafka: KafkaListenerEndpointRegistry implements SmartLifecycle, so it will go into it, okay, let's go into its start method:
@Override
public void start() {
for (MessageListenerContainer listenerContainer : getListenerContainers()) {
startIfNecessary(listenerContainer);
}
}
What is getListenerContainers()? I printed it out here.
It is the ConcurrentMessageListenerContainer object, like the class that starts the consumer (disclose first, this corresponds to the number of times **@KafkaListener** annotations are used in our code, that is to say If a method uses this annotation, it will generate such an object for you), when did it come in, with this question, first look at where it is assigned, and find the code location assigned to it:
public void registerListenerContainer(KafkaListenerEndpoint endpoint, KafkaListenerContainerFactory<?> factory,
boolean startImmediately) {
Assert.notNull(endpoint, "Endpoint must not be null");
Assert.notNull(factory, "Factory must not be null");
String id = endpoint.getId();
Assert.hasText(id, "Endpoint id must not be empty");
synchronized (this.listenerContainers) {
Assert.state(!this.listenerContainers.containsKey(id),
"Another endpoint is already registered with id '" + id + "'");
MessageListenerContainer container = createListenerContainer(endpoint, factory);
this.listenerContainers.put(id, container);
// 省略部分代码
Ok, then where did you adjust the registerListenerContainer method? Continue to deduce it and find that in the class KafkaListenerEndpointRegistrar, both the registerEndpoint method and the registerAllEndpoints method have been adjusted to it, so which one is adjusted? Let's look at the first registerEndpoint method first
public void registerEndpoint(KafkaListenerEndpoint endpoint, KafkaListenerContainerFactory<?> factory) {
Assert.notNull(endpoint, "Endpoint must be set");
Assert.hasText(endpoint.getId(), "Endpoint id must be set");
// Factory may be null, we defer the resolution right before actually creating the container
KafkaListenerEndpointDescriptor descriptor = new KafkaListenerEndpointDescriptor(endpoint, factory);
synchronized (this.endpointDescriptors) {
if (this.startImmediately) {
// Register and start immediately
this.endpointRegistry.registerListenerContainer(descriptor.endpoint,
resolveContainerFactory(descriptor), true);
}
else {
this.endpointDescriptors.add(descriptor);
}
}
}
startImmediately is false at the beginning, so it should not be it, then the registerAllEndpoints method is left. We see that registerAllEndpoints needs to get this.endpointDescriptors collection, so it must be written in advance,
And KafkaListenerEndpointRegistrar#registerEndpoint just did this, because if (this.startImmediately) is not satisfied, so I wrote this.endpointDescriptors.add(descriptor);
when calling the registerAllEndpoints method, I can get it.
So who tuned
KafkaListenerEndpointRegistrar#registerEndpoint?
It is found that there is a class KafkaListenerAnnotationBeanPostProcessor. You can know that it is a BeanPostProcessor by looking at the name, that is, the Bean post-processor. It is called during the bean initialization phase. Let's go in and see the methods in it.
public Object postProcessAfterInitialization(final Object bean, final String beanName) throws BeansException {
if (!this.nonAnnotatedClasses.contains(bean.getClass())) {
Class<?> targetClass = AopUtils.getTargetClass(bean);
Collection<KafkaListener> classLevelListeners = findListenerAnnotations(targetClass);
final boolean hasClassLevelListeners = classLevelListeners.size() > 0;
final List<Method> multiMethods = new ArrayList<Method>();
Map<Method, Set<KafkaListener>> annotatedMethods = MethodIntrospector.selectMethods(targetClass,
new MethodIntrospector.MetadataLookup<Set<KafkaListener>>() {
@Override
public Set<KafkaListener> inspect(Method method) {
Set<KafkaListener> listenerMethods = findListenerAnnotations(method);
return (!listenerMethods.isEmpty() ? listenerMethods : null);
}
});
if (hasClassLevelListeners) {
Set<Method> methodsWithHandler = MethodIntrospector.selectMethods(targetClass,
new MethodFilter() {
@Override
public boolean matches(Method method) {
return AnnotationUtils.findAnnotation(method, KafkaHandler.class) != null;
}
});
multiMethods.addAll(methodsWithHandler);
}
if (annotatedMethods.isEmpty()) {
this.nonAnnotatedClasses.add(bean.getClass());
if (this.logger.isTraceEnabled()) {
this.logger.trace("No @KafkaListener annotations found on bean type: " + bean.getClass());
}
}
else {
// Non-empty set of methods
for (Map.Entry<Method, Set<KafkaListener>> entry : annotatedMethods.entrySet()) {
Method method = entry.getKey();
for (KafkaListener listener : entry.getValue()) {
processKafkaListener(listener, method, bean, beanName);
}
}
if (this.logger.isDebugEnabled()) {
this.logger.debug(annotatedMethods.size() + " @KafkaListener methods processed on bean '"
+ beanName + "': " + annotatedMethods);
}
}
if (hasClassLevelListeners) {
processMultiMethodListeners(classLevelListeners, multiMethods, bean, beanName);
}
}
return bean;
}
It's a bit long here. I saw the familiar annotation @KafkaListener here. Yes, we usually add the @KafkaListener annotation to the method to play a role. This logic is to analyze all the methods of the current bean with this annotation, and then Enter this sentence:
processKafkaListener(listener, method, bean, beanName);
protected void processListener(MethodKafkaListenerEndpoint<?, ?> endpoint, KafkaListener kafkaListener, Object bean,
Object adminTarget, String beanName) {
endpoint.setBean(bean);
endpoint.setMessageHandlerMethodFactory(this.messageHandlerMethodFactory);
endpoint.setId(getEndpointId(kafkaListener));
endpoint.setGroupId(getEndpointGroupId(kafkaListener, endpoint.getId()));
endpoint.setTopicPartitions(resolveTopicPartitions(kafkaListener));
endpoint.setTopics(resolveTopics(kafkaListener));
endpoint.setTopicPattern(resolvePattern(kafkaListener));
String group = kafkaListener.containerGroup();
// 省略部分代码
this.registrar.registerEndpoint(endpoint, factory);
}
First analyze the properties of the annotation method, generate the object, and finally adjust to the question left above, who adjusted the this.registrar.registerEndpoint(endpoint, factory) method, well, the whole link is connected here.
We continue the analysis at the very beginning, that is: KafkaListenerEndpointRegistry#start
@Override
public void start() {
for (MessageListenerContainer listenerContainer : getListenerContainers()) {
startIfNecessary(listenerContainer);
}
}
After the above round of analysis, I know that listenerContainer is ConcurrentMessageListenerContainer, so it will eventually enter start() inside it:
@Override
public final void start() {
synchronized (this.lifecycleMonitor) {
if (!isRunning()) {
Assert.isTrue(
this.containerProperties.getMessageListener() instanceof KafkaDataListener,
"A " + GenericMessageListener.class.getName() + " implementation must be provided");
doStart();
}
}
}
doStart() finds that it is an abstract method, ConcurrentMessageListenerContainer implements it, go in and see
@Override
protected void doStart() {
if (!isRunning()) {
ContainerProperties containerProperties = getContainerProperties();
TopicPartitionInitialOffset[] topicPartitions = containerProperties.getTopicPartitions();
if (topicPartitions != null
&& this.concurrency > topicPartitions.length) {
this.logger.warn("When specific partitions are provided, the concurrency must be less than or "
+ "equal to the number of partitions; reduced from " + this.concurrency + " to "
+ topicPartitions.length);
this.concurrency = topicPartitions.length;
}
setRunning(true);
for (int i = 0; i < this.concurrency; i++) {
KafkaMessageListenerContainer<K, V> container;
if (topicPartitions == null) {
container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties);
}
else {
container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties,
partitionSubset(containerProperties, i));
}
String beanName = getBeanName();
container.setBeanName((beanName != null ? beanName : "consumer") + "-" + i);
if (getApplicationEventPublisher() != null) {
container.setApplicationEventPublisher(getApplicationEventPublisher());
}
container.setClientIdSuffix("-" + i);
container.start();
this.containers.add(container);
}
}
}
It's very simple. It is to create a KafkaMessageListenerContainer object according to concurrency. This is the same as ConcurrentMessageListenerContainer. It also implements the MessageListenerContainer interface.
Seeing that there is no container.start(), it calls the start method again. This time it is the start of KafkaMessageListenerContainer. Follow up and eventually Enter the doStart method, as follows
@Override
protected void doStart() {
if (isRunning()) {
return;
}
ContainerProperties containerProperties = getContainerProperties();
if (!this.consumerFactory.isAutoCommit()) {
AckMode ackMode = containerProperties.getAckMode();
if (ackMode.equals(AckMode.COUNT) || ackMode.equals(AckMode.COUNT_TIME)) {
Assert.state(containerProperties.getAckCount() > 0, "'ackCount' must be > 0");
}
if ((ackMode.equals(AckMode.TIME) || ackMode.equals(AckMode.COUNT_TIME))
&& containerProperties.getAckTime() == 0) {
containerProperties.setAckTime(5000);
}
}
Object messageListener = containerProperties.getMessageListener();
Assert.state(messageListener != null, "A MessageListener is required");
if (messageListener instanceof GenericAcknowledgingMessageListener) {
this.acknowledgingMessageListener = (GenericAcknowledgingMessageListener<?>) messageListener;
}
else if (messageListener instanceof GenericMessageListener) {
this.listener = (GenericMessageListener<?>) messageListener;
}
else {
throw new IllegalStateException("messageListener must be 'MessageListener' "
+ "or 'AcknowledgingMessageListener', not " + messageListener.getClass().getName());
}
if (containerProperties.getConsumerTaskExecutor() == null) {
SimpleAsyncTaskExecutor consumerExecutor = new SimpleAsyncTaskExecutor(
(getBeanName() == null ? "" : getBeanName()) + "-C-");
containerProperties.setConsumerTaskExecutor(consumerExecutor);
}
this.listenerConsumer = new ListenerConsumer(this.listener, this.acknowledgingMessageListener);
setRunning(true);
this.listenerConsumerFuture = containerProperties
.getConsumerTaskExecutor()
.submitListenable(this.listenerConsumer);
}
Created a ListenerConsumer object, which is the listening consumer thread object, and it is a Runnable , then
this.listenerConsumerFuture = containerProperties
.getConsumerTaskExecutor()
.submitListenable(this.listenerConsumer);
}
Here submitListenable submits tasks, getConsumerTaskExecutor defaults to SimpleAsyncTaskExecutor to wrap ListenerConsumer Become ListenableFutureTask, it is FutureTask , this is very important!
The simple understanding of the above paragraph is to submit the Task task, then start a thread to process it, and finally reach the run method, because it is a FutureTask, so it will enter the run method of the FutureTask.
Because the Runnable packaged by ListenableFutureTask is ListenerConsumer, the last executed run method must be in it, that is, it will go to org.springframework.kafka.listener.KafkaMessageListenerContainer.ListenerConsumer#run
@Override
public void run() {
this.consumerThread = Thread.currentThread();
if (this.theListener instanceof ConsumerSeekAware) {
((ConsumerSeekAware) this.theListener).registerSeekCallback(this);
}
if (this.transactionManager != null) {
ProducerFactoryUtils.setConsumerGroupId(this.consumerGroupId);
}
this.count = 0;
this.last = System.currentTimeMillis();
if (isRunning() && this.definedPartitions != null) {
initPartitionsIfNeeded();
}
long lastReceive = System.currentTimeMillis();
long lastAlertAt = lastReceive;
while (isRunning()) {
try {
if (!this.autoCommit && !this.isRecordAck) {
processCommits();
}
processSeeks();
ConsumerRecords<K, V> records = this.consumer.poll(this.containerProperties.getPollTimeout());
this.lastPoll = System.currentTimeMillis();
if (records != null && this.logger.isDebugEnabled()) {
this.logger.debug("Received: " + records.count() + " records");
}
if (records != null && records.count() > 0) {
// 省略部分代码
ConsumerRecords<K, V> records = this.consumer.poll(this.containerProperties.getPollTimeout());
This sentence is to go to the kafka server to pull the message, and find that it is opened while (1) so that the thread does not exit , loop to pull messages.
Summarized as follows:
After the refresh method loads all beans, it starts to complete the refresh (finishRefresh). The Lifecycle post-processor DefaultLifecycleProcessor executes its start method in phase order (the smaller the priority) for all Lifecycle implementation classes , and starts to enter the KafkaListenerEndpointRegistry of the spring-kafka module # start, traverse the listenerContainers collection (the collection contains a series of ConcurrentMessageListenerContainer objects), the timing of adding to the collection is that KafkaListenerAnnotationBeanPostProcessor executes the postProcessAfterInitialization method in the initialization phase to complete, why this method is called, familiar with the bean creation process should know, after the bean initialization is completed It will call the postProcessAfterInitialization method of BeanPostProcessor.
Traverse the listenerContainers collection, then enter ConcurrentMessageListenerContainer#doStart, create a KafkaMessageListenerContainer here, and usually annotate the concurrency concurrency configured by @KafkaListener, which is also used here, and then call KafkaMessageListenerContainer#doStart to generate a real working thread task ListenerConsumer , which Implement Runnable, then package it into ListenableFutureTask (FutureTask), finally submit the task, create a thread Thread, and finally execute the run method of ListenerConsumer, loop to the kafka service to pull messages back for consumption, while (true) ensures that the thread does not exit.