When upgrading a project from Springboot 2.0 to 2.1, we also upgraded spring-kafka-integration from 3.0.1 to 3.2.1. In doing so, our tests now all fail to start due to:
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=x] Connection to node -1 could not be established. Broker may not be available.
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=x] Connection to node -1 could not be established. Broker may not be available.
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=x] Connection to node -1 could not be established. Broker may not be available.
...
org.springframework.context.ApplicationContextException: Failed to start bean 'eventInboundFlow.kafka:message-driven-channel-adapter#0'; nested exception is org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
Our build machines do not have a Kafka running locally, and tests that utilize an EmbeddedKafkaBroker do so with a custom JUnit5 extension that stops/starts the container listeners between tests (while seeking all partitions and topics to latest, to prevent tests accidentally leaking messages to break expectations in later tests). While far faster than @DirtiesContext, it doesn't inject itself into the context during configuration like the @EmbddedKafka annotation does.
In the previous version, this wasn't a problem; We would see some log messages about not being able to connect while the extension configured and started the broker, but then all was fine.
In the new version the context fails to start entirely (the custom extension never even gets a chance to run). Looking at the properties, the only setting I can see around startup failure is spring.kafka.admin.fail-fast, but that is FALSE by default and we don't change it.
Comparing it to starting the project itself as a springboot app, the first difference I see is that containers are started in their own threads when run as an app, but on the main / Test Worker thread when run as a test. In the previous version, the tests also started the containers in their own thread.
Any insight into why the tests behave differently now? Or if there is a way to configure it to get them off the main thread?
Set the container property missingQueuesFatal to false.
See the Spring for Apacher Kafka 2.2. What's new.
A new container property (missingTopicsFatal) has been added. See Using KafkaMessageListenerContainer for more information.
Starting with version 2.2, a new container property called missingTopicsFatal has been added (default: true). This prevents the container from starting if any of the configured topics are not present on the broker. It does not apply if the container is configured to listen to a topic pattern (regex). Previously, the container threads looped within the consumer.poll() method waiting for the topic to appear while logging many messages. Aside from the logs, there was no indication that there was a problem. To restore the previous behavior, you can set the property to false.
Setting the property to false disables the check.