table of Contents
1. Use the default PartitionAssignor -> RangeAssignor
2. Use PartitionAssignor -> RoundRobinAssignor
1. Scenario one single-threaded consumption
2. Concurrent consumption in scenario two
I. Introduction
In the process of learning spring-kafka on the official website, MessageListenerContainer is not easy to understand, and real knowledge can be gained from practice, so we can deepen our understanding by typing code and testing.
Official website link: https://docs.spring.io/spring-kafka/docs/2.2.13.RELEASE/reference/html/#message-listener-container
Receive messages by configuring MessageListenerContainer and providing a message listener or using @KafkaListener annotation.
MessageListenerContainer provides two implementations:
KafkaMessageListenerContainer
ConcurrentMessageListenerContainer
The KafkaMessageListenerContainer receives all messages from all topics or partitions on a single thread. One or more of the ConcurrentMessageListenerContainer represents KafkaMessageListenerContainer instances to provide multi-threaded consumption.
ConcurrentMessageListenerContainer构造函数
public ConcurrentMessageListenerContainer(ConsumerFactory<K, V> consumerFactory,
ContainerProperties containerProperties)For this constructor, Kafka uses its group management function to distribute partitions among users.
@KafkaListener can configure clear topics and partitions ( do not configure partitions in the example in this article, because you need to test the partition allocation strategy )
@KafkaListener(id = "thing2", topicPartitions =
{ @TopicPartition(topic = "topic1", partitions = { "0", "1" }),
@TopicPartition(topic = "topic2", partitions = "0",
partitionOffsets = @PartitionOffset(partition = "1", initialOffset = "100"))
})
public void listen(ConsumerRecord<?, ?> record) {
...
}The official tutorial has the following content:
When listening to multiple topics, the default partition distribution may not be what you expect. For example, if you have three topics with five partitions each and you want to use concurrency=15, you see only five active consumers, each assigned one partition from each topic, with the other 10 consumers being idle. This is because the default Kafka PartitionAssignor is the RangeAssignor (see its Javadoc). For this scenario, you may want to consider using the RoundRobinAssignor instead, which distributes the partitions across all of the consumers. Then, each consumer is assigned one topic or partition. To change the PartitionAssignor, you can set the partition.assignment.strategyconsumer property (ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG) in the properties provided to the DefaultKafkaConsumerFactory.
Note: This article tests this content
2. Test preparation
1. Kafka client configuration
Modify the kafka configuration file server.properties (directory D:\ProgramFiles\kafka_2.12\config), configure five partitions
num.partitions = 5
Restart kafka
Note: If you start to report an error, you can clear these 2 directories
2.SpringBoot configuration
The code in this article is changed on the basis of the previous blog "Spring-kafka Introduction to Learning (3): Using SpringBoot to Send and Receive Messages" .
The number of configuration threads is 15
@Bean
ConcurrentKafkaListenerContainerFactory<Integer, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<Integer, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(15);//例如，container.setConcurrency(3)创建三个KafkaMessageListenerContainer实例。
return factory;
}Configure the listener
@KafkaListener(id = "Listener", topics = {"topic_1", "topic_2", "topic_3"}, groupId = "consumer_group_1")
public void listen1(String foo) throws Exception {
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
logger.info(sdf.format(new Date()) + " - Listener-接收消息:" + foo);
Thread.sleep(1000 * 2);
}Message sending interface
@RequestMapping(value = "test2")
public String test2() {
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
//每个topic发送5条数据,每条数据在一个分区
for (int i = 0; i < 5; i++) {
//send方法的参数(String topic, Integer partition, K key, @Nullable V data)
kafkaTemplate.send("topic_1", i, 0, "topic_1_" + i);
kafkaTemplate.send("topic_2", i, 0, "topic_2_" + i);
kafkaTemplate.send("topic_3", i, 0, "topic_3_" + i);
kafkaTemplate.flush();
}
return "test";
}3. Consumer distribution test
1. Use the defaultPartitionAssignor -> RangeAssignor
Start the project and sort out the following logs:
Only five active consumers will be seen, each consumer is assigned a partition for each topic, and the other 10 consumers are idle.
2020-05-12 14:30:24.917 INFO 13152 --- [Listener-10-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-12, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-2, topic_2-2, topic_3-2]
2020-05-12 14:30:24.917 INFO 13152 --- [ Listener-7-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-9, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.917 INFO 13152 --- [ Listener-4-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-6, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.917 INFO 13152 --- [Listener-13-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-15, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-3-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-5, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-5-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-7, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-9-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-11, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-1, topic_2-1, topic_3-1]
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-3, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-8-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-10, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-0, topic_2-0, topic_3-0]
2020-05-12 14:30:24.918 INFO 13152 --- [Listener-12-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-14, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-4, topic_2-4, topic_3-4]
2020-05-12 14:30:24.918 INFO 13152 --- [ Listener-6-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-8, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.919 INFO 13152 --- [ Listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-2, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.919 INFO 13152 --- [Listener-11-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-13, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-3, topic_2-3, topic_3-3]
2020-05-12 14:30:24.920 INFO 13152 --- [Listener-14-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-16, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.920 INFO 13152 --- [ Listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-4, groupId=consumer_group_1] Setting newly assigned partitions []2. Use PartitionAssignor -> RoundRobinAssignor
Assign a topic or section to each consumer. To change PartitionAssignor, you can set partition.assignment.strategyuser attributes ( ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG) in the attributes provided DefaultKafkaConsumerFactory.
Kafka default PartitionAssignoris RangeAssignor，需要设置成RoundRobinAssignor.
Only need to modify one place
Restart the project and sort out the following logs:
It can be seen that there are no idle consumers
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-2, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-2]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-4, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-4]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-8-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-10, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-0]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-3, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-3]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-9-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-11, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-1]
2020-05-12 15:02:57.306 INFO 13720 --- [Listener-11-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-13, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-3]
2020-05-12 15:02:57.306 INFO 13720 --- [ Listener-6-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-8, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-3]
2020-05-12 15:02:57.306 INFO 13720 --- [Listener-14-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-16, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-1]
2020-05-12 15:02:57.307 INFO 13720 --- [ Listener-4-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-6, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-1]
2020-05-12 15:02:57.308 INFO 13720 --- [Listener-12-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-14, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-4]
2020-05-12 15:02:57.308 INFO 13720 --- [ Listener-3-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-5, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-0]
2020-05-12 15:02:57.310 INFO 13720 --- [Listener-10-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-12, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-2]
2020-05-12 15:02:57.312 INFO 13720 --- [ Listener-7-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-9, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-4]
2020-05-12 15:02:57.313 INFO 13720 --- [Listener-13-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-15, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-0]
2020-05-12 15:02:57.315 INFO 13720 --- [ Listener-5-C-1] o.a.k.c.c.internals.ConsumerCoordinator : [Consumer clientId=consumer-7, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-2]Four, performance test
Start SpringBoot, visit the interface: http://127.0.0.1:8080/test2
The listener sleeps for 2s to simulate business time-consuming, which is convenient for log analysis
@KafkaListener(id = "Listener", topics = {"topic_1", "topic_2", "topic_3"}, groupId = "consumer_group_1")
public void listen1(String foo) throws Exception {
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
logger.info(sdf.format(new Date()) + " - Listener-接收消息:" + foo);
Thread.sleep(1000 * 2);
}1. Scenario one single-threaded consumption
factory.setConcurrency(1);Consumption result: poor consumption performance (consumption of a message every 2 seconds)
2020-05-12 15:15:15.223 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:15 - Listener-接收消息:topic_1_0
2020-05-12 15:15:17.224 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:17 - Listener-接收消息:topic_1_4
2020-05-12 15:15:19.224 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:19 - Listener-接收消息:topic_1_3
2020-05-12 15:15:21.224 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:21 - Listener-接收消息:topic_2_4
2020-05-12 15:15:23.225 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:23 - Listener-接收消息:topic_1_2
2020-05-12 15:15:25.225 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:25 - Listener-接收消息:topic_2_3
2020-05-12 15:15:27.226 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:27 - Listener-接收消息:topic_3_4
2020-05-12 15:15:29.227 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:29 - Listener-接收消息:topic_1_1
2020-05-12 15:15:31.227 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:31 - Listener-接收消息:topic_2_2
2020-05-12 15:15:33.228 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:33 - Listener-接收消息:topic_3_3
2020-05-12 15:15:35.229 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:35 - Listener-接收消息:topic_2_1
2020-05-12 15:15:37.230 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:37 - Listener-接收消息:topic_3_2
2020-05-12 15:15:39.230 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:39 - Listener-接收消息:topic_2_0
2020-05-12 15:15:41.231 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:41 - Listener-接收消息:topic_3_1
2020-05-12 15:15:43.232 INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:15:43 - Listener-接收消息:topic_3_02. Concurrent consumption in scenario two
factory.setConcurrency(15);Consumption result: very good consumption performance
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-0-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_2
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-10-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_2
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-12-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_4
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-2-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_4
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-8-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_0
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-5-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_2
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-6-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_3
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-4-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_1
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-7-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_4
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-1-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_3
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-9-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_1
2020-05-12 15:12:56.225 INFO 13720 --- [ Listener-3-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_0
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-11-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_3
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-14-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_1
2020-05-12 15:12:56.225 INFO 13720 --- [Listener-13-C-1] com.asyf.demo.config.Listener : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_0Five, summary
The use of concurrent consumption can increase the consumption rate of messages, but the following issues should be paid attention to:
1. When using the concurrent message listener container, a listener instance will be called on all consumer threads. Therefore, the listener must be thread-safe.
2. Messages that are not in the same partition cannot be read sequentially.