Aprendizado introdutório Spring-kafka (4): Exemplo de teste ConcurrentMessageListenerContainer do contêiner do listener de mensagem

Others 2021-03-29 01:40:35 views: null

índice

I. Introdução

2. Preparação de teste

1. Configuração do cliente Kafka

2. Configuração do SpringBoot

3. Teste de distribuição ao consumidor

1. Use o padrão PartitionAssignor -> RangeAssignor

2. Use PartitionAssignor -> RoundRobinAssignor

Quatro, teste de desempenho

1. Cenário um consumo de thread único

2. Consumo simultâneo no cenário dois

Cinco, resumo

I. Introdução

No processo de aprendizagem do spring-kafka no site oficial, MessageListenerContainer não é fácil de entender e o conhecimento real pode ser obtido com a prática, para que possamos aprofundar nosso entendimento digitando o código e testando.

Link do site oficial: https://docs.spring.io/spring-kafka/docs/2.2.13.RELEASE/reference/html/#message-listener-container

Receba mensagens configurando MessageListenerContainer e fornecendo um listener de mensagens ou usando a anotação @KafkaListener.

MessageListenerContainer fornece duas implementações:
KafkaMessageListenerContainer
ConcurrentMessageListenerContainer

O KafkaMessageListenerContainer recebe todas as mensagens de todos os tópicos ou partições em um único encadeamento. Um ou mais dos ConcurrentMessageListenerContainer representam instâncias KafkaMessageListenerContainer para fornecer consumo multiencadeado.

ConcurrentMessageListenerContainer构造函数

public ConcurrentMessageListenerContainer(ConsumerFactory<K, V> consumerFactory,
                            ContainerProperties containerProperties)

Para este construtor, o Kafka usa sua função de gerenciamento de grupo para distribuir partições entre os usuários.

@KafkaListener pode configurar tópicos e partições claros ( não configure partições no exemplo deste artigo, porque você precisa testar a estratégia de alocação de partição )

@KafkaListener(id = "thing2", topicPartitions =
        { @TopicPartition(topic = "topic1", partitions = { "0", "1" }),
          @TopicPartition(topic = "topic2", partitions = "0",
             partitionOffsets = @PartitionOffset(partition = "1", initialOffset = "100"))
        })
public void listen(ConsumerRecord<?, ?> record) {
    ...
}

O tutorial oficial tem o seguinte conteúdo:

Ao ouvir vários tópicos, a distribuição de partição padrão pode não ser o que você espera. Por exemplo, se você tiver três tópicos com cinco partições cada e desejar usar  concurrency=15, verá apenas cinco consumidores ativos, cada um atribuído a uma partição de cada tópico, com os outros 10 consumidores inativos. Isso ocorre porque o Kafka padrão  PartitionAssignor é o  RangeAssignor (consulte seu Javadoc). Para este cenário, você pode querer considerar o uso do  RoundRobinAssignor , que distribui as partições por todos os consumidores. Em seguida, cada consumidor recebe um tópico ou partição. Para alterar o  PartitionAssignor, você pode definir a partition.assignment.strategypropriedade do  consumidor ( ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG) nas propriedades fornecidas ao  DefaultKafkaConsumerFactory.

Nota: este artigo testa este conteúdo

2. Preparação de teste

1. Configuração do cliente Kafka

Modifique o arquivo de configuração kafka server.properties (diretório D: \ ProgramFiles \ kafka_2.12 \ config), configure cinco partições

num.partitions = 5

Reiniciar kafka

Nota: Se você começar a relatar um erro, pode limpar esses 2 diretórios

2. Configuração do SpringBoot

O código neste artigo foi alterado com base no blog anterior "Introdução ao spring-kafka (3): Use SpringBoot para enviar e receber mensagens" .

O número de threads de configuração é 15

  @Bean
    ConcurrentKafkaListenerContainerFactory<Integer, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<Integer, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        factory.setConcurrency(15);//例如，container.setConcurrency(3)创建三个KafkaMessageListenerContainer实例。
        return factory;
    }

Configure o ouvinte

    @KafkaListener(id = "Listener", topics = {"topic_1", "topic_2", "topic_3"}, groupId = "consumer_group_1")
    public void listen1(String foo) throws Exception {
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        logger.info(sdf.format(new Date()) + " - Listener-接收消息:" + foo);
        Thread.sleep(1000 * 2);
    }

Interface de envio de mensagens

    @RequestMapping(value = "test2")
    public String test2() {
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        //每个topic发送5条数据,每条数据在一个分区
        for (int i = 0; i < 5; i++) {
            //send方法的参数(String topic, Integer partition, K key, @Nullable V data)
            kafkaTemplate.send("topic_1", i, 0, "topic_1_" + i);
            kafkaTemplate.send("topic_2", i, 0, "topic_2_" + i);
            kafkaTemplate.send("topic_3", i, 0, "topic_3_" + i);
            kafkaTemplate.flush();
        }
        return "test";
    }

3. Teste de distribuição ao consumidor

1. Use o padrãoPartitionAssignor -> RangeAssignor

Inicie o projeto e classifique os seguintes registros:

Apenas cinco consumidores ativos serão vistos, cada consumidor recebe uma partição para cada tópico e os outros 10 consumidores estão inativos.

2020-05-12 14:30:24.917  INFO 13152 --- [Listener-10-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-12, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-2, topic_2-2, topic_3-2]
2020-05-12 14:30:24.917  INFO 13152 --- [ Listener-7-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-9, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.917  INFO 13152 --- [ Listener-4-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-6, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.917  INFO 13152 --- [Listener-13-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-15, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-3-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-5, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-5-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-7, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-9-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-11, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-1, topic_2-1, topic_3-1]
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-3, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-8-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-10, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-0, topic_2-0, topic_3-0]
2020-05-12 14:30:24.918  INFO 13152 --- [Listener-12-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-14, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-4, topic_2-4, topic_3-4]
2020-05-12 14:30:24.918  INFO 13152 --- [ Listener-6-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-8, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.919  INFO 13152 --- [ Listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.919  INFO 13152 --- [Listener-11-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-13, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-3, topic_2-3, topic_3-3]
2020-05-12 14:30:24.920  INFO 13152 --- [Listener-14-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-16, groupId=consumer_group_1] Setting newly assigned partitions []
2020-05-12 14:30:24.920  INFO 13152 --- [ Listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-4, groupId=consumer_group_1] Setting newly assigned partitions []

2. Use PartitionAssignor -> RoundRobinAssignor

Atribua um tópico ou seção a cada consumidor. Para alterar PartitionAssignor, você pode definir partition.assignment.strategyos atributos do usuário ( ConsumerConfigs.PARTITION_ASSIGNMENT_STRATEGY_CONFIG) nos atributos fornecidos DefaultKafkaConsumerFactory.

O padrão do Kafka  PartitionAssignoré RangeAssignor，需要设置成RoundRobinAssignor.

Só precisa modificar um lugar

Reinicie o projeto e classifique os seguintes registros:

Percebe-se que não há consumidores ociosos

2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-2]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-2-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-4, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-4]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-8-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-10, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-0]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-1-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-3, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-3]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-9-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-11, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-1]
2020-05-12 15:02:57.306  INFO 13720 --- [Listener-11-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-13, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-3]
2020-05-12 15:02:57.306  INFO 13720 --- [ Listener-6-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-8, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-3]
2020-05-12 15:02:57.306  INFO 13720 --- [Listener-14-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-16, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-1]
2020-05-12 15:02:57.307  INFO 13720 --- [ Listener-4-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-6, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-1]
2020-05-12 15:02:57.308  INFO 13720 --- [Listener-12-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-14, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-4]
2020-05-12 15:02:57.308  INFO 13720 --- [ Listener-3-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-5, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-0]
2020-05-12 15:02:57.310  INFO 13720 --- [Listener-10-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-12, groupId=consumer_group_1] Setting newly assigned partitions [topic_1-2]
2020-05-12 15:02:57.312  INFO 13720 --- [ Listener-7-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-9, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-4]
2020-05-12 15:02:57.313  INFO 13720 --- [Listener-13-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-15, groupId=consumer_group_1] Setting newly assigned partitions [topic_2-0]
2020-05-12 15:02:57.315  INFO 13720 --- [ Listener-5-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-7, groupId=consumer_group_1] Setting newly assigned partitions [topic_3-2]

Quatro, teste de desempenho

Inicie o SpringBoot, visite a interface: http://127.0.0.1:8080/test2

O ouvinte dorme por 2s para simular um negócio demorado, o que é conveniente para a análise de log

 @KafkaListener(id = "Listener", topics = {"topic_1", "topic_2", "topic_3"}, groupId = "consumer_group_1")
    public void listen1(String foo) throws Exception {
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        logger.info(sdf.format(new Date()) + " - Listener-接收消息:" + foo);
        Thread.sleep(1000 * 2);
    }

1. Cenário um consumo de thread único

factory.setConcurrency(1);

Resultado de consumo: baixo desempenho de consumo (consumo de uma mensagem a cada 2 segundos)

2020-05-12 15:15:15.223  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:15 - Listener-接收消息:topic_1_0
2020-05-12 15:15:17.224  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:17 - Listener-接收消息:topic_1_4
2020-05-12 15:15:19.224  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:19 - Listener-接收消息:topic_1_3
2020-05-12 15:15:21.224  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:21 - Listener-接收消息:topic_2_4
2020-05-12 15:15:23.225  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:23 - Listener-接收消息:topic_1_2
2020-05-12 15:15:25.225  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:25 - Listener-接收消息:topic_2_3
2020-05-12 15:15:27.226  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:27 - Listener-接收消息:topic_3_4
2020-05-12 15:15:29.227  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:29 - Listener-接收消息:topic_1_1
2020-05-12 15:15:31.227  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:31 - Listener-接收消息:topic_2_2
2020-05-12 15:15:33.228  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:33 - Listener-接收消息:topic_3_3
2020-05-12 15:15:35.229  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:35 - Listener-接收消息:topic_2_1
2020-05-12 15:15:37.230  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:37 - Listener-接收消息:topic_3_2
2020-05-12 15:15:39.230  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:39 - Listener-接收消息:topic_2_0
2020-05-12 15:15:41.231  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:41 - Listener-接收消息:topic_3_1
2020-05-12 15:15:43.232  INFO 17188 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:15:43 - Listener-接收消息:topic_3_0

2. Consumo simultâneo no cenário dois

factory.setConcurrency(15);

Resultado de consumo: desempenho de consumo muito bom

2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-0-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_2
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-10-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_2
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-12-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_4
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-2-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_4
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-8-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_0
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-5-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_2
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-6-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_3
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-4-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_1
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-7-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_4
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-1-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_3
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-9-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_1
2020-05-12 15:12:56.225  INFO 13720 --- [ Listener-3-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_3_0
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-11-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_1_3
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-14-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_1
2020-05-12 15:12:56.225  INFO 13720 --- [Listener-13-C-1] com.asyf.demo.config.Listener            : 2020-05-12 15:12:56 - Listener-接收消息:topic_2_0

Cinco, resumo

O uso de consumo simultâneo pode aumentar a taxa de consumo de mensagens, mas os seguintes problemas devem ser observados:

1. Ao usar o contêiner do ouvinte de mensagem simultânea, uma instância do ouvinte será chamada em todos os encadeamentos do consumidor. Portanto, o ouvinte deve ser thread-safe.

2. As mensagens que não estão na mesma partição não podem ser lidas sequencialmente.

Acho que você gosta

Origin blog.csdn.net/cs373616511/article/details/106075185
Recomendado
Clasificación
Diario
Más
2024-06-15(0)
2024-06-14(0)
2024-06-13(0)
2024-06-12(0)
2024-06-11(0)
2024-06-10(0)
2024-06-09(0)
2024-06-08(0)
2024-06-07(0)
2024-06-06(0)

Code World

© 2018 codetd.com