Multiple consumers subscribe to a Kafka Topic (using KafkaConsumer and KafkaProducer)

Records : 466

Scenario : A KafkaProducer publishes a message on a Topic, and multiple KafkaConsumers subscribe to the Kafka Topic. Each KafkaConsumer specifies a specific ConsumerGroup, so that a message can be consumed by multiple different ConsumerGroups.

Versions : JDK 1.8, Spring Boot 2.6.3, kafka_2.12-2.8.0, kafka-clients-3.0.0.

Kafka cluster installation : https://blog.csdn.net/zhangbeizhen18/article/details/131156084

1. Basic concepts

Topic : Kafka classifies messages according to Topic, and each message published to Kafka needs to specify a Topic.

Producer : The message producer, the client that sends messages to the Broker.

Consumer : Message consumer, the client that reads messages from Broker.

ConsumerGroup : Each Consumer belongs to a specific ConsumerGroup, and a message can be consumed by multiple different ConsumerGroups; but only one Consumer in a ConsumerGroup can consume the message.

publish : Publish, use Producer to write data to Kafka.

subscribe : Subscribe, use Consumer to read data from Kafka.

2. Configure Kafka information in microservices

2.1 Add dependencies in pom.xml

pom.xml file:

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>3.0.0</version>
</dependency>

Analysis: Use native kafka-clients, version: 3.0.0. Operate the topic of the Kafka cluster.

3. Configure Kafka producers and consumers

To use native kafka-clients, you need to configure KafkaProducer and KafkaConsumer, and inject the configuration information of the Kafka cluster into these two objects to operate the producer and consumer.

The configuration details are in the configuration of the official website: https://kafka.apache.org/documentation/

3.1 Configure KafkaProducer producer

(1) sample code

@Configuration
public class KafkaClusterConfig {
  @Bean
  public KafkaProducer kafkaProducer() {
      Map<String, Object> configs = new HashMap<>();
      //kafka集群
      Collection<String> cluster = Lists.newArrayList("192.168.19.161:29092",
              "192.168.19.162:29092",
              "192.168.19.163:29092");
      configs.put("bootstrap.servers", cluster);
      //客户端发送服务端失败的重试次数
      configs.put("retries", 2);
      //多个记录被发送到同一个分区时,生产者将尝试将记录一起批处理成更少的请求.
      //此设置有助于提高客户端和服务器的性能,配置控制默认批量大小(以字节为单位)
      configs.put("batch.size", 16384);
      //生产者可用于缓冲等待发送到服务器的记录的总内存字节数(以字节为单位)
      configs.put("buffer-memory", 33554432);
      //生产者producer要求leader节点在考虑完成请求之前收到的确认数,用于控制发送记录在服务端的持久化
      //acks=0,设置为0,则生产者producer将不会等待来自服务器的任何确认.该记录将立即添加到套接字(socket)缓冲区并视为已发送.在这种情况下,无法保证服务器已收到记录,并且重试配置(retries)将不会生效(因为客户端通常不会知道任何故障),每条记录返回的偏移量始终设置为-1.
      //acks=1,设置为1,leader节点会把记录写入本地日志,不需要等待所有follower节点完全确认就会立即应答producer.在这种情况下,在follower节点复制前,leader节点确认记录后立即失败的话,记录将会丢失.
      //acks=all,acks=-1,leader节点将等待所有同步复制副本完成再确认记录,这保证了只要至少有一个同步复制副本存活,记录就不会丢失.
      configs.put("acks", "-1");
      //指定key使用的序列化类
      Serializer keySerializer = new StringSerializer();
      //指定value使用的序列化类
      Serializer valueSerializer = new StringSerializer();
      //创建Kafka生产者
      KafkaProducer kafkaProducer = new KafkaProducer(configs, keySerializer, valueSerializer);
      return kafkaProducer;
  }
}

(2) Analysis code

Inject Kafka configuration information into KafkaProducer and create a KafkaProducer object.

Use @Configuration and @Bean annotations to inject the KafkaProducer object into Spring's IOC container, and you can use KafkaProducer in the Spring environment.

The underlying configuration class of KafkaProducer is ProducerConfig, which can be referred to during configuration.

全称：org.apache.kafka.clients.producer.ProducerConfig。

3.2 Configure the public configuration information of KafkaConsumer

(1) sample code

@Configuration
public class KafkaClusterConfig {
  @Bean("consumerConfig")
  public Map<String, Object> consumerConfigs() {
    Map<String, Object> configs = new HashMap<>();
    //kafka集群
    Collection<String> cluster = Lists.newArrayList("192.168.19.161:29092",
            "192.168.19.162:29092",
            "192.168.19.163:29092");
    configs.put("bootstrap.servers", cluster);
    //开启consumer的偏移量(offset)自动提交到Kafka
    configs.put("enable.auto.commit", true);
    //consumer的偏移量(offset) 自动提交的时间间隔,单位毫秒
    configs.put("auto.commit.interval", 5000);
    //在Kafka中没有初始化偏移量或者当前偏移量不存在情况
    //earliest, 在偏移量无效的情况下, 自动重置为最早的偏移量
    //latest, 在偏移量无效的情况下, 自动重置为最新的偏移量
    //none, 在偏移量无效的情况下, 抛出异常.
    configs.put("auto.offset.reset", "latest");
    //请求阻塞的最大时间(毫秒)
    configs.put("fetch.max.wait", 500);
    //请求应答的最小字节数
    configs.put("fetch.min.size", 1);
    //心跳间隔时间(毫秒)
    configs.put("heartbeat-interval", 3000);
    //一次调用poll返回的最大记录条数
    configs.put("max.poll.records", 500);
    return configs;
  }
  @Bean("keyDeserializer")
  public Deserializer consumerKeyDeserializer() {
    //指定key使用的反序列化类
    Deserializer keyDeserializer = new StringDeserializer();
    return keyDeserializer;
  }
  @Bean("valueDeserializer")
  public Deserializer consumerValueDeserializer() {
    //指定value使用的反序列化类
    Deserializer valueDeserializer = new StringDeserializer();
    return valueDeserializer;
  }
}

(2) Analysis code

Configure consumer KafkaConsumer public configuration information: @Bean("consumerConfig"), @Bean("keyDeserializer"), @Bean("valueDeserializer").

When creating a KafkaConsumer, you only need to get the public consumerConfig, keyDeserializer, valueDeserializer and set them to KafkaConsumer.

4. Producer (ZhejiangProvinceProducerController)

(1) sample code

@RestController
@RequestMapping("/hub/example/province/producer")
@Slf4j
public class ZhejiangProvinceProducerController {
  //1.注入Kafka生产者
  @Autowired
  private KafkaProducer kafkaProducer;
  //2.定义Kafka的Topic
  private final String topicName = "hub-topic-province-notice";
  @GetMapping("/f01_1")
  public Object f01_1(String msgContent) {
    try {
        //3.获取业务数据
        String uuid = UUID.randomUUID().toString().replace("-", "");
        long now = System.currentTimeMillis();
        String msgKey = "province" + ":" + uuid + ":" + now;
        MsgDto msgDto = MsgDto.buildDto(uuid, now, msgContent);
        String msgData = JSONObject.toJSONString(msgDto);
        log.info("ZhejiangProvince生产者向Kafka集群的Topic: {},写入Key:", topicName);
        log.info(msgKey);
        log.info("ZhejiangProvince生产者向Kafka集群的Topic: {},写入Data:", topicName);
        log.info(msgData);
        //4.使用KafkaProducer向Kafka集群写入数据
        ProducerRecord<String, String> producerRecord = new ProducerRecord<>(topicName, msgKey, msgData);
        kafkaProducer.send(producerRecord);
    } catch (Exception e) {
        log.info("ZhejiangProvince生产者写入Topic异常.");
        e.printStackTrace();
    }
    return "写入成功";
  }
}

(2) Analysis code

Use KafkaProducer to write JSON string data to the Kafka cluster's Topic: hub-topic-province-notice, publish a message, and consume it to subscribed consumers.

5. Consumer 1 (HangzhouCityConsumer)

(1) sample code

@Component
@Slf4j
public class HangzhouCityConsumer implements CommandLineRunner {
  //1.注入消费者配置信息,key和value的序列化对象
  @Autowired
  private Deserializer keyDeserializer;
  @Autowired
  private Deserializer valueDeserializer;
  //2.定义Kafka的Topic
  private final String topicName = "hub-topic-province-notice";
  @Override
  public void run(String... args) throws Exception {
      //3.创建线程并传入线程任务执行的Runnable
      Thread thread = new Thread(new HangzhouCityConsumer.ThreadRunnable());
      //4.启动线程
      thread.start();
  }
  //在线程中使用KafkaConsumer实时监听Kafka集群的Topic
  public class ThreadRunnable implements Runnable {
      @Override
      public void run() {
          log.info("HangzhouCityConsumer启动线程监听Kafka集群的Topic: {}", topicName);
          Collection<String> topics = Lists.newArrayList(topicName);
          //1.指定消费组(一条消息可以被多个不同的ConsumerGroup消费)
          Map<String, Object> configs = new HashMap<>();
          Map<String, Object> consumerConfig = SpringUtil.getBean("consumerConfig");
          configs.putAll(consumerConfig);
          configs.put("group.id", "hub-topic-province-notice-group-hangzhou");
          //2.创建Kafka消费者(传入消费者配置和key和value的序列化对象)
          KafkaConsumer kafkaConsumer = new KafkaConsumer(configs, keyDeserializer, valueDeserializer);
          //3.订阅Kafka的Topic
          kafkaConsumer.subscribe(topics);
          while (true) {
              //4.使用KafkaConsumer的poll按照指定周期轮询Kafka集群指定Topic的消息
              ConsumerRecords<String, String> consumerRecords = kafkaConsumer.poll(Duration.ofMillis(1000));
              //5.遍历从Kafka集群中读取数据集ConsumerRecords
              for (ConsumerRecord consumerRecord : consumerRecords) {
                  //6.从ConsumerRecord中取出消费数据
                  String msgKey = (String) consumerRecord.key();
                  String msgData = (String) consumerRecord.value();
                  log.info("HangzhouCityConsumer从Kafka集群中的Topic:{},消费的原始数据的Key:", topicName);
                  log.info(msgKey);
                  log.info("HangzhouCityConsumer从Kafka集群中的Topic:{},消费的原始数据的Data:", topicName);
                  log.info(msgData);
              }
          }
      }
  }
}

(2) Analysis code

Obtain configuration information consumerConfig, keyDeserializer, valueDeserializer from the configuration class.

Use the group.id attribute to specify the consumer group: hub-topic-province-notice-group-hangzhou.

Use the subscribe method of KafkaConsumer to subscribe to the topic: hub-topic-province-notice.

Use the poll method of KafkaConsumer to poll the Topic: hub-topic-province-notice, and the message data of the consumed Topic is stored in the record result set: ConsumerRecords.

Traverse the result set ConsumerRecords to obtain specific data.

6. Consumer 2 (NingboCityConsumer)

(1) sample code

@Component
@Slf4j
public class NingboCityConsumer implements InitializingBean {
  //1.注入消费者配置信息,key和value的序列化对象
  @Autowired
  private Deserializer keyDeserializer;
  @Autowired
  private Deserializer valueDeserializer;
  //2.定义Kafka的Topic
  private final String topicName = "hub-topic-province-notice";
  @Override
  public void afterPropertiesSet() throws Exception {
      //3.创建线程,在线程中使用KafkaConsumer实时监听Kafka集群的Topic
      Thread thread = new Thread(() -> {
          log.info("NingboCity启动线程监听Topic: {}", topicName);
          Collection<String> topics = Lists.newArrayList(topicName);
          //1.指定消费组(一条消息可以被多个不同的ConsumerGroup消费)
          Map<String, Object> configs = new HashMap<>();
          Map<String, Object> consumerConfig = SpringUtil.getBean("consumerConfig");
          configs.putAll(consumerConfig);
          configs.put("group.id", "hub-topic-province-notice-group-ningbo");
          //2.创建Kafka消费者
          KafkaConsumer kafkaConsumer = new KafkaConsumer(configs, keyDeserializer, valueDeserializer);
          //3.订阅Kafka的Topic
          kafkaConsumer.subscribe(topics);
          while (true) {
              ThreadUtil.sleep(200);
              //4.使用KafkaConsumer的poll按照指定周期轮询Kafka集群指定Topic的消息
              ConsumerRecords<String, String> consumerRecords = kafkaConsumer.poll(Duration.ofMillis(1000));
              for (ConsumerRecord consumerRecord : consumerRecords) {
                  //5.KafkaConsumer从集群中监听的消息存储在ConsumerRecord
                  String msgKey= (String) consumerRecord.key();
                  String msgData = (String) consumerRecord.value();
                  log.info("NingboCityConsumer从Kafka集群中的Topic:{},消费的原始数据的Key:",topicName);
                  log.info(msgKey);
                  log.info("NingboCityConsumer从Kafka集群中的Topic:{},消费的原始数据的Data:",topicName);
                  log.info(msgData);
              }
          }
      });
      //9.启动线程
      thread.start();
  }
}

(2) Analysis code

Obtain configuration information consumerConfig, keyDeserializer, valueDeserializer from the configuration class.

Use the group.id attribute to specify the consumer group: hub-topic-province-notice-group-ningbo.

Use the subscribe method of KafkaConsumer to subscribe to the topic: hub-topic-province-notice.

Use the poll method of KafkaConsumer to poll the Topic: hub-topic-province-notice, and the message data of the consumed Topic is stored in the record result set: ConsumerRecords.

Traverse the result set ConsumerRecords to obtain specific data.

7. Test

(1) Use the Postman test to call the producer to write data

请求RUL：http://127.0.0.1:18210/hub-210-kafka/hub/example/province/producer/f01_1

Parameters: msgContent="Zhejiang Province is fully committed to economic development"

(2) Producer log

ZhejiangProvince生产者向Kafka集群的Topic: hub-topic-province-notice,写入Key:
province:b9418d4ae1f44a198684abfa59aaec2a:1687791091150
ZhejiangProvince生产者向Kafka集群的Topic: hub-topic-province-notice,写入Data:
{"msgContent":"浙江省全力发展经济","publicTime":"2023-06-26 22:51:31","uuid":"b9418d4ae1f44a198684abfa59aaec2a"}

(3) Consumer-log

HangzhouCityConsumer从Kafka集群中的Topic:hub-topic-province-notice,消费的原始数据的Key:
province:b9418d4ae1f44a198684abfa59aaec2a:1687791091150
HangzhouCityConsumer从Kafka集群中的Topic:hub-topic-province-notice,消费的原始数据的Data:
{"msgContent":"浙江省全力发展经济","publicTime":"2023-06-26 22:51:31","uuid":"b9418d4ae1f44a198684abfa59aaec2a"}

(4) Consumer 2 log

NingboCityConsumer从Kafka集群中的Topic:hub-topic-province-notice,消费的原始数据的Key:
province:b9418d4ae1f44a198684abfa59aaec2a:1687791091150
NingboCityConsumer从Kafka集群中的Topic:hub-topic-province-notice,消费的原始数据的Data:
{"msgContent":"浙江省全力发展经济","publicTime":"2023-06-26 22:51:31","uuid":"b9418d4ae1f44a198684abfa59aaec2a"}

(5 Conclusion

Each Consumer specifies a specific ConsumerGroup, and a message can be consumed by multiple different ConsumerGroups.

8. Auxiliary class

@Data
@Builder
public class MsgDto implements Serializable {
  private String uuid;
  private String publicTime;
  private String msgContent;
  public static MsgDto buildDto(String uuid,
                      long publicTime,
                      String msgContent) {
      return builder().uuid(uuid)
          .publicTime(DateUtil.formatDateTime(new Date(publicTime)))
          .msgContent(msgContent).build();
  }
}

Above, thanks.

June 26, 2023