3.kafka生产者消费者测试

開発 2023-06-21 10:21:19 訪問数: null

1.eclipse新建maven项目

eclipse新建maven项目，pom.xml文件添加kafka-clients依赖
在这里插入图片描述
kafka-clients下载地址

2.修改配置

idea 之前已经启动了kafka的core模块，修改server.properties

添加localhost
listeners=PLAINTEXT://localhost:9092

配置文件尽可能使用主机名，不使用ip

core模块启动参考

3.生产者代码


import java.util.Properties;
import java.util.concurrent.Future;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class Producer {
    
    
	private static final Logger logger = LoggerFactory.getLogger(Producer.class);

	@SuppressWarnings("resource")
	public static void main(String[] args) throws Exception {
    
    
		Properties properties = new Properties();
		properties.put("bootstrap.servers", "localhost:9092");
		properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
		properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
		
		KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);
		String data = "hi";
		ProducerRecord<String, String> record = new ProducerRecord<String, String>("hi", data);
		Future<RecordMetadata> future = producer.send(record);
		RecordMetadata recordMetadata = future.get();
		logger.info("send data： {},offset ：{}", data, recordMetadata.offset());
	}

}

注意

  1. server.properties的listeners=PLAINTEXT://localhost:9092注释要放开，添加主机名
  2. 生产者配置信息bootstrap.servers，主机名和端口要配置正确

否则报错

 MINAServer Logger--> INFO{
    
    AppInfoParser.java:83}-Kafka version : 0.11.0.3
  MINAServer Logger--> INFO{
    
    AppInfoParser.java:84}-Kafka commitId : 26ddb9e3197be39a
  MINAServer Logger--> WARN{
    
    NetworkClient.java:589}-Connection to node -1 could not be established. Broker may not be available.
  MINAServer Logger--> WARN{
    
    NetworkClient.java:589}-Connection to node -1 could not be established. Broker may not be available.
  MINAServer Logger--> WARN{
    
    NetworkClient.java:589}-Connection to node -1 could not be established. Broker may not be available.

4.运行生产者

执行两次
运行部分日志如下

MINAServer Logger--> INFO{
    
    AbstractConfig.java:223}-ProducerConfig values: 
	略
	org.apache.kafka.clients.producer.internals.DefaultPartitioner
	略

  MINAServer Logger--> INFO{
    
    AppInfoParser.java:83}-Kafka version : 0.11.0.3
  MINAServer Logger--> INFO{
    
    AppInfoParser.java:84}-Kafka commitId : 26ddb9e3197be39a
  MINAServer Logger--> INFO{
    
    Producer.java:27}-send data： hi,offset ：1

5. 消费者代码

import java.util.Collections;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class Consumer {
    
    
	private static final Logger logger = LoggerFactory.getLogger(Consumer.class);

	@SuppressWarnings("resource")
	public static void main(String[] args) {
    
    
		Properties properties = new Properties();
		properties.put("bootstrap.servers", "localhost:9092");
		properties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
		properties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
		properties.put("group.id", "hi");
		properties.put("client.id", "consumer-client");
		properties.put("auto.offset.reset", "earliest");

		KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(properties);
		consumer.subscribe(Collections.singletonList("hi"));
		while(true) {
    
    
			ConsumerRecords<String, String> records = consumer.poll(1000L);
			records.forEach(record -> {
    
    
				logger.info("offset is :{}, value is :{}", record.offset(), record.value());
			});
		}
	}
}

6. 运行消费者

部分日志如下

MINAServer Logger--> INFO{
    
    AbstractConfig.java:223}-ConsumerConfig values: 
  略
  MINAServer Logger--> INFO{
    
    AppInfoParser.java:83}-Kafka version : 0.11.0.3
  MINAServer Logger--> INFO{
    
    AppInfoParser.java:84}-Kafka commitId : 26ddb9e3197be39a
  MINAServer Logger--> INFO{
    
    AbstractCoordinator.java:607}-Discovered coordinator localhost:9092 (id: 2147483647 rack: null) for group hi.
  MINAServer Logger--> INFO{
    
    ConsumerCoordinator.java:419}-Revoking previously assigned partitions [] for group hi
  MINAServer Logger--> INFO{
    
    AbstractCoordinator.java:442}-(Re-)joining group hi
  MINAServer Logger--> INFO{
    
    AbstractCoordinator.java:652}-Marking the coordinator localhost:9092 (id: 2147483647 rack: null) dead for group hi
  MINAServer Logger--> INFO{
    
    AbstractCoordinator.java:607}-Discovered coordinator localhost:9092 (id: 2147483647 rack: null) for group hi.
  MINAServer Logger--> INFO{
    
    AbstractCoordinator.java:442}-(Re-)joining group hi
  MINAServer Logger--> INFO{
    
    AbstractCoordinator.java:409}-Successfully joined group hi with generation 1
  MINAServer Logger--> INFO{
    
    ConsumerCoordinator.java:262}-Setting newly assigned partitions [hi-0] for group hi
  MINAServer Logger--> INFO{
    
    Consumer.java:30}-offset is :0, value is :hi
  MINAServer Logger--> INFO{
    
    Consumer.java:30}-offset is :1, value is :hi

7. 观察idea中启动的kafka core日志

  略
 INFO Loading producer state from offset 0 for partition __consumer_offsets-3 with message format version 2 (kafka.log.Log)
 INFO Completed load of log __consumer_offsets-3 with 1 log segments, log start offset 0 and log end offset 0 in 7 ms (kafka.log.Log)
 INFO Created log for partition [__consumer_offsets,3] in F:\resources\kafka\logs with properties {
    
    compression.type -> producer, message.format.version -> 0.11.0-IV2, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> compact, flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 104857600, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
 INFO Partition [__consumer_offsets,3] on broker 0: No checkpointed highwatermark is found for partition __consumer_offsets-3 (kafka.cluster.Partition)
 [ INFO Replica loaded for partition __consumer_offsets-3 with initial high watermark 0 (kafka.cluster.Replica)
 INFO Partition [__consumer_offsets,3] on broker 0: __consumer_offsets-3 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)
 INFO Loading producer state from offset 0 for partition __consumer_offsets-13 with message format version 2 (kafka.log.Log)
 [ INFO Completed load of log __consumer_offsets-13 with 1 log segments, log start offset 0 and log end offset 0 in 8 ms (kafka.log.Log)
 INFO Created log for partition [__consumer_offsets,13] in F:\resources\kafka\logs with properties {
    
    compression.type -> producer, message.format.version -> 0.11.0-IV2, file.delete.delay.ms -> 60000, max.message.bytes -> 1000012, min.compaction.lag.ms -> 0, message.timestamp.type -> CreateTime, min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096, unclean.leader.election.enable -> false, retention.bytes -> -1, delete.retention.ms -> 86400000, cleanup.policy -> compact, flush.ms -> 9223372036854775807, segment.ms -> 604800000, segment.bytes -> 104857600, retention.ms -> 604800000, message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. (kafka.log.LogManager)
 INFO Partition [__consumer_offsets,13] on broker 0: No checkpointed highwatermark is found for partition __consumer_offsets-13 (kafka.cluster.Partition)
 INFO Replica loaded for partition __consumer_offsets-13 with initial high watermark 0 (kafka.cluster.Replica)
 INFO Partition [__consumer_offsets,13] on broker 0: __consumer_offsets-13 starts at Leader Epoch 0 from offset 0. Previous Leader Epoch was: -1 (kafka.cluster.Partition)
  
  略
  
 INFO [Group Metadata Manager on Broker 0]: Finished loading offsets and group metadata from __consumer_offsets-36 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
 INFO [Group Metadata Manager on Broker 0]: Finished loading offsets and group metadata from __consumer_offsets-39 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
 INFO [Group Metadata Manager on Broker 0]: Finished loading offsets and group metadata from __consumer_offsets-42 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
 INFO [Group Metadata Manager on Broker 0]: Finished loading offsets and group metadata from __consumer_offsets-45 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
 INFO [Group Metadata Manager on Broker 0]: Finished loading offsets and group metadata from __consumer_offsets-48 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
 INFO [GroupCoordinator 0]: Preparing to rebalance group hi with old generation 0 (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 0]: Stabilized group hi generation 1 (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 0]: Assignment received from leader for group hi for generation 1 (kafka.coordinator.group.GroupCoordinator)
 INFO Updated PartitionLeaderEpoch. New: {
    
    epoch:0, offset:0}, Current: {
    
    epoch:-1, offset-1} for Partition: __consumer_offsets-29. Cache now contains 0 entries. (kafka.server.epoch.LeaderEpochFileCache)
 INFO [GroupCoordinator 0]: Preparing to rebalance group hi with old generation 1 (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 0]: Stabilized group hi generation 2 (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 0]: Assignment received from leader for group hi for generation 2 (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 0]: Preparing to rebalance group hi with old generation 2 (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 0]: Member consumer-client-0b3a48bb-5f12-4ec9-ab89-bd019ca1d339 in group hi has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 0]: Member consumer-client-8b6972ab-56a3-4584-b382-f7e8d88c52b4 in group hi has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 0]: Stabilized group hi generation 3 (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator)
 INFO [GroupCoordinator 0]: Assignment received from leader for group hi for generation 3 (kafka.coordinator.group.GroupCoordinator)
 INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 7 milliseconds. (kafka.coordinator.group.GroupMetadataManager)

おすすめ

転載: blog.csdn.net/u010895512/article/details/118369280
おすすめ
LFOSSA Yuanlaisusu オープン コース | クラウド ネイティブの未来をマスターする: CNCF 認定資格と試験準備のヒントの包括的なガイド
ランキング
[ブリックジャワ]学ぶための5分、算術演算子
英国のエッセイを書くには、次の点に注意してください。
【質疑応答シリーズ】バックエンドから返されるjsonの出力をフォーマットし、コードを強調表示する方法
イベントが発生したYunqi 2019会議、？
非常に使いやすいAndroid 6.0の許可申請ツールを推奨
インターセプトし、使用にVueの-CLI3 axiosルート
CentOS Stream 9 でピーナッツ シェルをインストールして使用し、イントラネットに侵入して Web サイトを構築する方法
CreditEaseマイクロタスクスケジューリングサービスプラットフォームの構築の練習|シェアを記録
WrapPanelのWPFのレイアウト
keras 深層学習フレームワークは、畳み込みニューラル ネットワーク cnn を通じて手書き数字認識を実現します
アーカイブ
もっと
2024-04-26(22)
2024-04-25(31)
2024-04-24(30)
2024-04-23(31)
2024-04-22(5)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)
2024-04-17(31)

コードワールド

© 2018 codetd.com