Exploration of kafka topic related issues under CDH

Preface: fan102 corresponds to broker.id=66, fan103 corresponds to broker.id=67, fan104 corresponds to broker.id=65

 

1 topic , partitions , replication

1.1 Create topic

# kafka安装目录
[root@fan102 kafka]# pwd 
/opt/cloudera/parcels/KAFKA/lib/kafka

# 创建topic 主题名：topic_test 分区数：3 副本数：2
[root@fan102 kafka]# ./bin/kafka-topics.sh  --create --zookeeper fan102:2181 --topic topic_test --partitions 3 --replication-factor 2

1.1.2 Results display

1.1.2.1 Log

08:35:19 INFO zk.AdminZkClient: Creating topic topic_test with configuration {} and initial partition assignment Map(2 -> ArrayBuffer(65, 67), 1 -> ArrayBuffer(67, 66), 0 -> ArrayBuffer(66, 65))

 

1.1.2.2 View toppic creation status under each node

The native Hadoop cluster directory is the location specified by log.dirs in the server.properties file

The location of the CDH version is /var/local/kafka/data/

Under the node of broker.id=65, it contains topic_test-0, topic_test-2

Under the node of broker.id=66, it contains topic_test-0, topic_test-1

Under the node of broker.id=67, it contains topic_test-1, topic_test-2

 

1.1.2.3 Conclusion

The topic generation situation is consistent with the log display situation

 

1.2 View topic

[root@fan102 kafka]# ./bin/kafka-topics.sh --zookeeper fan102:2181 --list

1.3 Detailed description of a topic

[root@fan102 kafka]# ./bin/kafka-topics.sh  --describe --zookeeper fan102:2181 --topic topic_test

1.4.1 Modify topic retention is the duration  retention.ms This parameter is the number of milliseconds

[root@fan102 kafka]# kafka-topics.sh --zookeeper node1:2181 -topic xxxx --alter --config retention.ms=259200000

1.4.2 Or modify log.retention.hours in the configuration file server.properties

2 Producer Consumer

Because kerberos is configured, the producer needs to specify  --producer.config , and the consumer needs to specify  --consumer.config

Open kafka under kerberos for reference ( https://blog.csdn.net/shenyuye/article/details/107312806 )

# 生产者
[root@fan102 ~]# kafka-console-producer --broker-list fan102:9092 --topic topic_test --producer.config /etc/kafka/conf/consumer.properties

# 消费者
[root@fan102 ~]# kafka-console-consumer --bootstrap-server fan102:9092 --topic topic_test --from-beginning --consumer.config /etc/kafka/conf/consumer.properties

Input content

>hello
>moto

3 Export message file

3.1 Command

# Convert the .log message file into a .txt file and export

[root@fan102 kafka]# ./bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /var/local/kafka/data/topic_test-0/00000000000000000110.log --print-data-log > 00000000000000000110.txt

3.2 Results

# 导出的.txt文件内容
Dumping /var/local/kafka/data/topic_test-0/00000000000000000110.log
Starting offset: 110
baseOffset: 110 lastOffset: 110 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 26 isTransactional: false isControl: false position: 0 CreateTime: 1595936255468 size: 71 magic: 2 compresscodec: NONE crc: 2806001368 isvalid: true
| offset: 110 CreateTime: 1595936255468 keysize: -1 valuesize: 3 sequence: -1 headerKeys: [] payload: hello
baseOffset: 111 lastOffset: 111 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 26 isTransactional: false isControl: false position: 71 CreateTime: 1595936258382 size: 73 magic: 2 compresscodec: NONE crc: 3650597 isvalid: true
| offset: 111 CreateTime: 1595936258382 keysize: -1 valuesize: 5 sequence: -1 headerKeys: [] payload: moto

4 View topic under ZK 

[root@fan102 ~]# cd /opt/cloudera/parcels/CDH/lib/zookeeper/bin
[root@fan102 ~]# ./zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /
[cluster, controller, brokers, zookeeper, hadoop-ha, admin, isr_change_notification, log_dir_event_notification, controller_epoch, consumers, hive_zookeeper_namespace_hive, latest_producer_id_block, config]
[zk: localhost:2181(CONNECTED) 1] ls /brokers
[ids, topics, seqid]
[zk: localhost:2181(CONNECTED) 2] ls /brokers/ids
[66, 67, 65]
[zk: localhost:2181(CONNECTED) 3] ls /brokers/topics
[topic_test, test, __consumer_offsets, topic_start]

5 Delete topic

5.1 Mark deletion

[root@fan102 kafka]# ./bin/kafka-topics  --delete --zookeeper fan102：2181 --topic topic_test

If " delete.topic.enable=true " is not configured in server.properties in the configuration file loaded when Kafaka starts , then the deletion at this time is not a real deletion, but the topic is marked as deleted: marked for deletion.

5.2 completely delete topic method

5.2.1 Delete the kafka storage directory (the native Hadoop cluster directory is the location specified by log.dirs in the server.properties file, and the location of the CDH version is /var/local/kafka/data/ ) related topic directories

5.2.2.1 If delete.topic.enable=true is configured , delete it directly through the command. If the command cannot be deleted, delete the topic under the broker directly through zookeeper-client.

5.2.2.2 Remove from Zookeeper

[root@fan102 ~]# cd /opt/cloudera/parcels/CDH/lib/zookeeper/bin
[root@fan102 ~]# ./zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /brokers/topics
[topic_test, test, __consumer_offsets,topic_start]
[zk: localhost:2181(CONNECTED) 1] rmr /brokers/topics/topic_test

+++++++++++++++++++++++++++++++++++++++++
+ If you have any questions, you can +Q: 1602701980 Discuss together +
+++++++++++++++++++++++++++++++++++++++++