Regarding command line operations (Note: Before performing command line operations, be sure to enable all zookeeper)
1. View the names of all topics currently in Kafka
bin/kafka-topics.sh --zookeeper hadoop102:2181 --list
[root@hadoop102 kafka]# bin/kafka-topics.sh --zookeeper hadoop102:2181 --list
demo1
demo2
2. View the details of a topic, here I am viewing the details of the topic demo2
bin/kafka-topics.sh --zookeeper hadoop102:2181 --describe --topic xx主题
[root@hadoop102 kafka]# bin/kafka-topics.sh --zookeeper hadoop102:2181 --topic demo2 --describe
Topic:demo2 PartitionCount:3 ReplicationFactor:2 Configs:
Topic: demo2 Partition: 0 Leader: 2 Replicas: 2,1 Isr: 2,1
Topic: demo2 Partition: 1 Leader: 0 Replicas: 0,2 Isr: 0,2
Topic: demo2 Partition: 2 Leader: 1 Replicas: 1,0 Isr: 1,0
[root@hadoop102 kafka]#
3. Create a theme
bin/kafka-topics.sh --zookeeper hadoop102:2181 --create --partitions xx --replication-factor xx --topic xx
Parameter interpretation:
2181: Zookeeper's port number, because operations such as creating nodes must be registered on zk.
--Partitions: defines the number of partitions
--replication-factor defines the number of copies
bin/kafka-topics.sh --zookeeper hadoop102:2181 --create --partitions 3 --replication-factor 2 --topic demo
4. Delete a topic
bin/kafka-topics.sh --zookeeper hadoop102:2181 --delete --topic demo2
Note that you need to set delete.topic.enable=true in /opt/moudlu/kafka/conf/server.properties, otherwise it will just mark the deletion without actually deleting
5. The producer sends a message to the Kafka cluster
bin/kafka-console-producer.sh --broker-list hadoop102:9092 --topic xx主题
6. Consumers accept the message
1) Before the 0.11 version, the consumer's offset was stored in zookeeper. Although we are now using the 0.11 version of Kafka, zk can still receive the message.
bin/kafka-console-consumer.sh \
--zookeeper hadoop102:2181 --topic first
2) After the 0.11 version, the offset containing the 0.11 consumer is stored in the Kafka cluster. If the above connection zk is used to consume messages, it will appear
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [ bootstrap-server ] instead of [zookeeper]
is also recommended to use bootstrap-server to connect
bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --topic demo2
Since there is very little to modify the partition in daily work, I will not post his command line operation here.
Next, let’s take a look at the partition allocation problem in
Kafka. We can first create a topic called demo2, which contains 3 partitions and 2 copies.
bin/kafka-topics.sh --zookeeper hadoop102:2181 --create --partitions 3 --replication-factor 2 --topic demo2
After it is created, let’s describe its specific information. Let’s first emphasize that there are two copies in the topic we just created. It stands to reason that there should be one leader plus two copies for three, but in fact, the so-called copy is equal to one leader + more. One follow, but one copy is leader, and the rest are follow . Here you can see that there are three partitions of theme demo2, which are 0,1,2. About Leader2,1,0 these three numbers mean our leader storage On which kafka machine, because we use broker.id to uniquely identify our kafka machine, here my kafka cluster is three.
Here we look at partition 0. From Replicas, we can see that it has two copies. The first one is the id number of the broker where the leader is located, and the following is the id number of the broker where the follower is located. For the
same reason, let’s take a look. For partition 1, its leader is brokenerid number 0, and follow is on the machine with id number 2.
Because broker0, 1, and 2 respectively represent my hadoop102, 103, and 104 machines, let's check the corresponding /kafka/logs (the producer's data is under this path). Note The system naming rule here is the subject plus the partition number.
hadoop102 equivalent broker0
hadoop103 equivalent broker1
hadoop104 equivalent broker2
From the above figure, we can get the final distribution method as
