Basic commands of Kafka Shell (including adding, deleting, modifying and checking topics)

Create kafka topic

bin/kafka-topics.sh --zookeeper node01:2181 --create --topic t_cdr --partitions 30  --replication-factor 2

Note: partitions specifies the number of topic partitions, and replication-factor specifies the number of replicas for each partition of the topic

 

partitionsNumber of partitions:

partitions : The number of partitions, which controls how many logs the topic will be sharded into. It can be specified explicitly, if not specified, the number of num.partitions configuration in broker (server.properties) will be used

Although increasing the number of partitions can improve the throughput of the kafka cluster, too many partitions or too many partitions on a single server will increase the risk of unavailability and latency. Because of the large number of partitions, it means that more file handles need to be opened, the point-to-point delay is increased, and the memory consumption of the client is increased.

The number of partitions also limits the parallelism of the consumer, that is, the number of threads for parallel consumer messages cannot be greater than the number of partitions

The number of partitions also limits the producer to send messages to the specified partition. If the partition is set to 1 when creating a topic, it will be an error if the producer sends a message by specifying a partition of 2 or more through a custom partition method; in this case, you can use alter –partitions to increase the number of partitions.

replication-factor replica

The replication factor control message is stored on several brokers (servers), which is generally equal to the number of brokers.

If it is not explicitly specified when creating or producing messages to a non-existing topic through the API, the number of default.replication.factor configured in broker (server.properties) will be used

 

View a list of all topics

bin/kafka-topics.sh --zookeeper node01:2181 --list

 

View the specified topic information

bin/kafka-topics.sh --zookeeper node01:2181 --describe --topic t_cdr

 

Console produces data to topic

bin/kafka-console-producer.sh --broker-list node86:9092 --topic t_cdr

 

Console consumes topic data

bin/kafka-console-consumer.sh  --zookeeper node01:2181  --topic t_cdr --from-beginning

 

View the maximum (small) value of the offset of a certain partition of the topic

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic hive-mdatabase-hostsltable  --time -1 --broker-list node86:9092 --partitions 0

Note: When time is -1, it means the maximum value, and when time is -2, it means the minimum value

 

 

Increase the number of topic partitions

Add 10 partitions for topic t_cdr

 

bin/kafka-topics.sh --zookeeper node01:2181  --alter --topic t_cdr --partitions 10

 

Delete topic, use it with caution, only the metadata in zookeeper will be deleted, and the message file must be deleted manually

bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --zookeeper node01:2181 --topic t_cdr

 

View topic consumption progress

This will show the offset of the consumer group. The parameter must be --group, if --topic is not specified, the default is all topics

 

Displays the: Consumer Group, Topic, Partitions, Offset, logSize, Lag, Owner for the specified set of Topics and Consumer Group

 

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker

 

required argument: [group] 

Option Description 

------ ----------- 

--broker-info Print broker info 

--group Consumer group. 

--help Print this message. 

--topic Comma-separated list of consumer 

   topics (all topics if absent). 

--zkconnect ZooKeeper connect string. (default: localhost:2181)

 

Example,

 

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group pv

 

Group           Topic              Pid Offset   logSize    Lag    Owner 

pv              page_visits        0   21       21         0      none 

pv              page_visits        1   19       19         0      none 

 

pv              page_visits        2   20       20         0      none