In the previous article "Linux installation Kafka" has been describes how to install Kafka, and Kafka's start in Linux / close, and create messages and send topics and generate consumer news. This article will introduce those commonly used commands introduction of Kafka.
Start and stop on the Kafka / create topic / produce and consume messages and other commands on the article "Linux installation Kafka" has been pointed out in, can not say here. Talk about other commonly used commands.
Sometimes we need to be concerned about the state of consumer applications, the general consumer applications on their own through the log to know the current consumption which offset which partition of which the topic, but when the consumer problems, or for reasons of monitoring, we need to know consumers the status and details, we need to provide the means of the relevant command kafka.
Format:
bin/kafka-consumer-groups.sh --bootstrap-server BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --list
bin/kafka-consumer-groups.sh --bootstrap-server BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --group GROUP_NAME --describe
Tips:
- BROKER_HOST is kafka server's ip address, PORT is the server listening port. Separated by commas plurality of host port
- The first group is to obtain a list of commands, in general, applications are aware of the consumer group, generally in the configuration of the application, if known, this step can be omitted
- The second command is to look at specific consumer group details information, you need to give the name of the group
Example:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group console-consumer-68682 --describe
Renderings:
- TOPIC: Topic name of the group in consumption
- PARTITION: Partition number
- CURRENT-OFFSET: The partition of the current consumption to offset
- LOG-END-OFFSET: The area to which the latest offset
- LAG: Consumer lag interval for LOG-END-OFFSET-CURRENT- OFFSET, applications need to see the specific size of the rate of consumption and producers speeds generally too large potential consumer can not keep up, applications need to attract attention
- ID-CONSUMER: Server end consumer assigned to the partition number
- HOST: Host consumers worldwide
- CLIENT-ID: customer id, generally designated by the application
Use the following command to query the topic: demo broker: localhost: 9092 is the minimum offset:
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic demo --time -2
Queries the maximum offset of:
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic demo --time -1
Renderings:
Some scenes may wish to modify consumer spending to offset position, in order to achieve re-consumption, or skip part of the purpose of the message, this time to reset the tool offset is very practical.
Format:
bin/kafka-consumer-groups.sh --bootstrap-server BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --group GROUP_NAME --reset-offsets --execute --to-offset NEW_OFFSET --topic TOPIC_NAME
bin/kafka-consumer-groups.sh --bootstrap-server BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --group GROUP_NAME --reset-offsets --execute --to-earliest/--to-latest --topic TOPIC_NAME
Tips:
- BROKER_HOST is kafka server's ip address, PORT is the server listening port. Separated by commas plurality of host port
- The first command is specified GROUP_NAME topic and modify the offset position to NEW_OFFSET after the restart consumers, consumption from the specified offset consumption. Note that only NEW_OFFSET can only set a value, that is, all the partitions will use this value, if the partition is not the message load balancing, consider whether you apply.
- The second command is specified GROUP_NAME and topic of offset changes to the earliest or latest position so that consumers from the beginning or from the end consumer.
Example:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group console-consumer-68682 --reset-offsets --execute --to-offset 3 --topic demo
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group console-consumer-68682 --reset-offsets --execute --to-latest --topic demo
Renderings:
Can be replaced by direct consumer group id of the way, with consumer spending default policy, you can achieve a similar effect, but more simple, efficient and safe.
When the broker downtime, recovery, we can look at whether the topic of leader load balancing. Because all requests to read and write messages kafka, it is sent to the partition leader, and thus the production environment, load balancing is especially important.
Format:
bin/kafka-topics.sh --zookeeper ZOOKEEPER_HOST1:PORT1,ZOOKEEPER_HOST2:PORT2 --describe --topic TOPIC_NAME
Tips:
- ZOOKEEPER_HOST is the ip address of the zookeeper's kafka used, PORT zookeeper is listening. Separated by commas plurality of host port
- Similar, the ZooKeeper cluster does not need all of the columns gives an available address and port to zk
Example:
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic demo
Renderings:
If you find the following description kafka abnormal phenomenon:
- Each partition of a topic, inconsistent synchronized copy number and the number of copies set
- id value of each partition of a topic, leader or none is -1
Scratch consumption:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --from-beginning
Renderings:
From the beginning of the tail:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --offset latest --partition 0
Renderings:
Specify the partition:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --offset latest --partition 0
Take specified number:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --offset latest --partition 0 --max-messages 2
As part of an example, take two messages, automatically take complete end session.
Kafka's own test using the test script, written by the command initiated the request MQ messages and Kafka consumption MQ message to Kafka. Simulate different order of magnitude of MQ MQ message and write messages consumption scene, according to the results of the processing of Kafka, assess the ability to handle more than one hundred million messages whether Kafka met.
Format:
bin/kafka-producer-perf-test.sh --topic demo --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=localhost:9092
Example:
| test item | Pressure measurement message number (unit: W) | Test command |
| Write MQ message | 10 |
|
| 100 |
|
|
| 1000 |
|
|
| Consumer MQ message | 10 |
|
| 100 |
|
|
| 1000 |
|
This article translated in part or reference to the following learning materials, special thanks to:
- Using Apache Kafka Command-line Tools
- Resetting the offset of kafka
- Kafka Broker | Command-line Options and Procedure
- kafka operation and maintenance of commonly used commands
- kafka command
Author: Please call me brother head
out at: http://www.cnblogs.com/toutou/
About the author: focus on project development foundation platform. If you have questions or suggestions, please enlighten me a lot!
Disclaimer: This article belongs to the author and blog Park total, welcome to reprint, but without the author's consent declared by this section must be retained, and given the original article page link in the apparent position.
Hereby declare: All comments and private messages will reply in the first time. We greatly welcome the garden also correct errors and common progress. Or direct private letter I
in solidarity Bloggers: If you think the article is helpful to you, you can click on the bottom right corner of the article [ recommend ] it. You are encouraged to adhere to the original author continued writing and maximum power!