Article Directory
Zookeeper cluster configuration
Kafka relies heavily on Zookeeper, so Zookeeper must be installed. The following is a simple configuration for local installation. Because there is only one machine and no virtual machine is used, different ports are used.
For details, please refer to Zookeeper cluster installation
tickTime=2000
initLimit=10
syncLimit=5
dataDir=F:\\zkkfk\\data\\zkdata\\zk1
clientPort=2181
tickTime=2000
initLimit=10
syncLimit=5
dataDir=F:\\zkkfk\\data\\zkdata\\zk2
clientPort=2182
tickTime=2000
initLimit=10
syncLimit=5
dataDir=F:\\zkkfk\\data\\zkdata\\zk3
clientPort=2183
Kafka cluster configuration
The following is a simple Kafka configuration, the broker.id in the cluster must be different
broker.id=0
port=9092
log.dirs=F:\\zkkfk\\data\\kfkdata\\k1
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
zookeeper.connection.timeout.ms=6000
offsets.topic.replication.factor=1
broker.id=1
port=9093
log.dirs=F:\\zkkfk\\data\\kfkdata\\k2
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
zookeeper.connection.timeout.ms=6000
offsets.topic.replication.factor=1
#broker集群中的唯一表示，非负数，只要broker.id没有变化，则不会影响consumers的消息情况
broker.id=2
#broker server服务端口
port=9094
#kafka数据的存放地址，多个地址的话用逗号分割D:\\data11，D:\\data12
log.dirs=F:\\zkkfk\\data\\kfkdata\\k3
#ZK集群的地址，可以是多个，多个之间用逗号分割hostname1:port1,hostname2:port2
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
zookeeper.connection.timeout.ms=6000
offsets.topic.replication.factor=1
broker.id is used to uniquely represent the broker in the cluster, and the broker.id in the same cluster must be set differently.
The port is the designated communication port. Both the consumer and producer need to interact with the broker. This port is used. The pseudo cluster of the same machine must be set to different. It is really recommended to use the same port for the cluster.
zookeeper.connect is to set the Zookeeper communication address and must be configured
log.dirs is the data directory, which is the logs directory under the root directory of kafka by default.
Introduction to Kafka Directory
The above picture is the directory where the cluster is installed locally, and data is the directory where the data is stored
kafka-topic
kafka-topic can be used to create and view topics in the cluster
# 创建topic时，通过--config进行指定项的参数配置，覆盖默认配置
kafka-topics.sh --zookeeper localhost:2181 --create --topic topic-name --partitions 1 --replication-factor 1 --config max.message.bytes=64000
kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic multipar_topic
kafka-topics.bat --list --zookeeper localhost:2181
kafka-topics.sh --zookeeper localhost:2181 --topic topic-name --describe
Topic: topic-name PartitionCount:2 ReplicationFactor:3 Configs:
Topic: topic-name Partition: 0 Leader: 218 Replicas: 218,216,217 Isr: 218,216,217
Topic: topic-name Partition: 1 Leader: 216 Replicas: 216,217,218 Isr: 216,218,217
kafka-console-producer
kafka-console-producer is a simple producer that can send messages through the command line, and --broker-list specifies the list of brokers
kafka-console-producer.sh --broker-list 192.168.6.10:9092,192.168.6.11:9092,192.168.6.12:9092 --topic test
kafka-console-producer.sh --broker-list 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094 --topic test
kafka-console-consumer
kafka-console-consumer is a simple consumer, –bootstrap-server specifies the list of brokers
kafka-console-consumer.sh --bootstrap-server 192.168.6.10:9092,192.168.6.11:9092,192.168.6.12:9092 --from-beginning --topic test
kafka-configs
kafka-configs can be used to dynamically view and modify the configuration
# 修改配置
kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name topic-name --alter --add-config max.message.bytes=128000
# 删除配置
kafka-configs.sh --zookeeper localhost:2181 --entity-type topics --entity-name topic-name --alter --delete-config max.message.bytes
# 查看配置参数
kafka-configs.sh --entity-type topics --entity-name topic-name --describe --zookeeper localhost:2181
kafka-producer-perf-test
kafka-producer-perf-test is used for stress testing and can test the impact of the producer on the cluster
| parameter | Description |
|---|---|
| –Num-records | Total number of messages sent |
| –record-size | The size of the simulated message, KB |
| –throughput | Tps upper limit threshold, if there is no tps limit, set to -1 |
| –compression.type | Set message compression type, snappy, lz4 |
kafka-producer-perf-test.sh --topic topicname --num-records 100000000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=localhost:9092 acks=-1 compression.type=lz4
kafka-producer-perf-test.sh --producer-props bootstrap.servers=127.0.0.1:9092 --topic topicname --num-records 1 --record-size 10 --throughput 400
kafka-consumer-perf-test
kafka-consumer-perf-test can be used to pressure test the impact of consumers on the cluster
kafka-consumer-perf-test.sh --zookeeper 127.0.0.1:2181 --topic topicname --message-size 10 --messages 1 --threads 1
kafka-run-class
kafka-run-class can be used to view Kafka data log file information
kafka-run-class.sh kafka.tools.DumpLogSegments --print-data-log --files /usr/local/kafka/data/topicname-partition/00000000000000000000.log
kafka-run-class.bat kafka.tools.DumpLogSegments --print-data-log --files F:/zkkfk/data/kfkdata/k1/multipar_topic-0/00000000000000000000.log
to sum up
