Installation and simple use of Kafka cluster

1. Kafka cluster deployment

1.1. Download the installation package
http://kafka.apache.org/downloads.html
Use the wget command in linux to download the installation package
wget http://mirrors.hust.edu.cn/apache/kafka/0.8.2.2/kafka_2. 11-0.8.2.2.tgz
1.2. Unzip the installation package
tar -zxvf /export/software/kafka_2.11-0.8.2.2.tgz -C /export/servers/
1.3. Modify the configuration file
vi /export/servers/kafka/config /server.properties
enter the following:

#broker的全局唯一编号，不能重复
broker.id=0

#用来监听链接的端口，producer或consumer将在此端口建立连接
port=9092

#处理网络请求的线程数量
num.network.threads=3

#用来处理磁盘IO的线程数量
num.io.threads=8

#发送套接字的缓冲区大小
socket.send.buffer.bytes=102400

#接受套接字的缓冲区大小
socket.receive.buffer.bytes=102400

#请求套接字的缓冲区大小
socket.request.max.bytes=104857600

#kafka运行日志存放的路径
log.dirs=/root/kafkalog

#topic在当前broker上的分片个数
num.partitions=2

#用来恢复和清理data下数据的线程数量
num.recovery.threads.per.data.dir=1

#segment文件保留的最长时间，超时将被删除
log.retention.hours=168

#滚动生成新的segment文件的最大时间
log.roll.hours=168

#日志文件中每个segment的大小，默认为1G
log.segment.bytes=1073741824

#周期性检查文件大小的时间
log.retention.check.interval.ms=300000

#日志清理是否打开
log.cleaner.enable=true

#broker需要使用zookeeper保存meta数据
zookeeper.connect=shizhan:2181,mini2:2181,mini3:2181

#zookeeper链接超时时间
zookeeper.connection.timeout.ms=6000

#partion buffer中，消息的条数达到阈值，将触发flush到磁盘
log.flush.interval.messages=10000

#消息buffer的时间，达到阈值，将触发flush到磁盘
log.flush.interval.ms=3000

#删除topic需要server.properties中设置delete.topic.enable=true否则只是标记删除
delete.topic.enable=true

#此处的host.name为本机IP(重要),如果不改,则客户端会抛出:Producer connection to localhost:9092 unsuccessful 错误!
host.name=192.168.112.200

1.4. Distribute the installation package
scp -r /export/servers/kafka_2.11-0.8.2.2 kafka02:/export/servers

1.5. Modify the configuration file again (important)
Modify the broker.id of the configuration file on each server in turn, which are 0, 1, and 2 and must not be repeated.
Change the IP address of the corresponding host to the IP address of each host.
In addition , change the output address of the generated log file and change the address of
zk to the address of your own machine.
1.6. Start the cluster
Start kafka
bin/kafka-server- start.sh config/server.properties

2. Common operation commands of Kafka

• View all topics in the current server
  bin/kafka-topics.sh --list --zookeeper zk01:2181
• 创建topic
  ./kafka-topics.sh –create –zookeeper mini1:2181 –replication-factor 1 –partitions 3 –topic first
• To delete topic
  sh bin/kafka-topics.sh –delete –zookeeper zk01:2181 –topic test
  needs to set delete.topic.enable=true in server.properties, otherwise it is just marked for deletion or restarted directly.
• Send messages via shell command
  kafka-console-producer.sh --broker-list kafka01:9092 --topic itheima
• Consume messages through shell
  sh bin/kafka-console-consumer.sh --zookeeper zk01:2181 --from-beginning --topic test1
• View the consumption location
  sh kafka-run-class.sh kafka.tools.ConsumerOffsetChecker –zookeeper zk01:2181 –group testGroup
• View the details of a topic
  sh kafka-topics.sh –topic test –describe –zookeeper zk01:2181