[6] Kafka cluster construction

Create a data directory for kafka:

mkdir -p data/kafka
scp -r data/ 192.168.56.102:`pwd`
scp -r data/ 192.168.56.103:`pwd`

# 将 kafka 解压至 
tar -zxvf kafka_2.12-2.3.0.tgz -C /usr/local/
# 将 kafka 目录同步到其他两台机器
scp -r kafka_2.12-2.3.0/ 192.168.56.102:/usr/local
scp -r kafka_2.12-2.3.0/ 192.168.56.103:/usr/local

Configure on three machines separately

config/server.properties
192.168.56.101：

broker.id=0
# 数据目录
log.dirs=/root/data/kafka
# 指定hostname, 可通过 hostname 命令查看 hostname
listeners=PLAINTEXT://192.168.56.101:9092
zookeeper.connect=192.168.56.101:2181,192.168.56.102:2181,192.168.56.103:2181

192.168.56.102：

broker.id=1
log.dirs=/root/data/kafka
listeners=PLAINTEXT://192.168.56.102:9092
# 配置zk集群
zookeeper.connect=192.168.56.101:2181,192.168.56.102:2181,192.168.56.103:2181

192.168.56.103:

broker.id=2
log.dirs=/root/data/kafka
listeners=PLAINTEXT://192.168.56.103:9092
zookeeper.connect=192.168.56.101:2181,192.168.56.102:2181,192.168.56.103:2181

Start Kafka

Start kafka on three machines separately

bin/kafka-server-start.sh config/server.properties &

Create topic

bin/kafka-topics.sh  --zookeeper 192.168.56.101:2181 --create    --topic market_topic --partitions 4  --replication-factor 3
# 查看topic列表
bin/kafka-topics.sh --list --zookeeper 192.168.56.101:2181
#启动一个消费者
bin/kafka-console-consumer.sh --bootstrap-server 192.168.56.101:9092,192.168.56.102:9092,192.168.56.103:9092 --topic market_topic
# 启动一个生产者
bin/kafka-console-producer.sh --broker-list 192.168.56.101:9092,192.168.56.102:9092,192.168.56.103:9092 --topic market_topic

After creation, we can see our 4 partitions in the data directory:

# 查看 topic 信息
bin/kafka-topics.sh --zookeeper 192.168.56.101:2181 --describe pic market_topic

The red box indicates which machine the partition is on, the machine where the replicas are distributed, and Isr indicates that the data of the follower is synchronized with the leader.