Ubuntu虚拟机 Zookeeper+Kafka集群搭建

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/qq_26437925/article/details/82933140

mac下使用了FinalShell操作虚拟机

机器和环境

三台ubuntu 16.04 64bit

• 设置相互免密码ssh
• 设置可以通过root用户ssh

ip地址	zookeeper主从	broker id
172.16.227.128	follower	1
172.16.227.129	leader	2
172.16.227.130	follower	3

• jdk 1.8
• kafka_2.11-1.1.0
• zookeeper-3.4.12/

172.16.227.128的配置包括如下(同理其它两台机器)

• /opt/zookeeper/zookeeper-3.4.12/conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/opt/zookeeper/data
dataLogDir=/opt/zookeeper/log
# the port at which the clients will connect
clientPort=2181
server.1=172.16.227.128:2888:3888
server.2=172.16.227.129:2888:3888
server.3=172.16.227.130:2888:3888

• /opt/zookeeper/data/myid (对应server.1)

1

• /opt/kafka_2.11-1.1.0/config/ server.properties

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
hostname.name=172.16.227.128
port=9092

advertised.host.name=172.16.227.128
advertised.port=9092

zookeeper.connect=172.16.227.128:2181,172.16.227.129:2182,172.16.227.130:2181

zookeeper和kafka的启动

---

kafka 分别在三台机器上启动

cd /opt/kafka_2.11-1.1.0
bin/kafka-server-start.sh config/server.properties 

在172.16.227.129机器上创建topic并查看

bin/kafka-topics.sh --create --zookeeper 172.16.227.128:2181,172.16.227.129:2181,172.16.227.130:2181 --replication-factor 2 --partitions 2 --topic kafkatest

bin/kafka-topics.sh --describe --zookeeper 172.16.227.128:2181,172.16.227.129:2181,172.16.227.130:2181 --topic kafkatest

该topic有两个partion 和2个replica
第0个 partition 的 leader 是1(即172.16.227.128)
第1个 partition 的 leader 是2(即172.16.227.129)

• 利用kafka-tool工具连接查看
  

可用性测试

root@ubuntu:/opt/kafka_2.11-1.1.0# bin/kafka-topics.sh --describe --zookeeper 172.16.227.128:2181,172.16.227.129:2181,172.16.227.130:2181 --topic kafkatest
Topic:kafkatest PartitionCount:2        ReplicationFactor:2     Configs:
        Topic: kafkatest        Partition: 0    Leader: 1       Replicas: 1,2   Isr: 1,2
        Topic: kafkatest        Partition: 1    Leader: 2       Replicas: 2,3   Isr: 3,2
root@ubuntu:/opt/kafka_2.11-1.1.0# 

如上, broker 2是个leader

• 产生一些消息

root@ubuntu:/opt/kafka_2.11-1.1.0#  bin/kafka-console-producer.sh --broker-list 172.16.227.128:9092,172.16.227.129:9092,172.16.227.130:9092 --topic kafkatest
>msg1
>mgs22
>

• 消费

 bin/kafka-console-consumer.sh --bootstrap-server 172.16.227.128:9092,172.16.227.129:9092,172.16.227.130:9092 --from-beginning --topic kafkatest

通过kafka-tool查看

broker 2不可用的测试

此时运行bin/zkServer.sh status发现，broker 1是follower, broker 3变成了leader

kafka-tool创建一个新连接，kafka依旧是可用的

• broker 2 kafka Server ctrl+c
  

查看此时的topic (leader变化了)

虽然broker 2挂了，但消息仍能被消费

参考：

https://blog.csdn.net/u014388408/article/details/52871718