文章目录
1.集群规划
部署kafka集群最好是奇数台至少3台服务器，也可以在一台服务中部署三个kafka，修改端口号即可。
| IP | 主机名 | 部署的服务 |
|---|---|---|
| 192.168.20.10 | kafka-1 | kafka+zookeeper |
| 192.168.20.11 | kafka-2 | kafka+zookeeper |
| 192.168.20.12 | kafka-3 | kafka+zookeeper |
2.部署Zookeeper集群
2.1.在三台服务器中分别部署Zookeeper节点
1.安装JAVA环境
tar xf jdk-8u211-linux-x64.tar.gz -C /data/
vim /etc/profile
JAVA_HOME=/data/jdk1.8.0_211
PATH=$JAVA_HOME/bin:$PATH
source /etc/profile
2.安装Zookeeper
tar xf apache-zookeeper-3.5.8-bin.tar.gz -C /data/
mv /data/apache-zookeeper-3.5.8-bin/ /data/zookeeper_cluster
3.创建数据存储路径
mkdir /data/zookeeper_cluster/data
4.准备配置文件
cp /data/zookeeper_cluster/conf/zoo_sample.cfg /data/zookeeper_cluster/conf/zoo.cfg
2.2.配置各个节点的数据存储路径
1.zookeeper-1节点
[root@zookeeper-1 ~]# vim /data/zookeeper_cluster/conf/zoo.cfg
dataDir=/data/zookeeper_cluster/data
2.zookeeper-1节点
[root@zookeeper-2 ~]# vim /data/zookeeper_cluster/conf/zoo.cfg
dataDir=/data/zookeeper_cluster/data
3.zookeeper-1节点
[root@kzookeeper-3~]# vim /data/zookeeper_cluster/conf/zoo.cfg
dataDir=/data/zookeeper_cluster/data
到此为止每个节点都配置完成了，下面进行集群模式的配置
2.3.将三节点配置成集群模式
1.为每个节点创建ServerID文件
1.zookeeper-1节点
[root@zookeeper-1 ~]# echo 1 > /data/zookeeper_cluster/data/myid
2.zookeeper-1节点
[root@zookeeper-2 ~]# echo 2 > /data/zookeeper_cluster/data/myid
3.zookeeper-1节点
[root@zookeeper-3 ~]# echo 3 > /data/zookeeper_cluster/data/myid
2.修改每个节点的配置文件增加集群各节点的通信地址
server.1=192.168.20.10:2888:3888server.后面的数字就是每个节点创建的serverid号，2888是服务器之间的通信端口，3888是服务器之间的投票选举端口。
1.zookeeper-1节点
[root@kzookeeper-1~]# vim /data/zookeeper_cluster/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper_cluster/data
clientPort=2181
server.1=192.168.20.10:2888:3888
server.2=192.168.20.11:2888:3888
server.3=192.168.20.12:2888:3888
2.zookeeper-2节点
[root@kzookeeper-2~]# vim /data/zookeeper_cluster/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper_cluster/data
clientPort=2181
server.1=192.168.20.10:2888:3888
server.2=192.168.20.11:2888:3888
server.3=192.168.20.12:2888:3888
3.zookeeper-3节点
[root@kzookeeper-3~]# vim /data/zookeeper_cluster/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper_cluster/data
clientPort=2181
server.1=192.168.20.10:2888:3888
server.2=192.168.20.11:2888:3888
server.3=192.168.20.12:2888:3888
2.4.启动Zookeeper集群
1.zookeeper-1节点
[root@zookeeper-1 ~]# /data/zookeeper_cluster/bin/zkServer.sh start
2.zookeeper-1节点
[root@zookeeper-2 ~]# /data/zookeeper_cluster/bin/zkServer.sh start
3.zookeeper-1节点
[root@zookeeper-3 ~]# /data/zookeeper_cluster/bin/zkServer.sh start
2.5.查看集群的状态
可以看到集群中zookeeper-1节点是Follower角色，zookeeper-2节点是Leader角色，zookeeper-3节点是Follower角色，正就验证了我们起那么说的那句话，当ServerID大并且投票超过半数则当选Leader节点，zookeeper-2节点启动后，ServerID最大，并且zookeeper-1和zookeeper-2都投了投票，票数也超过了集群数量的半数以上，因此当选成功。
1.zookeeper-1节点
[root@zookeeper-1 ~]# /data/zookeeper_cluster/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/zookeeper_cluster/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
2.zookeeper-1节点
[root@zookeeper-2 ~]# /data/zookeeper_cluster/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/zookeeper_cluster/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
3.zookeeper-1节点
[root@zookeeper-3 ~]# /data/zookeeper_cluster/bin/zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /data/zookeeper_cluster/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
3.部署Kafka集群
3.1.在三台服务中部署Kafka
tar xf kafka_2.11-2.4.1.tgz -C /data/
mv /data/kafka_2.11-2.4.1/ /data/kafka
3.2.配置Kafka每个节点
kafka-1节点配置
[root@kafka-1 ~]# vim /data/kafka/config/server.properties
broker.id=0 #指定BrokerID，每个节点都不同
listeners=PLAINTEXT://192.168.20.10:9092 #指定当前Kafka节点的地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/data/kafka-logs #指定Kafka数据存储路径
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.20.10:2181,192.168.20.11:2181,192.168.20.12:2181 #指定ZK集群地址
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
kafka-2节点配置
[root@kafka-2 ~]# vim /data/kafka/config/server.properties
broker.id=1 #指定BrokerID，每个节点都不同
listeners=PLAINTEXT://192.168.20.11:9092 #指定当前Kafka节点的地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/data/kafka-logs #指定Kafka数据存储路径
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.20.10:2181,192.168.20.11:2181,192.168.20.12:2181 #指定ZK集群地址
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
kafka-3节点配置
[root@kafka-3 ~]# vim /data/kafka/config/server.properties
broker.id=2 #指定BrokerID，每个节点都不同
listeners=PLAINTEXT://192.168.20.12:9092 #指定当前Kafka节点的地址
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/data/kafka-logs #指定Kafka数据存储路径
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.20.10:2181,192.168.20.11:2181,192.168.20.12:2181 #指定ZK集群地址
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
3.3.启动Kafka集群
[root@kafka-1 ~]# /data/kafka/bin/kafka-server-start.sh -daemon /data/kafka/config/server.properties
[root@kafka-2 ~]# /data/kafka/bin/kafka-server-start.sh -daemon /data/kafka/config/server.properties
[root@kafka-3 ~]# /data/kafka/bin/kafka-server-start.sh -daemon /data/kafka/config/server.properties
4.在ZK集群中查看Kafka集群信息
在zookeeper集群中查看Kafka注册的broker信息，三个节点会在brokers子节点中有三个节点。
[root@kafka-1 ~]# /data/zookeeper_cluster/bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 3] ls /brokers/ids
[0, 1, 2]