kafka zookeeper 集群搭建

Kafka Zookeeper 集群环境部署

环境说明

os version：centos 7.4
kafka version：kafka_2.11-2.0.0
zookeeper version：zookeeper-3.4.10
node count：3

目录规则

/data/zookeeper：zookeeper 存储路径
/data/zookeeper/zookeeper_data：zookeeper 数据路径
/data/zookeeper/zookeeper_datalogs：zookeeper 数据日志路径
/data/zookeeper/zookeeper_logs：zookeeper 日志路径
/data/kafka：kafka 存储路径
/data/kafka/kafka_logs：kafka 日志路径
/data/kafka/kafka_datalogs：kafka 数据日志路径

一、安装zookeeper集群

解压缩

tar zxf zookeeper-3.4.10.tar.gz -C /usr/local

生成配置

cd /usr/local/zookeeper-3.4.10
cp conf/zoo_sample.cfg conf/zoo.cfg

修改配置 zoo.cfg（三台node）
clientPortAddress根据本机ip进行调整

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/zookeeper_data
dataLogDir=/data/zookeeper/zookeeper_datalogs
clientPort=2181
clientPortAddress=10.80.227.169
server.1=10.80.227.169:2888:3888
server.2=10.80.238.7:2888:3888
server.3=10.81.55.98:2888:3888

创建myid文件

echo "1" > /data/zookeeper/zookeeper_data/myid # server.1
echo "2" > /data/zookeeper/zookeeper_data/myid # server.2
echo "3" > /data/zookeeper/zookeeper_data/myid # server.3

编辑 log4j.properties

zookeeper.log.dir=/data/zookeeper/zookeeper_logs
zookeeper.tracelog.dir=/data/zookeeper/zookeeper_logs

编辑 zkServer.sh

_ZOO_DAEMON_OUT="/data/zookeeper/zookeeper_logs/zookeeper.out"

启动 zookeeper 集群

node1：bin/zkServer.sh start
node2：bin/zkServer.sh start
node3：bin/zkServer.sh start

查看 zookeeper 集群状态

node1：bin/zkServer.sh status
node2：bin/zkServer.sh status
node3：bin/zkServer.sh status

二、安装 kafka 集群

解压缩

tar zxf kafka_2.11-2.0.0.tgz -C /usr/local

编辑配置 server.properties（broker.id 和 listeners 要根据节点来调整）

broker.id=0
listeners=PLAINTEXT://10.80.227.169:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka/kafka_datalogs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.80.227.169:2181,10.80.238.7:2181,10.81.55.98:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
message.max.bytes=5242880
delete.topic.enable=true

修改 bin/kafka-server-start.sh

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx4G -Xms4G"
fi

修改 bin/kafka-run-class.sh

JMX_PORT="9999"
LOG_DIR="/data/kafka/kafka_logs"

启动 kafka 集群

node1：bin/kafka-server-start.sh -daemon config/server.properties
node2：bin/kafka-server-start.sh -daemon config/server.properties
node3：bin/kafka-server-start.sh -daemon config/server.properties