6, kafka installation
5.1 three machines installed zookeeper
Note: Before installing the zookeeper make sure the three machines clock synchronization
*/1 * * * * /usr/sbin/ntpdate us.pool.ntp.org;
Three machine configuration file modifications
tickTime=2000
initLimit = 10
syncLimit=5
dataDir=/export/servers/zookeeper-3.4.9/zkData/data
dataLogDir=/export/servers/zookeeper-3.4.9/zkData/log
clientPort = 2181
autopurge.purgeInterval = 1
autopurge.snapRetainCount=3
server.1=node01:2888:3888
server.2=node02:2888:3888
server.3=node03:2888:3888
Three machines were added to the file myid in /export/servers/zookeeper-3.4.9/zkData/data directory and edit the contents of each file
node01 machine myid content 1
node02 machine myid content
node03 machine myid content of 3
Three machines start zookeeper
bin/zkServer.sh start
5.2 three machines installed kafka cluster
5.2.1 download kafka installation archive
http://archive.apache.org/dist/kafka/
5.2.2 Upload archive and extract
Here unified use this version kafka_2.11-1.0.0.tgz
5.2.3 modify configuration files kafka
The first machine to modify the configuration file server.properties kafka
broker.id=0
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/export/servers/kafka_2.11-1.0.0/logs
num.partitions = 2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=node01:2181,node02:2181,node03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
host.name=node01
The second machine kafka modify configuration files server.properties
broker.id=1
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/export/servers/kafka_2.11-1.0.0/logs
num.partitions = 2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=node01:2181,node02:2181,node03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
host.name=node02
The third machine revise kafka profile server.properties
broker.id=2
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/export/servers/kafka_2.11-1.0.0/logs
num.partitions = 2
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=node01:2181,node02:2181,node03:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true
host.name=node03
5.2.4 Start kafka cluster
Three machines start kafka Service
./kafka-server-start.sh ../config/server.properties
nohup bin / kafka-server-start.sh config / server.properties> / dev / null 2> & 1 & start command background
7, kafka use command-line management
Creating topickafka-topics.sh --create --partitions 3 --replication-factor 2 --topic kafkatopic --zookeeper node01: 2181, node02: 2181, node03: 2181 Analog producer kafka-console-producer.sh --broker -list node01: 9092, node02: 9092, node03: 9092 --topic kafkatopic simulate consumption