Start zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
Setting up a multi-broker cluster
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
cp config/server.properties config/server-3.properties
config/server-1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dir=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dir=/tmp/kafka-logs-2
config/server-3.properties:
broker.id=3
listeners=PLAINTEXT://:9094
log.dir=/tmp/kafka-logs-3
Start multi-broker cluster
bin/kafka-server-start.sh config/server-1.properties &
bin/kafka-server-start.sh config/server-2.properties &
bin/kafka-server-start.sh config/server-3.properties &
Create a topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
Check topics
bin/kafka-topics.sh --list --zookeeper localhost:2181
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Send some messages
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic
Start a consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
Fault-tolerance Test
Kill one Kafka process
ps to | grep server-1.properties
kill -9 7564
Check the topic again
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
The messages are still available for consumption even though the leader that took the writes originally is down
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
refer to:
http://kafka.apache.org/documentation/