Next, build the Kafka runtime environment step by step.
Step 1: Download Kafka
Click to download the latest version and unzip it.
> tar -xzf kafka_2.9.2-0.8.1.1.tgz > cd kafka_2.9.2-0.8.1.1
Step 2: Start the service
Kafka uses Zookeeper, so start Zookper first, and then simply enable a single-instance Zookeeper service. You can add an ampersand at the end of the command, so that you can leave the console after starting.
> bin/zookeeper-server-start.sh config/zookeeper.properties & [2013-04-22 15:01:37,495] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) ...Now start Kafka:
> bin/kafka-server-start.sh config/server.properties [2013-04-22 15:01:47,028] INFO Verifying properties (kafka.utils.VerifiableProperties) [2013-04-22 15:01:47,051] INFO Property socket.send.buffer.bytes is overridden to 1048576 (kafka.utils.VerifiableProperties) ...
Step 3: Create topic
Create a topic called "test" with only one partition and one replica.
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testYou can view the created topics with the list command:
> bin/kafka-topics.sh --list --zookeeper localhost:2181 test
In addition to manually creating topics, you can also configure the broker to automatically create topics.
Step 4: Send a message.
Kafka uses a simple command-line producer to read messages from files or from standard input and send them to the server. The default will send one message per command.
Run the producer and type some messages in the console, which will be sent to the server:
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test This is a messageThis is another message
ctrl+c can quit sending.
Step 5: Start the consumer
Kafka also has a command line consumer that will dump out messages to standard output.
Kafka also has a command line consumer that can read messages and output to standard output:
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning This is a message This is another message
You run the consumer command line in one terminal and the producer command line in another terminal, and you can type messages in one terminal and read them in the other terminal.
These two commands have their own optional parameters, you can see the help information when running without any parameters.
Step 6: Build a cluster of multiple brokers
Just started a single broker, now start a cluster consisting of 3 brokers, and these broker nodes are also on the local machine:
First write a configuration file for each node:
> cp config/server.properties config/server-1.properties
> cp config/server.properties config/server-2.propertiesAdd the following parameters to the copied new file:
config/server-1.properties:
broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
The broker.id uniquely marks a node in the cluster. Because it is on the same machine, different ports and log files must be specified to avoid data overwriting.
We already have Zookeeper and our single node started, so we just need to start the two new nodes:
Just started Zookeeper and one node, now start the other two nodes:
> bin/kafka-server-start.sh config/server-1.properties & ... > bin/kafka-server-start.sh config/server-2.properties & ...Create a topic with 3 replicas:
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
Now that we have built a cluster, how do we know the information of each node? Just run the ""describe topics" command:
> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
These outputs are explained below. The first line is a description of all partitions, and then each partition will have a corresponding line, because we only have one partition, so only one line is added below.
- Leader: Responsible for reading and writing messages, the leader is randomly selected from all nodes.
- replicas: Lists all replica nodes, whether the node is in service or not.
- isr: is the node being served.
In our example, node 1 is running as leader.
Send a message to topic:
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic ... my test message 1my test message 2^CConsume these messages:
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic ... my test message 1 my test message 2 ^C
Test the fault tolerance. Broker 1 is running as leader, now we kill it:
> ps | grep server-1.properties7564 ttys002 0:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/bin/java... > kill -9 7564
Another node is elected as the leader, and node 1 no longer appears in the in-sync replica list:
> bin/kafka-topics.sh --describe --zookeeper localhost:218192 --topic my-replicated-topic
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 2 Replicas: 1,2,0 Isr: 2,0
Although the leader originally responsible for continuing to write messages is down, the previous messages can still be consumed:
> bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic my-replicated-topic ... my test message 1 my test message 2 ^C
It seems that Kafka's fault tolerance mechanism is still good.