kafka environment configuration

1. Download the kafka installation package and unzip it

sudo tar -zxvf kafka_2.12-1.0.1.tgz
sudo mv kafka_2.12-1.0.1 /usr/local/kafka

 

2. Modify the configuration file

zookeeper.properties

cd kafka/config
sudo vim zookeeper.properties

dataDir=/home/hadoop/zookeeper/kafka/data/zookeeper
dataLogDir=/home/hadoop/zookeeper/kafka/log/zookeeper
clientPort=2181
maxClientCnxns=100
tickTime=2000
initLimit=10
syncLimit=5

server.properties

cd kafka/config
sudo vim server.properties

broker.id=0
port=9092
host.name=hadoop  #此处为主机名或IP地址
log.dirs=/home/hadoop/zookeeper/kafka/log/kafka/
zookeeper.connect=localhost:2181

 

3. Create a startup and shutdown file

    1. Create a startup file

cd kafka
sudo vim ./kafkastart.sh

#!/bin/bash
/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties &


sleep 3

/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties &

 

2. Create a shutdown file

cd kafka
sudo vim ./kafkastop.sh

#!/bin/bash
#关闭zookeeper
/usr/local/kafka/bin/zookeeper-server-stop.sh /usr/local/kafka/config/zookeeper.properties &
sleep 3 #等3秒后执行
#关闭kafka
/usr/local/kafka/bin/kafka-server-stop.sh /usr/local/kafka/config/server.properties &

 

4. There are three starting zookeeper processes

$ cd kafka
$ ./kafkastart.sh
hadoop@hadoop:/usr/local/zookeeper/bin$ jps
14517 Kafka
14245 QuorumPeerMain
15383 Jps

 

create topic

hadoop@hadoop:/usr/local/kafka/bin$ ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic te
Created topic "te".

View topic

cd /usr/local/kafka/bin
./kafka-topics.sh --list --zookeeper localhost:2181 

production news

$ cd /usr/local/kafka/bin
$ ./kafka-console-producer.sh --broker-list localhost:9092 --topic test
laoyang I love you! 

Consumption message test

$ cd /usr/local/kafka/bin
$ ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
I'm laoyang #之前测试输入的内容
laoyang I love you! 

 

5. Close kafka

$ cd /usr/local/kafka
$ ./kafkastop.sh