The most complete hands-on stand-alone deployment of pseudo-cluster zookeeper and kafka

Note that all the following ips are 127.0.0.1 and need to be replaced by the local machine ip, otherwise the subsequent client will fail to connect through the ip connection

Deploy pseudo-cluster zookeeper

Create a directory and upload the package

# 新建目录
mkdir /opt/zookeeper
cd /opt/zookeeper
# 上传apache-zookeeper-3.6.4-bin.tar.gz包 
tar zxf apache-zookeeper-3.6.4-bin.tar.gz
mv apache-zookeeper-3.6.4-bin zookeeper-1

Edit zk node 1 configuration file

cd zookeeper-1/conf/
mv zoo_sample.cfg zoo.cfg
vim zoo.cfg

Mainly modify the following places, the port must be modified, and the log directory is different for each node

clientPort=2181
dataLogDir=/opt/zookeeper/zookeeper-1/logs
dataDir=/opt/zookeeper/zookeeper-1/data
server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2889:3889
server.3=127.0.0.1:2890:3890

Create node id file

mkdir /opt/zookeeper/zookeeper-1/data
echo "1" > /opt/zookeeper/zookeeper-1/data/myid 

Node 1 configuration ends

------------------------------------------------------------------------------------------------------------------------

Copy Node 2 and Node 3

cd /opt/zookeeper
cp -r zookeeper-1 zookeeper-2
cp -r zookeeper-1 zookeeper-3

configure node 2

cd zookeeper-2
vi conf/zoo.cfg

Modify as follows

clientPort=2182
dataLogDir=/opt/zookeeper/zookeeper-2/logs
dataDir=/opt/zookeeper/zookeeper-2/data

Create node id file

echo "2" > /opt/zookeeper/zookeeper-2/data/myid 

configure node 3

cd /opt/zookeeper/zookeeper-3
vi conf/zoo.cfg

Modify as follows

clientPort=2183
dataLogDir=/opt/zookeeper/zookeeper-3/logs
dataDir=/opt/zookeeper/zookeeper-3/data

Create node id file

echo "3" > /opt/zookeeper/zookeeper-3/data/myid 

start zk

/opt/zookeeper/zookeeper-1/bin/zkServer.sh start
/opt/zookeeper/zookeeper-2/bin/zkServer.sh start
/opt/zookeeper/zookeeper-3/bin/zkServer.sh start

View zk node status

/opt/zookeeper/zookeeper-1/bin/zkServer.sh status
/opt/zookeeper/zookeeper-2/bin/zkServer.sh status
/opt/zookeeper/zookeeper-3/bin/zkServer.sh status

Deploy pseudo-cluster kafka

Create directory upload package

mkdir /opt/kafka
cd /opt/kafka
#上传 kafka_2.13-2.8.2.tgz
tar zxf kafka_2.13-2.8.2.tgz
mv kafka_2.13-2.8.2 kafka-1

Edit the kafka configuration file

vim kafka-1/config/server.properties

The main modifications are as follows

broker.id=0 # 节点1
listeners=PLAINTEXT://127.0.0.1:9092  # 这个默认配置没有新加
zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183   # 这个默认配置有，注释原配置新增
log.dirs=/opt/kafka/kafka-1/kafka-logs   # 这个默认配置有，注释原配置新增

Copy kafka-2, 3

cp -r kafka-1 kafka-2
cp -r kafka-1 kafka-3

Placement kafka-2

vim kafka-2/config/server.properties

broker.id=1 # 节点2
listeners=PLAINTEXT://127.0.0.1:9093
zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
log.dirs=/opt/kafka/kafka-2/kafka-logs

Placement kafka-3

vim kafka-3/config/server.properties

broker.id=2 # 节点3
listeners=PLAINTEXT://127.0.0.1:9094
zookeeper.connect=127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
log.dirs=/opt/kafka/kafka-3/kafka-logs

start kafka

# 启动kafka
/opt/kafka/kafka-1/bin/kafka-server-start.sh -daemon /opt/kafka/kafka-1/config/server.properties
/opt/kafka/kafka-2/bin/kafka-server-start.sh -daemon /opt/kafka/kafka-2/config/server.properties
/opt/kafka/kafka-3/bin/kafka-server-start.sh -daemon /opt/kafka/kafka-3/config/server.properties

The following commands create and check whether the operation is normal

# 创建名为test的topic
/opt/kafka/kafka-3/bin/kafka-topics.sh --create --zookeeper 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183 --replication-factor 3 --partitions 3 --topic test
# 列出已创建的topic
/opt/kafka/kafka-3/bin/kafka-topics.sh --list --zookeeper 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183
# 模拟生产者
/opt/kafka/kafka-3/bin/kafka-console-producer.sh --broker-list 172.25.25.3:9092,172.25.25.4:9092,172.25.25.5:9092 --topic test
# 模拟消费者
/opt/kafka/kafka-3/bin/kafka-console-consumer.sh --bootstrap-server 172.25.25.3:9092,172.25.25.4:9092,172.25.25.5:9092 --from-beginning --topic test
# 查看指定主题
/opt/kafka/kafka-3/bin/kafka-topics.sh --describe --zookeeper 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183 --topic test