Kafka3.0.0 version - cluster deployment (linux environment-centos7)

1. Three server information

three servers

name of server	server ip
centos7 virtual machine 1	192.168.136.27
centos7 virtual machine 2	192.168.136.28
centos7 virtual machine 3	192.168.136.29

Two and three services need to install Zookeeper respectively ( Kafka needs Zookeeper support )

Zookeeper cluster installation refers to this blog post link: https://wwwxz.blog.csdn.net/article/details/129280956?spm=1001.2014.3001.5502

3. Kafka3.0.0 official website download

3.1. Official website download address

Official website download address: http://kafka.apache.org/downloads.html

3.2. Download steps

The download steps are as follows:

4. Kafka3.0.0 cluster deployment

4.1, centos7 virtual machine 1 (192.168.136.27) server installation steps

4.1.1. Unzip and install (the root user used for all the following demonstration operations lz)

Upload the installation package to the home directory
Create a module folder in the opt directory (if not created)
```
[root@localhost home]# mkdir /opt/module
```
Enter the home directory, unzip the installation package to the opt/module directory
```
[root@localhost home]# tar -zxvf kafka_2.12-3.0.0.tgz -C /opt/module/
```

Enter the opt/module directory, modify the decompressed kafka_2.12-3.0.0 name to kafka-3.0.0

[root@localhost home]# cd /opt/module/
[root@localhost module]# mv kafka_2.12-3.0.0/ kafka-3.0.0

insert image description here

4.1.2. Create a datas file in the Kafka root directory

Create a datas folder in the /opt/module/kafka-3.0.0 directory for the log storage directory
```
[root@localhost kafka-3.0.0]# mkdir datas
```

4.1.3. Create /kafka-3.0.0 folder in the Zookeeper root directory

Create a kafka-3.0.0 folder in the /opt/module/zookeeper-3.5.7 directory for easy management of kafka

[root@localhost kafka-3.0.0]# cd /opt/module/zookeeper-3.5.7
[root@localhost zookeeper-3.5.7]# mkdir kafka-3.0.0

insert image description here

4.1.4. Modify the server.properties configuration file

Go to the /opt/module/kafka-3.0.0/config directory and modify the server.properties configuration file
```
[root@localhost module]# cd kafka-3.0.0/config/
[root@localhost config]# vim server.properties
```

Modify the following 4 places in the server.properties configuration file

#broker 的全局唯一编号，不能重复，只能是数字。
broker.id=0

insert image description here

#放开端口
listeners=PLAINTEXT://192.168.136.27:9092
advertised.listeners=PLAINTEXT://192.168.136.27:9092

insert image description here

#kafka 运行日志(数据)存放的路径，路径不需要提前创建，kafka 自动帮你创建，可以
配置多个磁盘路径，路径与路径之间可以用"，"分隔
log.dirs=/opt/module/kafka-3.0.0/datas

insert image description here

#配置连接 Zookeeper 集群地址（在 zk 根目录下创建/kafka，方便管理）
zookeeper.connect=192.168.136.27:2181,192.168.136.28:2181,192.168.136.29:2181/kafka-3.0.0

insert image description here

4.1.5. Configure kafka environment variables

Add the kafka environment variable configuration in the /etc/profile.d/my_env.sh file, and add the following content:
```
#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka-3.0.0
export PATH=$PATH:$KAFKA_HOME/bin
```

Refresh environment variables

[root@localhost zookeeper-3.5.7]# source /etc/profile

insert image description here

4.1.6. Open the port 9092 to be used in the firewall, as follows:

Open the firewall, open port 9092, and close the firewall again as shown in the figure below.

#查看防火墙的状态命令
systemctl status firewalld
#打开防火墙的状态命令
sudo systemctl start firewalld
#开放2181、2888和3888端口
firewall-cmd --permanent --zone=public --add-port=2181/tcp
firewall-cmd --permanent --zone=public --add-port=2888/tcp
firewall-cmd --permanent --zone=public --add-port=3888/tcp
#重新加载
firewall-cmd --reload
#关闭防火墙的状态命令
sudo systemctl stop firewalld

insert image description here

4.1.7 Interpretation of server.properties configuration file parameters

Interpretation of parameters

#broker 的全局唯一编号，不能重复，只能是数字。
broker.id=0
#处理网络请求的线程数量
num.network.threads=3
#用来处理磁盘 IO 的线程数量
num.io.threads=8
#发送套接字的缓冲区大小
socket.send.buffer.bytes=102400
#接收套接字的缓冲区大小
socket.receive.buffer.bytes=102400
#请求套接字的缓冲区大小
socket.request.max.bytes=104857600
#kafka 运行日志(数据)存放的路径，路径不需要提前创建，kafka 自动帮你创建，可以
配置多个磁盘路径，路径与路径之间可以用"，"分隔
log.dirs=/opt/module/kafka/datas
#topic 在当前 broker 上的分区个数
num.partitions=1
#用来恢复和清理 data 下数据的线程数量
num.recovery.threads.per.data.dir=1
# 每个 topic 创建时的副本数，默认时 1 个副本
offsets.topic.replication.factor=1
#segment 文件保留的最长时间，超时将被删除
log.retention.hours=168
#每个 segment 文件的大小，默认最大 1G
log.segment.bytes=1073741824

4.2, centos7 virtual machine 2 (192.168.136.28) server installation steps

4.2.1. Create a module folder in the opt directory (if not created)

Create the module folder in the opt directory and the same location as the decompressed installation package of kafka-3.0.0 in the centos7 virtual machine 1 (192.168.136.27) server.
```
[root@localhost home]# mkdir /opt/module
```

4.2.2. Copy the configured kafka-3.0.0 to centos7 virtual machine 2 (192.168.136.28) server

scp basic syntax

scp  -r   $pdir/$fname       $user@$host:$pdir/$fname
命令  递归 要拷贝的文件路径/名称 目的地用户@主机:目的地路径/名称

Copy the configured kafka-3.0.0 on the centos7 virtual machine 1 (192.168.136.27) server to the centos7 virtual machine 2 (192.168.136.28) server
```
[root@localhost home]# scp -r /opt/module/kafka-3.0.0/ root@192.168.136.28:/opt/module/
```

4.2.3. Copy the configured environment variable file to the centos7 virtual machine 2 (192.168.136.28) server and refresh the environment variable

scp basic syntax

scp  -r   $pdir/$fname       $user@$host:$pdir/$fname
命令  递归 要拷贝的文件路径/名称 目的地用户@主机:目的地路径/名称

Copy the configured environment variable file on the centos7 virtual machine 1 (192.168.136.27) server to the centos7 virtual machine 2 (192.168.136.28) server
```
[root@localhost home]# scp -r /etc/profile.d/my_env.sh root@192.168.136.28:/etc/profile.d/my_env.sh
```
Refresh environment variables on centos7 virtual machine 2 (192.168.136.28) server
```
[root@localhost zookeeper-3.5.7]# source /etc/profile
```

4.2.4. Modify the content in the configuration file /opt/module/kafka-3.0.0/config/server.properties

Modify broker.id=1 in the configuration file /opt/module/kafka-3.0.0/config/server.properties on the centos7 virtual machine 2 (192.168.136.28) server ( Note: broker.id must not be repeated, the entire cluster only in
```
broker.id=1
```
Modify the server ip specified by the port in the configuration file /opt/module/kafka-3.0.0/config/server.properties on the centos7 virtual machine 2 (192.168.136.28) server.

4.2.5. Create /kafka-3.0.0 folder in the Zookeeper root directory

Create a kafka-3.0.0 folder in the /opt/module/zookeeper-3.5.7 directory for easy management of kafka

[root@localhost kafka-3.0.0]# cd /opt/module/zookeeper-3.5.7
[root@localhost zookeeper-3.5.7]# mkdir kafka-3.0.0

insert image description here

4.2.6. Open the port 9092 to be used in the firewall, as follows:

Open the firewall, open port 9092, and close the firewall again as shown in the figure below.

#查看防火墙的状态命令
systemctl status firewalld
#打开防火墙的状态命令
sudo systemctl start firewalld
#开放2181、2888和3888端口
firewall-cmd --permanent --zone=public --add-port=2181/tcp
firewall-cmd --permanent --zone=public --add-port=2888/tcp
firewall-cmd --permanent --zone=public --add-port=3888/tcp
#重新加载
firewall-cmd --reload
#关闭防火墙的状态命令
sudo systemctl stop firewalld

insert image description here

4.3, centos7 virtual machine 3 (192.168.136.29) server installation steps

4.3.1. Create a module folder in the opt directory (if not created)

Create the module folder in the opt directory and the same location as the decompressed installation package of kafka-3.0.0 in the centos7 virtual machine 1 (192.168.136.27) server.
```
[root@localhost home]# mkdir /opt/module
```

4.3.2. Copy the configured kafka-3.0.0 to centos7 virtual machine 3 (192.168.136.29) server

scp basic syntax

scp  -r   $pdir/$fname       $user@$host:$pdir/$fname
命令  递归 要拷贝的文件路径/名称 目的地用户@主机:目的地路径/名称

Copy the configured kafka-3.0.0 on the centos7 virtual machine 1 (192.168.136.27) server to the centos7 virtual machine 3 (192.168.136.29) server
```
[root@localhost home]# scp -r /opt/module/kafka-3.0.0/ root@192.168.136.29:/opt/module/
```

4.3.3. Copy the configured environment variable file to the centos7 virtual machine 3 (192.168.136.29) server and refresh the environment variable

scp basic syntax

scp  -r   $pdir/$fname       $user@$host:$pdir/$fname
命令  递归 要拷贝的文件路径/名称 目的地用户@主机:目的地路径/名称

Copy the configured environment variable file on the centos7 virtual machine 1 (192.168.136.27) server to the centos7 virtual machine 3 (192.168.136.29) server
```
[root@localhost home]# scp -r /etc/profile.d/my_env.sh root@192.168.136.29:/etc/profile.d/my_env.sh
```
Refresh environment variables on centos7 virtual machine 3 (192.168.136.29) server
```
[root@localhost ~]# source /etc/profile
```

4.3.4. Modify the content in the configuration file /opt/module/kafka-3.0.0/config/server.properties

Modify broker.id=2 in the configuration file /opt/module/kafka-3.0.0/config/server.properties on the centos7 virtual machine 3 (192.168.136.29) server ( Note: broker.id must not be repeated, the entire cluster only in
```
broker.id=2
```
Modify the server ip specified by the port in the configuration file /opt/module/kafka-3.0.0/config/server.properties on the centos7 virtual machine 3 (192.168.136.29) server.

4.2.5. Create /kafka-3.0.0 folder in the Zookeeper root directory

Create a kafka-3.0.0 folder in the /opt/module/zookeeper-3.5.7 directory for easy management of kafka

[root@localhost kafka-3.0.0]# cd /opt/module/zookeeper-3.5.7
[root@localhost zookeeper-3.5.7]# mkdir kafka-3.0.0

insert image description here

4.3.6. Open the port 9092 to be used in the firewall, as follows:

Open the firewall, open port 9092, and close the firewall again as shown in the figure below.

#查看防火墙的状态命令
systemctl status firewalld
#打开防火墙的状态命令
sudo systemctl start firewalld
#开放2181、2888和3888端口
firewall-cmd --permanent --zone=public --add-port=2181/tcp
firewall-cmd --permanent --zone=public --add-port=2888/tcp
firewall-cmd --permanent --zone=public --add-port=3888/tcp
#重新加载
firewall-cmd --reload
#关闭防火墙的状态命令
sudo systemctl stop firewalld

insert image description here

4.4. Cluster operation

4.4.1. First, start the Zookeeper service of the three servers respectively

Start the Zookeeper service of the three servers
```
[root@localhost home]# sh zk.sh start
```

4.4.2. Start the Kafka services of the three servers respectively

Start the Kafka service of centos7 virtual machine 1 (192.168.136.27) server

[root@localhost kafka-3.0.0]# cd /opt/module/kafka-3.0.0/
[root@localhost kafka-3.0.0]# bin/kafka-server-start.sh -daemon config/server.properties

insert image description here

Start the Kafka service of the centos7 virtual machine 2 (192.168.136.28) server

[root@localhost kafka-3.0.0]# cd /opt/module/kafka-3.0.0/
[root@localhost kafka-3.0.0]# bin/kafka-server-start.sh -daemon config/server.properties

insert image description here

Start the Kafka service of the centos7 virtual machine 3 (192.168.136.29) server

[root@localhost kafka-3.0.0]# cd /opt/module/kafka-3.0.0/
[root@localhost kafka-3.0.0]# bin/kafka-server-start.sh -daemon config/server.properties

insert image description here

4.4.3. Stop the Kafka services of the three servers respectively

Stop the Kafka service of centos7 virtual machine 1 (192.168.136.27) server

[root@localhost kafka-3.0.0]# cd /opt/module/kafka-3.0.0/
[root@localhost kafka-3.0.0]# bin/kafka-server-stop.sh

insert image description here

Stop the Kafka service of the centos7 virtual machine 2 (192.168.136.28) server

[root@localhost kafka-3.0.0]# cd /opt/module/kafka-3.0.0/
[root@localhost kafka-3.0.0]# bin/kafka-server-stop.sh

insert image description here

Stop the Kafka service of the centos7 virtual machine 3 (192.168.136.29) server

[root@localhost kafka-3.0.0]# cd /opt/module/kafka-3.0.0/
[root@localhost kafka-3.0.0]# bin/kafka-server-stop.sh

insert image description here