No need for zookeeper, kafka3.0 version cluster installation actual combat

1. Kafka cluster instance role planning

In kafka3.0, the zookeeper can be removed, and the kraft mechanism is used to realize the election of the controller master controller. So let's briefly understand the difference between Kafka 2.0 and 3.0 in this regard. In the above figure, black represents Broker (message broker service), and brown/blue represents Controller (cluster controller service)

Left picture (kafka2.0): All nodes in a cluster are Brokers. Kafka elects a Controller from the three Brokers. The controller saves the cluster metadata information (such as topic classification, consumption progress, etc.) to zookeeper. Used for distributed interaction between nodes in the cluster.
Right picture (kafka3.0): Suppose a cluster has four Brokers, three of them are designated as Conreoller roles (blue), one Controller is elected from the three Controllers as the main controller (brown), and the other two spare. zookeeper is no longer needed! Relevant metadata information exists in the form of kafka logs (ie: in the form of message queue messages).

After understanding the right picture above, it is not difficult for us to understand that before building a kafka 3.0 cluster, we need to do a good job in planning the role of kafka instances. (Four Brokers, three need to be designated as Controllers through active configuration, and Controllers need an odd number, which is the same as zk)

hostname	ip	Role	`node.id`
zimug1	192.168.1.111	broker,controller	1
zimug2	192.168.1.112	broker,controller	2
zimug3	192.168.1.113	broker,controller	3
zimug4	192.168.1.113	broker	4

2. Preparations

Create a new directory under the kafka user (new kafka user, do not use the root user) as the kafka3 installation directory, and use wget to download a version 3.10 installation package.

$mkdir kafka3-setup;
$ cd kafka3-setup/;
$ wget https://dlcdn.apache.org/kafka/3.1.0/kafka_2.13-3.1.0.tgz

复制代码

In addition, kafka3.0 no longer supports JDK8. It is recommended to install JDK11 or JDK17 and install it in advance.
新建1个目录用于保存kafka3的持久化日志数据mkdir -p /home/kafka/data/kafka3;，并保证安装kafka的用户具有该目录的读写权限。
（这里需要使用root用户）所有安装kafka3服务器实例防火墙开放9092、9093端口，使用该端口作为controller之间的通信端口。该端口的作用与zk的2181端口类似。
下载完成安装包之后，解压到/home/kafka目录下。也可以修改-C参数自定义解压路径，如果自定义路径，注意路径下的新建的kafka用户的操作权限。

tar -xzvf ./kafka_2.13-3.1.0.tgz -C /home/kafka

复制代码

三、修改Kraft协议配置文件

在kafka3.0版本中，使用Kraft协议代替zookeeper进行集群的Controller选举，所以要针对它进行配置，所以要针对它进行配置，配置文件在kraft目录下，这与kafka2.0版本依赖zookeeper安装方式的配置文件是不同的。

vim /home/kafka/kafka_2.13-3.1.0/config/kraft/server.properties

复制代码

具体的配置参数如下：

node.id=1
process.roles=broker,controller
listeners=PLAINTEXT://zimug1:9092,CONTROLLER://zimug1:9093
advertised.listeners = PLAINTEXT://:9092
controller.quorum.voters=1@zimug1:9093,2@zimug2:9093,3@zimug3:9093
log.dirs=/home/kafka/data/kafka3

复制代码

**node.id**：这将作为集群中的节点 ID,唯一标识，按照我们事先规划好的（上文），在不同的服务器上这个值不同。其实就是kafka2.0中的broker.id,只是在3.0版本中kafka实例不再只担任broker角色，也有可能是controller角色，所以改名叫做node节点。
「process.roles「：一个节点可以充当」broker」或「controller」或「两者兼而有之」。按照我们事先规划好的（上文），在不同的服务器上这个值不同。多个角色用逗号分开。
「listeners」：broker 将使用 9092 端口，而 kraft controller控制器将使用 9093端口。
「advertised.listeners」:这里指定kafka通过代理暴漏的地址，如果都是局域网使用，就配置PLAINTEXT://:9092即可。
「controller.quorum.voters「：这个配置用于指定」**controller主控」**选举的投票节点，所有process.roles包含controller角色的规划节点都要参与，即：zimug1、zimug2、zimug3。其配置格式为:node.id1@host1:9093,node.id2@host2:9093
「log.dirs」：kafka 将存储数据的日志目录，在准备工作中创建好的目录。

所有kafka节点都要按照上文中的节点规划进行配置，完成config/kraft/server.properties配置文件的修改。

三、格式化存储目录

生成一个唯一的集群ID（在一台kafka服务器上执行一次即可），这一个步骤是在安装kafka2.0版本的时候不存在的。

$ /home/kafka/kafka_2.13-3.1.0/bin/kafka-storage.sh random-uuid
SzIhECn-QbCLzIuNxk1A2A

复制代码

使用生成的集群ID+配置文件格式化存储目录log.dirs,所以这一步确认配置及路径确实存在，并且kafka用户有访问权限（检查准备工作是否做对）。「每一台主机服务器都要执行这个命令」

 /home/kafka/kafka_2.13-3.1.0/bin/kafka-storage.sh format \
-t SzIhECn-QbCLzIuNxk1A2A \
-c /home/kafka/kafka_2.13-3.1.0/config/kraft/server.properties

复制代码

格式化操作完成之后，你会发现在我们定义的log.dirs目录下多出一个meta.properties文件。meta.properties文件中存储了当前的kafka节点的id(node.id)，当前节点属于哪个集群（cluster.id）

$ cat /home/kafka/data/kafka3/meta.properties
#
#Tue Apr 12 07:39:07 CST 2022
node.id=1
version=1
cluster.id=SzIhECn-QbCLzIuNxk1A2A

复制代码

四启动集群，完成基础测试

zimug1 zimug2 zimug3是三台应用服务器的主机名称(参考上文中的角色规划)，在linux的/etc/hosts主机名与ip进行关系映射。将下面的命令集合保存为一个shell脚本，并赋予执行权限。执行该脚本即可启动kafka集群所有的节点，使用该脚本前提是：你已经实现了集群各节点之间的ssh免密登录。

#!/bin/bash
kafkaServers='zimug1 zimug2 zimug3'
#启动所有的kafka
for kafka in $kafkaServers
do
    ssh -T $kafka <<EOF
    nohup /home/kafka/kafka_2.13-3.1.0/bin/kafka-server-start.sh /home/kafka/kafka_2.13-3.1.0/config/kraft/server.properties 1>/dev/null 2>&1 &
EOF
echo 从节点 $kafka 启动kafka3.0...[ done ]
sleep 5
done

复制代码

If your installation path is different from mine, you /home/kafka/kafka_2.13-3.1.0need to modify it according to your own situation.

5. One-click stop cluster script

The script to stop each node of the kafka cluster with one click is the same as the usage method and principle of the startup script.

#!/bin/bash
kafkaServers='zimug1 zimug2 zimug3'
#停止所有的kafka
for kafka in $kafkaServers
do
    ssh -T $kafka <<EOF

    cd /home/kafka/kafka_2.13-3.1.0
    bin/kafka-server-stop.sh
EOF
echo 从节点 $kafka 停止kafka...[ done ]
sleep 5
done

复制代码

Code text is not easy, if you find it helpful, please help click to watch or share, without your support I may not be able to persevere! Welcome to the official account: Antetokounmpo, reply 003 to present the PDF version of the author's column "The Way of Docker Cultivation". Antetokounmpo Blog : zimug.com