Construction and use of Kafka visualization platform EFAK

Article directory

1. Introduction to EFAK visualization platform

The predecessor of EFAK is Kafka-eagle, and the new version of Kafka-eagle is called EFAK (Eagle For Apache Kafka).

EFAK is open source visualization and management software. It allows you to query, visualize, alert on and explore your metrics no matter where they are stored. In plain English, it gives you the tools to convert your Kafka cluster data into beautiful graphs and visualizations.

EFAK can directly query or dynamically read the message data of Topic in Kafka through KSQL.

EFAK official document address: https://www.kafka-eagle.org/articles/docs/introduce/what-is-kafka-eagle.html

EFAK depends on the JAVA environment.

2. Build the EFAK visualization platform

2.1. Install JDK environment

[root@kafka-1 ~]# tar xf jdk-8u211-linux-x64.tar.gz -C /data/
[root@kafka-1 ~]# vim /etc/profile
JAVA_HOME=/usr/local/jdk1.8.0_211
PATH=$JAVA_HOME/bin:$PATH
[root@kafka-1 ~]# source /etc/profile
[root@kafka-1 ~]# java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

2.2. Install the MySQL database

1.安装mysql服务
[root@mysql ~]# yum -y install mariadb-server

2.启动mysql
[root@mysql ~]# systemctl start mariadb

3.设置root用户密码
[root@mysql ~]# mysqladmin -uroot password '123456'

2.3. Download the EAK binary installation package and deploy it

1.下载安装包
[root@kafka-1 ~]# wget https://github.com/smartloli/kafka-eagle-bin/archive/v2.1.0.tar.gz

2.解压即可使用
[root@kafka-1 ~]# tar xf kafka-eagle-bin-2.1.0.tar.gz
[root@kafka-1 ~]# tar xf kafka-eagle-bin-2.1.0/efak-web-2.1.0-bin.tar.gz -C /data

3.修改目录名称
[root@kafka-1 ~]# mv /data/efak-web-2.1.0/ /data/kafka-efak
[root@kafka-1 ~]# ll /data/kafka-efak
总用量 0
drwxr-xr-x 2 root root 52 3月  21 09:51 bin
drwxr-xr-x 2 root root 75 3月  21 09:51 conf
drwxr-xr-x 2 root root  6 9月  13 2021 db
drwxr-xr-x 2 root root 23 3月  21 09:51 font
drwxr-xr-x 9 root root 91 2月  23 11:13 kms
drwxr-xr-x 2 root root  6 2月  19 19:45 logs

4.配置系统环境变量
[root@kafka-1 ~]# vim /etc/profile
KE_HOME=/data/kafka-efak
PATH=$KE_HOME/bin:$PATH
[root@kafka-1 ~]# source /etc/profile

2.4. Configure EFAK to connect to Zookeeper cluster

As long as EFAK can connect to ZOokeeper, it can obtain the information of Kafka cluster, and there is no need to configure and connect to Kafka.

[root@kafka-1 ~]# vim /data/kafka-efak/conf/system-config.properties 
efak.zk.cluster.alias=cluster1					#zk集群，我们只有一套kafka集群，连接了一个zk集群，因此只保留了clustrer1，如果我们希望通过这一个eagle平台管理多个kakfa集群，在这里可以以逗号形式隔开，添加多个zk集群
cluster1.zk.list=192.168.20.10:2181,192.168.20.11:2181,192.168.20.12:2181		#zk集群各节点的地址
#cluster2.zk.list=xdn10:2181,xdn11:2181,xdn12:2181			#没有其他kafka集群注释掉cluster2

#eagle平台的数据是要存储在数据库的，需要配置数据库连接信息
efak.driver=com.mysql.cj.jdbc.Driver
efak.url=jdbc:mysql://192.168.20.13:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
efak.username=root
efak.password=123456

2.5. Adjust the variable information in the Eagle startup file

KE_HOMEAdd the above and two variables in the Eagle startup file JAVA_HOME, otherwise an error will be reported at startup and the definition information of these two variables cannot be found.

[root@kafka-1 ~]# vim /data/kafka-efak/bin/ke.sh
KE_HOME=/data/kafka-efak
JAVA_HOME=/usr/local/jdk1.8.0_211

2.6. Start the EFAK visualization platform

[root@kafka-1 ~]# /data/kafka-efak/bin/ke.sh start

After the startup is successful, the login account and password of the platform will be prompted.

3. Use the EFAK visualization platform

3.1. Log in to the EFAK visualization platform

Access the machine's IP+8048 port, account number: admin, password: 123456.

3.2. EFAK dashboard display

On the home page of the system, you can see the number of Brokers, Topics, ZK clusters, and consumer groups in the Kafaka cluster.

You can also see the accumulation of messages in Topic below, and you can see that there are 5 accumulated messages in my-rc-topic.

Click the BScreen on the left to jump to the dynamic monitoring view.

3.3. Select a different Kafka cluster

In the upper right corner, you can switch between different Clusters to select different Kafka clusters. Cluster is the ZK cluster information defined in the configuration file.