Table of contents
3.2 Configuration modification
3.6 Add kafka cluster configuration
3.8 Topic and Broker information viewing
I. Overview
In order to simplify the work of developers and service engineers maintaining Kafka clusters, yahoo has built a web-based tool called Kafka Manager, called Kafka Manager (renamed cmak).
This management tool can easily find out which topics are distributed unevenly in the cluster, or the situation where partitions are unevenly distributed throughout the cluster.
It supports managing multiple clusters, selecting replicas, reassigning replicas, and creating topics. At the same time, this management tool is also a very good tool for quickly browsing the cluster, with the following functions:
- Manage multiple clusters
- Easily check cluster status (topics, consumers, offsets, brokers, replica distribution, partition distribution)
- run preferred copy selection
- Generate partition assignments, with an option to select which broker to use
- Run partition reallocation (based on generated allocation)
- Create a theme with optional theme configuration (0.8.1.1 has a different configuration than 0.8.2+)
- delete topic (supported only in 0.8.2+, remember to set Delete.topic.enable=true in broker config)
- Themes list now shows themes marked for deletion (only supported in 0.8.2+)
- Batch generate partition assignments for multiple topics with optional broker to use
- Run partition reassignments for multiple topics in batches
- Add a partition to an existing topic
- Update the configuration of an existing theme
- Optionally enable JMX polling for agent-level and topic-level metrics.
Optionally filter out consumers that don't have an id /owner/ & offset/ directory in zookeeper.
https://github.com/yahoo/CMAK
2. Environment
Ambari cluster version information
Note: The cmak environment requires JDK version 11
1、jdk
java version "jdk-11.0.18"
2、kafka集群信息
服务器：
192.168.2.152:6667
192.168.2.153:6667
软件：
kafka_2.0.0
zookeeper-3.4.63. Deploy kafka-manager
3.1 download
The cmak download address is:
https://github.com/yahoo/CMAK/releases/download/3.0.0.6/cmak-3.0.0.6.zip
3.2 Configuration modification
export JAVA_HOME=/hadoop/software/jdk-11.0.18Because the JDK1.8 environment variable has been added locally, I add the JDK environment variable in the startup script
Modify the application.conf configuration file in cmak:
kafka-manager.zkhosts="192.168.2.152:2181,192.168.2.153:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
cmak.zkhosts="192.168.2.152:2181,192.168.2.153:2181"
cmak.zkhosts=${?ZK_HOSTS}
##开启认证
basicAuthentication.enabled=true
basicAuthentication.username="admin"
basicAuthentication.username=${?KAFKA_MANAGER_USERNAME}
basicAuthentication.password="admin"
basicAuthentication.password=${?KAFKA_MANAGER_PASSWORD}
#kafka-manager.consumer.properties.file=${?CONSUMER_PROPERTIES_FILE}
kafka-manager.consumer.properties.file=/hadoop/software/cmak-3.0.0.6/conf/consumer.properties
添加conf/consumer.properties配置
security.protocol=SASL_PLAINTEXT
key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
添加 conf/jaas.conf 配置
Client{
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
principal="kafka/[email protected]"
serviceName="kafka"
doNotPrompt=true;
};
KafkaClient{
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
principal="kafka/[email protected]"
serviceName="kafka"
doNotPrompt=true;
};3.3 Kafka opens JMX
If you enable JMX rotation training, you must open the JMX port in the kafka startup script (all kafka nodes are open)
vim /usr/hdp/3.1.4.0-315/kafka/bin/kafka-server-start.sh
export JMX_PORT=9999
Then restart kafka and check the kafka-manager port
3.4 Start CMAK
Create startup script startCamk.sh
#！/bin/bash
echo '-------------------------------------------------------------------分界线'$(date +%F%t%T)> manager.out
# 配置kafka-manager元数据使用的zookeeper,此处必须是用export
export ZK_HOSTS=192.168.2.152:2181,192.168.2.153:2181,192.168.2.154:2181
# kafka-manager 路径
MANAGER_HOME=/hadoop/software/cmak-3.0.0.6
# 可执行文件路径
KAFKA_MANAGER=$MANAGER_HOME/bin/cmak
# 日志位置
APP_HOME=-Dapplication.home=$MANAGER_HOME
# 端口
HTTP_PORT=-Dhttp.port=9001
# SASL安全认证
JAAS_CONF=-Djava.security.auth.login.config=$MANAGER_HOME/conf/jaas.conf
KRB5_CONF=-Djava.security.krb5.conf=$MANAGER_HOME/conf/krb5.conf
nohup $KAFKA_MANAGER $JAAS_CONF $KRB5_CONF $APP_HOME $HTTP_PORT >manager.out 2>&1 &
echo "$!"
tailf manager.outYou can start it in the foreground first to see if there is an error
./bin/cmak -Djava.security.krb5.conf=./conf/krb5.conf -Djava.security.auth.login.config=./conf/jaas.conf -Dhttp.port=90013.5 Login
The account password (admin/admin) has been configured before, if it has not been changed, it is (admin/password)
3.6 Add kafka cluster configuration
When we open this interface for the first time, it is blank to configure and add clusters
Configuration: zookeeper address, the following namespace does not need to be added
The authentication mode is selected because kerberos is enabled. save save and then go to view
3.7 Questions
Failed to save configuration: Kafka manager added cluster error KeeperErrorCode
KeeperErrorCode = Unimplemented for /kafka-manager/mutex Try again.
Solution
Log in to the zookeeper client, manually create /kafka-manager/mutex/locks and /kafka-manager/mutex/leases nodes
[root@hdp103 ~]# zookeeper-client \
create /kafka-manager/mutex/locks ""
create /kafka-manager/mutex ""
create /kafka-manager/mutex/leases ""
3.8 Topic and Broker information viewing
On the Summary page, you can see zk connection information, kafka version information, Topic and number of Brokers.
View the topic status, you can view the Broker information topic status on the surface
Click List to view the Topics list
View the details of topic kangtest
3.9Topic creation
Other parameter configuration, default
topic list view topic kangll-manager-test created successfully
Command line view topic
Next, we understand the above settings based on a picture:
In a Kafka cluster in the above figure, there are two servers, each with 2 partitions. P0 and P3 may belong to the same topic, or they may be two different topics.
If both Partitons and Replication Factor are set to 2, the distribution of the topic in this case is the same as that shown in the Kafka cluster in the above figure. At this time, P0 and P3 are two partitions of the same topic. P1 and P2 are also two partitions of the same topic. One of Server1 and Server2 will act as the leader for read and write operations, and the other will be synchronized through replication.
If both Partitons and Replication Factor are set to 1, only one partition will be created on a server according to the algorithm, which can be one of P0~4 (the partitions are all newly created, instead of existing 4 first and then taking it from it) 1).