Three Kafka Monitoring Tools

1. Overview

      Kafka is used by many teams within Yahoo, and the media team uses it for real-time analysis pipelines, which can handle peak bandwidths up to 20Gbps (compressed data).
　　In order to simplify the work of developers and service engineers to maintain Kafka clusters, a web-based tool called Kafka Manager was built, called Kafka Manager. This management tool makes it easy to find out which topics are distributed unevenly across the cluster, or if partitions are unevenly distributed across the cluster. It supports managing multiple clusters, selecting replicas, replica redistribution, and creating topics. At the same time, this management tool is also a very good tool to quickly browse the cluster.
　　The software is written in Scala language. At present (February 03, 2015) Yahoo has open sourced the Kafka Manager tool. This Kafka cluster management tool mainly supports the following functions:
　　1. Manage several different clusters;
　　2. Easily check the status of the cluster (topics, brokers, replica distribution, partition distribution);
　　3. Select replicas;
　　4. Generate partition assignments based on the current state of the cluster;
　　5. Reassign partitions.

2. Kafka Manager download and installation

     project address: https://github.com/yahoo/kafka-manager

     This project is easier to use than https://github.com/claudemamo/kafka-web-console, and the displayed information is more Rich, kafka-manager can itself be a cluster.

     However, kafka-manager also has no permission management function.

     Download:

1
git clone [email protected]:yahoo/kafka-manager.git
    After downloading, you can't do anything except the energy code. We need to compile and package the project. The software is written in Scala language, and the compiled and packaged It's troublesome, he relies on sbt. sbt is more difficult to install.



3. sbt installation

    1. Download sbt-0.13.11.

         My server is centos. Automatic installation was unsuccessful several times. I still choose to install manually.

         Please go to http://www.scala-sbt.org/download.html to download the latest version, my version is sbt-0.13.11

    2. Create a directory, extract the files to the created directory

1
2
$ sudo mkdir /opt/ scala/sbt
$ sudo tar zxvf sbt-0.13.11.tgz -C /opt/scala/
　 3. Create a script file for starting sbt

1
2
3
4
5
6
7
/*Select a location and create a script text file for starting sbt , such as a new text file named sbt under the /opt/scala/sbt/ directory */
$ cd /opt/scala/sbt/
$ vim sbt
/*Add
BT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
java $SBT_OPTS -jar /opt/scala/sbt/bin/sbt-launch.jar "$ to the sbt text file @"
Then press the esc key and enter: wq to save and exit. Note that the path in the red font can be an absolute path or a relative path, as long as the sbt-launch.jar file in the decompressed sbt file package can be correctly located* /
　　And modify the sbt file permissions

1
$ chmod u+x sbt
　　4. Configure the PATH environment variable to ensure that the sbt command can be used in the console

1
2
3
$ vim /etc/profile
/*After adding the following code at the end of the file, save and exit* /
export PATH=/opt/scala/sbt/:$PATH
　　

1
2
/*Make the configuration file take effect immediately*/
$ source /etc/profile
　  5. Test whether sbt is installed successfully
          When it is executed for the first time, some file packages will be downloaded. Then it can be used normally, to ensure that the Internet is connected, the download process is very slow. After the installation is successful, it will display the following

1
sbt sbt-version<br>[info] Set current project to sbt (in build file:/opt/scala/sbt/)<br>[info] 0.13.11
　　

Fourth, compile and package

1
2
cd kafka-manager
sbt clean The package generated by dist
　　will be under kafka-manager/target/universal. The generated package only needs the java environment to run, and sbt does not need to be installed on the deployed machine.

      If the packaging will be very slow, you need to be patient, and there may be failures in packaging. You can consider configuring a proxy.

Fourth, Kafka Manager deployment

    1. Pack the package, unzip it on the deployment machine, modify the configuration file, and you can run it -- unzip

1
unzip kafka-manager-1.0-SNAPSHOT.zip
　2. Modify conf/application.conf, Change kafka-manager.zkhosts to your own zookeeper server address

1
kafka-manager.zkhosts="192.168.1.237:2181"
　3. Start

1
2
cd kafka-manager-1.0-SNAPSHOT/bin
./kafka-manager -Dconfig. file=../conf/application.conf
　4. View help and run in the background

1
2
./kafka-manager -h
nohup ./kafka-manager -Dconfig.file=../conf/application.conf >/dev/null 2>&1 &
　Description: Normally, The play framework should automatically load the content in the conf/application.conf configuration, but it seems that this does not work, it needs to be specified explicitly.

Reference: https://github.com/yahoo/kafka-manager/issues/16

    5. The default http port is 9000, you can modify the value of http.port in the configuration file, or pass it through command line parameters:

1
./kafka -manager -Dhttp.port=9001


Five, sbt configuration proxy

    sbt configuration http proxy reference document: http://www.scala-sbt.org/0.12.1/docs/Detailed-Topics/Setup-Notes.html# http-proxy

    can set parameters through -D:

1
java -Dhttp.proxyHost=myproxy -Dhttp.proxyPort=8080 -Dhttp.proxyUser=username -Dhttp.proxyPassword=mypassword
　　You can also use the following method to set the SBT_OPTS Environment variables can be:

1
export SBT_OPTS="$SBT_OPTS -Dhttp.proxyHost=myproxy -Dhttp.proxyPort=myport"
　　Note: myproxy, do not include the http prefix or port number in this value.

      For example, if your proxy is http://localhost:8123, then it should be configured like this:

1
export SBT_OPTS="$SBT_OPTS -Dhttp.proxyHost=localhost -Dhttp.proxyPort=8123"

Reference: http://www.cnblogs.com /yuanermen/p/5453483.html

apache kafka monitoring series-KafkaOffsetMonitor
https://blog.csdn.net/lizhitao/article/details/27199863

Apache Kafka monitoring Kafka Web Console
https://www.iteblog.com/archives/ 1084.html