Kafka monitoring and JMX

JMX

JMX (Java Management Extensions, or Java Management Extensions) is a framework for implanting management functions for applications, devices, systems, etc. JMX can span a series of heterogeneous operating system platforms, system architectures and network transmission protocols, and flexibly develop seamlessly integrated system, network and service management applications.

In layman's terms, with it, you can monitor the basic information and running status of Java programs.

Kafka opens JMX configuration

Windows [modify kafka-server-start.bat file, add JMX port after setting heap memory]

@echo off
rem Licensed to the Apache Software Foundation (ASF) under one or more
rem contributor license agreements.  See the NOTICE file distributed with
rem this work for additional information regarding copyright ownership.
rem The ASF licenses this file to You under the Apache License, Version 2.0
rem (the "License"); you may not use this file except in compliance with
rem the License.  You may obtain a copy of the License at
rem
rem     http://www.apache.org/licenses/LICENSE-2.0
rem
rem Unless required by applicable law or agreed to in writing, software
rem distributed under the License is distributed on an "AS IS" BASIS,
rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
rem See the License for the specific language governing permissions and
rem limitations under the License.

IF [%1] EQU [] (
    echo USAGE: %0 server.properties
    EXIT /B 1
)

SetLocal
IF ["%KAFKA_LOG4J_OPTS%"] EQU [""] (
    set KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:%~dp0../../config/log4j.properties
)
IF ["%KAFKA_HEAP_OPTS%"] EQU [""] (
    rem detect OS architecture
    wmic os get osarchitecture | find /i "32-bit" >nul 2>&1
    IF NOT ERRORLEVEL 1 (
        rem 32-bit OS
        set KAFKA_HEAP_OPTS=-Xmx512M -Xms512M
    ) ELSE (
        rem 64-bit OS
        set KAFKA_HEAP_OPTS=-Xmx1G -Xms1G
    )
    set JMX_PORT="9999"
)
"%~dp0kafka-run-class.bat" kafka.Kafka %*
EndLocal

Linux [modify kafka-server-start.sh file, configure JMX after setting heap memory]

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

if [ $# -lt 1 ];
then
    echo "USAGE: $0 [-daemon] server.properties [--override property=value]*"
    exit 1
fi
base_dir=$(dirname $0)

if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export JMX_PORT="9999"
fi

EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc ' }

COMMAND=$1
case $COMMAND in
  -daemon)
    EXTRA_ARGS="-daemon "$EXTRA_ARGS
    shift
    ;;
  *)
    ;;
esac

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"

So how can externally perceive Kafka's monitoring indicators?

JConsole

jconsole is a monitoring tool that comes with JDK.

Open, enter the address and JMX port

Homepage:

Some summary

There are also important MBeans

Java connects to JMX applications

Take the Kafka information above as an example

public static void jmx()throws Exception{
    JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi");
    JMXConnector jmxc = JMXConnectorFactory.connect(url);
    MBeanServerConnection connection = jmxc.getMBeanServerConnection();

    System.out.println("=========Domains=========");
    String[] domains = connection.getDomains();
    for (String d : domains) {
        System.out.println(d);
    }

    System.out.println("=========MBeans=========");
    System.out.println(connection.getMBeanCount());


    System.out.println("=========Invoke=========");
    ObjectName mBeanName = new ObjectName("kafka.log:type=Log,name=Size,topic=my-topic,partition=0");
    // 获取值
    Object value = connection.getAttribute(mBeanName, "Value");
    System.out.println(value);
    // 执行MBean的方法
    Object invoke = connection.invoke(mBeanName, "objectName", null, null);
    System.out.println(invoke);


    System.out.println("=========MBean Info=========");
    mBeanName = new ObjectName("kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec");
    MBeanInfo info = connection.getMBeanInfo(mBeanName);
    System.out.println("ClassName："+info.getClassName());
    for(MBeanAttributeInfo attr : info.getAttributes()){
        System.out.println("属性：" + attr.getName() + "，类型：" + attr.getType() + "，值：" + connection.getAttribute(mBeanName, attr.getName()));

    }
    for(MBeanOperationInfo op : info.getOperations()){
        System.out.println("操作：" + op.getName());
    }

    jmxc.close();
}

Output:

=========Domains=========
java.util.logging
kafka.utils
kafka.controller
java.nio
kafka.network
JMImplementation
kafka.log
kafka.coordinator.group
java.lang
com.sun.management
kafka.server
kafka.cluster
kafka
kafka.coordinator.transaction
=========MBeans=========
1098
=========Invoke=========
24997
kafka.log:type=Log,name=Size,topic=my-topic,partition=0
=========MBean Info=========
ClassName：com.yammer.metrics.reporting.JmxReporter$Meter
属性：Count，类型：long，值：0
属性：EventType，类型：java.lang.String，值：bytes
属性：RateUnit，类型：java.util.concurrent.TimeUnit，值：SECONDS
Property: MeanRate, Type: double, Value: 0.0
Property: OneMinuteRate, Type: double, Value: 0.0
Property: FiveMinuteRate, Type: double, Value: 0.0
Property: FifteenMinuteRate, Type: double, Value: 0.0
Operation: objectName

Explanation:

1. Output all current Domain information.

2. It can also output the total number of MBeans.

3. The output of monitoring information is what we really need: the value of ObjectName corresponds to JConsole

 

4. There are two pieces of information under each specific Object: attributes and operations (methods), it is easy to understand it

 

More ways to explore on your own