Use zabbix to monitor tomcat (including jvm monitoring)

1 Introduction:

Our monitoring system is zabbix. Recently, we have just finished monitoring of jvm and tomcat. We need to complain about the following:

The jvm and tomcat monitoring templates officially provided by zabbix have pits in the item writing format, resulting in a lot of content that cannot be monitored.
In the zabbix monitoring java articles written by netizens, 99% only have environment construction, without zabbix item key value, so environmental monitoring is on, and item key value does not know how to write.

This article monitors the jvm content as follows:

内存使用状态：堆内存（Heap memory）和非堆内存（No Heap memory），包括已用值、最大值、已提交；
堆内存内存池：新生代（eden space），survivor space,老年代（old gen）的内存使用状态；
非堆内存内存池：代码缓存（Code cache），元空间（meta space），压缩类空间（compressed class space）；
类加载：加载总数，已加载，已卸载。
Java线程：总开启线程，活动线程，线程峰值。

This article monitors tomcat content as follows:

Tomcat请求数：包括每秒请求数，每秒出错数；
Tomcat网络流量统计：包括进流量统计，出流量统计；
Tomcat线程：包括最大线程数，当前线程数，当前繁忙线程数。

I did not add garbage collection (gc) monitoring in zabbix, but added gc daily input in catalina configuration to analyze the development. The setting method is as follows.
CATALINA_OPTS=”-XX:ParallelGCThreads=4 -XX:+PrintGCDetails -Xloggc:Log storage path” The
monitoring effect is as follows:
Use zabbix to monitor tomcat (including jvm monitoring)

2. Monitoring environment construction

Environment introduction:

[root@tomcat-01 ~]# /usr/local/tomcat/bin/version.sh
Server version: Apache Tomcat/8.0.23
Server built: May 19 2015 14:58:38 UTC
Server number: 8.0.23.0
OS Name: Linux
OS Version: 2.6.32-573.22.1.el6.x86_64
Architecture: amd64
JVM Version: 1.8.0_65-b17
JVM Vendor: Oracle Corporation

2.1. Server configuration
1. Install java jdk environment on Zabbix server and enable javaGateway to support java monitoring.
The method of enabling javaGateway is as follows:
Zabbix is installed through the rpm package : only the zabbix -java-gateway package needs to be installed.
Zabbix is installed by compiling : when compiling, you need to add -enable-java to support jmx monitoring, if you haven't added it before, you need to recompile.
2. Modify zabbix-java-gateway configuration file

[root@zabbix ~]# vim /etc/zabbix/zabbix_java_gateway.conf
LISTEN_IP="192.168.10.3"
LISTEN_PORT=10052
PID_FILE="/var/run/zabbix/zabbix_java.pid"
START_POLLERS=5

修改zabbix-server配置文件
[root@zabbix ~]# vim /etc/zabbix/zabbix_server.conf
JavaGateway=192.168.10.3
JavaGatewayPort=10052
StartJavaPollers=5

启动zabbix-java-gateway服务
[root@zabbix ~]# /etc/init.d/zabbix-java-gateway start

2.2. Tomcat server configuration
1. Download the catalina-jmx-remote.jar package to the lib directory under the tomcat installation directory

wget -O /usr/local/tomcat/lib/catalina-jmx-remote.jar http://archive.apache.org/dist/tomcat/tomcat-8/v8.0.23/bin/extras/catalina-jmx-remote.jar

2. Modify catalina.sh and add the following content

CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=8090
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false"

3. Restart tomcat and monitoring test
(1) Download cmdline-jmxclient-0.10.3.jar file, download address http://crawler.archive.org/cmdline-jmxclient/downloads.html
(2) execute the following command locally to view tomcat Heap memory information

[root@tomcat-01 ~]# java -jar /root/cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=Memory HeapMemoryUsage
11/04/2016 15:23:16 +0800 org.archive.jmx.Client HeapMemoryUsage:
committed: 2146959360
init: 2147483648
max: 2146959360
used: 407611808

3. Monitoring data collection

3.1. Heap memory
Tomcat local view heap memory information:

[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Memory HeapMemoryUsage
11/04/2016 15:36:58 +0800 org.archive.jmx.Client HeapMemoryUsage:
committed: 2145910784
init: 2147483648
max: 2145910784
used: 741540536

Zabbix monitors heap memory key value:

堆内存最大值：jmx["java.lang:type=Memory","HeapMemoryUsage.max"]
已用堆内存：jmx["java.lang:type=Memory","HeapMemoryUsage.used"]
已提交堆内存：jmx["java.lang:type=Memory","HeapMemoryUsage.committed"]

A complete zabbix item is filled in as follows, and different content can be filled in with different key values:

Use zabbix to monitor tomcat (including jvm monitoring)
3.2. Memory pool eden space:
Tomcat local view eden space:

java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=PS\ Eden\ Space Usage

Zabbix monitors the eden area key value:

最大空间：jmx["java.lang:type=MemoryPool,name=PS Eden Space",Usage.max]
已用空间：jmx["java.lang:type=MemoryPool,name=PS Eden Space",Usage.used]
提交空间：jmx["java.lang:type=MemoryPool,name=PS Eden Space",Usage.committed]

3.3. Memory pool survivor space:
Tomcat local view Survivor space area:

java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=PS\ Survivor\ Space Usage

zabbix monitors Survivor key value:

jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.committed]
jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.max]
jmx["java.lang:type=MemoryPool,name=PS Survivor Space",Usage.used]

3.4. Memory pool old gen:
Tomcat local view old gen area usage:

java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:name=PS\ Old\ Gen,type=MemoryPool Usage

zabbix monitors the old gen key value:

jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.committed]
jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.max]
jmx["java.lang:type=MemoryPool,name=PS Old Gen",Usage.used]

3.5. Non-heap memory:
Tomcat local view non-heap memory usage:

java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Memory NonHeapMemoryUsag1
zabbix监控非堆内存使用
1jmx["java.lang:type=Memory","NonHeapMemoryUsag.committed"]
jmx["java.lang:type=Memory","NonHeapMemoryUsag.used"]

3.6. Memory pool meta space:
Tomcat local view meta space area usage:

java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=Metaspace Usage

Zabbix monitors the key value of the mete space area

jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.committed]
jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.max]
jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.used]

3.7. Memory pool code cache:
Tomcat local view code cache area usage:

java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=Code\ Cache Usage

Zabbix monitors the use of code cache area:

jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.committed]
jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.max]
jmx["java.lang:type=MemoryPool,name=Code Cache",Usage.used]

3.8. Memory pool compressed class space:
Tomcat local view compressed class space area usage:

java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:type=MemoryPool,name=Compressed\ Class\ Space Usage

Zabbix monitors the use of key values in the compressed class space area:

jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.committed]
jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.max]
jmx["java.lang:type=MemoryPool,name=Compressed Class Space",Usage.used]

3.9. Class loading:
Tomcat local view class loading information:

加载总数： java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=ClassLoading TotalLoadedClassCoun
已加载： java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=ClassLoading LoadedClassCount
已卸载： java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=ClassLoading UnloadedClassCount

Zabbix monitoring class load key value:

加载总数： jmx["java.lang:type=ClassLoading","TotalLoadedClassCount"]
已加载： jmx["java.lang:type=ClassLoading","LoadedClassCount"]
已卸载： jmx["java.lang:type=ClassLoading","UnloadedClassCount"]

3.10.
java thread: tomcat local view java thread:

总开启线程： java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Threading TotalStartedThreadCount
活动线程： java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Threading PeakThreadCount
线程峰值： java -jar cmdline-jmxclient-0.10.3.jar controlRole:tomcat 127.0.0.1:8090 java.lang:type=Threading PeakThreadCount

Zabbix monitors java thread keys:

总开启线程： jmx["java.lang:type=Threading","TotalStartedThreadCount"]
活动线程： jmx["java.lang:type=Threading","ThreadCount"]
线程峰值： jmx["java.lang:type=Threading","PeakThreadCount"]

3.11. Tomcat thread:
view tomcat thread information locally:

最大线程：java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name=\"http-nio-8080\",type=ThreadPool maxThreads
当前线程：java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name=\"http-nio-8080\",type=ThreadPool currentThreadCount
繁忙线程：java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name=\"http-nio-8080\",type=ThreadPool currentThreadsBusy

Zabbix monitors tomcat thread keys:

最大线程：jmx["Catalina:type=ThreadPool,name=\"http-nio-8080\"",maxThreads]
当前线程：jmx["Catalina:type=ThreadPool,name=\"http-nio-8080\"",currentThreadCount]
繁忙线程 jmx["Catalina:type=ThreadPool,name=\"http-nio-8080\"",currentThreadsBusy]

3.12. Network traffic:
View the received bytes locally by Tomcat:

接收的字节：java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name=\"http-nio-8080\",type=GlobalRequestProcessor bytesReceived
发送的字节：java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 Catalina:name=\"http-nio-8080\",type=GlobalRequestProcessor bytesSent

Zabbix monitors tomcat accepts byte key values:

接收的字节：jmx["Catalina:type=GlobalRequestProcessor,name=\"http-nio-8080\"",bytesReceived]
发送的字节：jmx["Catalina:type=GlobalRequestProcessor,name=\"http-nio-8080\"",bytesSent]

3.13.
Tomcat request, error request: tomcat local view tomcat request number:

tomcat请求数： java -jar cmdline-jmxclient-0.10.3.jar - 192.168.10.46:8090 Catalina:name=\"http-nio-8080\",type=GlobalRequestProcessor requestCount
tomcat出错请求： java -jar cmdline-jmxclient-0.10.3.jar - 192.168.10.46:8090 Catalina:name=\"http-nio-8080\",type=GlobalRequestProcessor errorCount

Zabbix monitors the number of tomcat requests:

tomcat请求数： jmx["Catalina:type=GlobalRequestProcessor,name=\"http-nio-8080\"",requestCount]
tomcat出错请求：jmx["Catalina:type=GlobalRequestProcessor,name=\"http-nio-8080\"",errorCount]

4. Answers to questions

Recently, a netizen contacted me and said that after looking at my blog post to finish setting up the environment, I couldn't get the data. I helped to troubleshoot and found the problem, and hereby record it.
The netizen’s error is as follows:

# java -jar /root/cmdline-jmxclient-0.10.3.jar - 127.0.0.1:9080 java.lang:type=MemoryPool,name=PS\ Eden\ Space Usag
11/11/2016 10:03:37 +0800
org.archive.jmx.Client java.lang:name=PS Eden Space,type=MemoryPool is not a registered bean

4.1. Solutions

If you can't get the data using command line monitoring, then use jconsole to see if there is any data.
If jconsole has data, then look down, otherwise check your environment.
If jconsole has data, but there is no data on the command line, then there is a problem with the Object Name or properties of the Mbean, and netizens copy mine, but their local environment is different from mine, so the data cannot be obtained, and you can query your local There are two Mbean methods, namely graphics and command lines. Netizens report an error saying that there is a problem with their Eden Space space. Then I posted a method to view the Object Name and properties of the local memory pool.

(1) View through jconsole:
Use zabbix to monitor tomcat (including jvm monitoring)

(2) View through the command line: directly use the java -jar cmdline-jmxclient-0.10.3.jar – 127.0.0.1:8090 command to get all the Mbean information. If there is too much output, the output result will not be posted here. I get the monitoring information of all memory pools through the grep command as follows.


[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 | grep MemoryPool
java.lang:name=Compressed Class Space,type=MemoryPool
java.lang:name=Metaspace,type=MemoryPool
java.lang:name=PS Old Gen,type=MemoryPool
java.lang:name=PS Eden Space,type=MemoryPool
java.lang:name=PS Survivor Space,type=MemoryPool
java.lang:name=Code Cache,type=MemoryPool

Then I want to get all the attribute information of Eden Space as follows: (some special characters need to be escaped)

[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:name=PS\ Eden\ Space,type=MemoryPool
Attributes:
Usage: Usage (type=javax.management.openmbean.CompositeData)
PeakUsage: PeakUsage (type=javax.management.openmbean.CompositeData)
MemoryManagerNames: MemoryManagerNames (type=[Ljava.lang.String;)
UsageThreshold: UsageThreshold (type=long)
UsageThresholdExceeded: UsageThresholdExceeded (type=boolean)
UsageThresholdCount: UsageThresholdCount (type=long)
UsageThresholdSupported: UsageThresholdSupported (type=boolean)
CollectionUsageThreshold: CollectionUsageThreshold (type=long)
CollectionUsageThresholdExceeded: CollectionUsageThresholdExceeded (type=boolean)
CollectionUsageThresholdCount: CollectionUsageThresholdCount (type=long)
CollectionUsage: CollectionUsage (type=javax.management.openmbean.CompositeData)
CollectionUsageThresholdSupported: CollectionUsageThresholdSupported (type=boolean)
Valid: Valid (type=boolean)
Name: Name (type=java.lang.String)
Type: Type (type=java.lang.String)
ObjectName: ObjectName (type=javax.management.ObjectName)
Operations:
resetPeakUsage: resetPeakUsage
Parameters 0, return type=void

Then we are looking at the usage information of Eden Space. I saw the data has come out

[root@tomcat-01 ~]# java -jar cmdline-jmxclient-0.10.3.jar - 127.0.0.1:8090 java.lang:name=PS\ Eden\ Space,type=MemoryPool Usage
12/03/2016 08:33:58 +0800 org.archive.jmx.Client Usage:
committed: 712507392
init: 537395200
max: 712507392
used: 396006304

文章系作者原创投稿，作者：西门飞冰，一名90后it男，一直在北京工作，热爱运动，热爱冒险，热爱旅行。 原文：西门飞冰的博客-专注于Linux运维 » zabbix监控tomcat（包含jvm监控）。