java application monitoring (3) - These command-line tools you have mastered yet

tags: java, troubleshooting, monitor,jvm

---

Summarized in one sentence: the original jdk that comes with command-line tool so easy to use, this article will detail.

1 Introduction

Monitoring of java applications, the most convenient is the direct use of ready-made tools jdk provided in the bin directory of the installed jdk, has provided a variety of command-line monitoring tools to monitor and diagnose problems on java application developers and operation and maintenance personnel, Thus, such tools are an important means of java application monitoring. But also as java developers need to master basic skills.

2 common monitoring command-line tool

In general, common command-line tools include jps, jinfo, jmap, jstack, jstat, these tools are in JAVA_HOME/bin/the directory, the following outline:

• jpsView java process ID
• jinfoCheck and adjust the parameters of the virtual machine
• jmapView Heap (heap) and the use of raw heap snapshot
• jstackView thread running state and generates a snapshot of the thread
• jstatShow the process of class loading, operating data memory, garbage collection and so on.

With these tools, you can basically understand the change in the state memory java application thread running status and other information, and then provide the basis for application monitoring and problem diagnosis. The following examples will combine the use of these tools will be explained in detail, the sample code used herein is java-monitor-exampleuploaded to my GitHub , address: https://github.com/mianshenglee.

3 Process Query Tooljps

3.1 jpsDescription

To monitor java application, the first step is to know which application process, what its operating parameters Yes. jpsIt is that you can process the query tool. Linux familiar with the students, probably all know the query process uses ps -ef|grep javasuch a command, jps is similar, but it does not use name lookup, but to find all java jdk processes that are currently running, and only find the current user of the Java process, rather than the current system in all processes.

3.2 jpsUse

As a command-line tool, you can -helpview the parameter Help is also available on the official document : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jps.htmlas follows:

[root@test bin]# jps -help
usage: jps [-help]
       jps [-q] [-mlvV] [<hostid>]

Definitions:
    <hostid>:      <hostname>[:<port>]

参数解释：
-q：只显示java进程的pid
-m：输出传递给main方法的参数，在嵌入式jvm上可能是null
-l：输出应用程序main class的完整package名 或者 应用程序的jar文件完整路径名
-v：输出传递给JVM的参数
复制代码

Examples of projects java-monitor-exampleup and running in a linux machine, use jpscan output the following information:

• Only the process ID

[root@test bin]# jps -q
13680
14214
复制代码

• The full name of the program output and JVM parameters

[root@test bin]# jps -lv
13680 java-monitor-example-0.0.1-SNAPSHOT.jar -Xms128m -Xmx128m -Dserver.port=8083
14289 sun.tools.jps.Jps -Denv.class.path=.:/opt/jdk8/lib:/opt/jdk8/jre/lib -Dapplication.home=/opt/jdk8
复制代码

The contents of the output in java-monitor-example-0.0.1-SNAPSHOT.jaris -lthe full name of output, -Xms128m -Xmx128m -Dserver.port=8083is passed to the JVM parameters.

• Use the command in a shell script and get java process ID as a variable use

JAVA_HOME="/opt/jdk8"
APP_MAINCLASS=java-monitor-example
#初始化psid变量（全局）
psid=0

#查看进程ID函数
checkpid() {
   javaps=`$JAVA_HOME/bin/jps -l | grep $APP_MAINCLASS`
 
   if [ -n "$javaps" ]; then
      psid=`echo $javaps | awk '{print $1}'`
   else
      psid=0
   fi
}

#调用函数后通过psid进行业务逻辑操作，如根据进程id杀进程
checkpid
echo "(pid=$psid)"
复制代码

The above script, more suitable for operation and maintenance personnel to open and close applications, automatically get java process ID, and whether to run (start) based on the ID determination program, or close the application ( kill -9).

4 configuration tooljinfo

4.1 jinfoDescription

Know the process ID belongs java application is the first step in the article "java application monitoring (2) -java secret command", the startup parameters already know there are a lot of java, java before monitoring applications need to understand clearly its What startup parameters Yes. Then you need to use jinfotools. jinfoJVM may output system parameters and application parameters JAVA. jinfo also be able to modify the adjustable parameters during part of running virtual machines, many operating parameters can not be adjusted, if "can not be changed" abnormal, indicating that can not be adjusted. But the official documentation states that this command may no longer be used in subsequent releases, current JDK8 can still be used.

4.2 jinfoUse

By -helpviewing parameters help is also available on jinfothe official documentation : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jinfo.htmlas follows:

[root@test bin]# jinfo -help
Usage:
    jinfo [option] <pid>
        (to connect to running process)

where <option> is one of:
    -flag <name>         to print the value of the named VM flag
    -flag [+|-]<name>    to enable or disable the named VM flag
    -flag <name>=<value> to set the named VM flag to the given value
    -flags               to print VM flags
    -sysprops            to print Java system properties
    <no option>          to print both of the above
    -h | -help           to print this help message

复制代码

Use jpsAfter obtaining the application process ID (PID example is 13,680), directly if jps <pid>all of the system parameters and JVM is output parameters, other parameters described helpalso said very clearly. Or a combination of the following sample code java-monitor-exampleto practice what:

• The initial value of the acquisition heap java applications

[root@test bin]# jinfo -flag InitialHeapSize 13680
-XX:InitialHeapSize=134217728
复制代码

• See all the JVM arguments

[root@test bin]# jinfo -flags 13680
Attaching to process ID 13680, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.51-b03
Non-default VM flags: -XX:CICompilerCount=2 -XX:InitialHeapSize=134217728 -XX:MaxHeapSize=134217728 -XX:MaxNewSize=44564480 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=44564480 -XX:OldSize=89653248 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseParallelGC
Command line:  -Xms128m -Xmx128m -Dserver.port=8083

复制代码

Visible, because we set the start -Xmsand -Xmxthat is the corresponding -XX:InitialHeapSizeand -XX:MaxHeapSizevalue. In addition, the parameters -Dserver.portare system parameters, jinfo -sysprops 13680you can view the system parameters.

5 heap viewerjmap

5.1 jmapDescription

After java application starts, it runs in the JVM, memory is where the need to focus on monitoring, jmapis one such tool that takes a snapshot of a running jvm heap, including the overall situation, the heap occupancy histograms, dump out snapshot files for offline analysis. Official documentation states that this command may no longer be used in subsequent releases, current JDK8 can still be used.

5.2 jmapUse

By -helpviewing parameters help is also available on jmapthe official documentation : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jmap.htmlhelp as follows:

[root@test bin]# jmap -help
Usage:
    jmap [option] <pid>
        (to connect to running process)

where <option> is one of:
    <none>               to print same info as Solaris pmap
    -heap                to print java heap summary
    -histo[:live]        to print histogram of java object heap; if the "live"
                         suboption is specified, only count live objects
    -clstats             to print class loader statistics
    -finalizerinfo       to print information on objects awaiting finalization
    -dump:<dump-options> to dump java heap in hprof binary format
                         dump-options:
                           live         dump only live objects; if not specified,
                                        all objects in the heap are dumped.
                           format=b     binary format
                           file=<file>  dump heap to <file>
                         Example: jmap -dump:live,format=b,file=heap.bin <pid>
    -F                   force. Use with -dump:<dump-options> <pid> or -histo
                         to force a heap dump or histogram when <pid> does not
                         respond. The "live" suboption is not supported
                         in this mode.
    -h | -help           to print this help message
    -J<flag>             to pass <flag> directly to the runtime system

复制代码

As indicated above, jmapparameters are used -heap, -histoand -dump, in conjunction with examples java-monitor-example, as follows:

• Print jvm memory usage overall

[root@test bin]# jmap -heap 13680
Attaching to process ID 13680, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.51-b03

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 134217728 (128.0MB)
   NewSize                  = 44564480 (42.5MB)
   MaxNewSize               = 44564480 (42.5MB)
   OldSize                  = 89653248 (85.5MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 31981568 (30.5MB)
   used     = 5306632 (5.060798645019531MB)
   free     = 26674936 (25.43920135498047MB)
   16.59278244268699% used
From Space:
   capacity = 6291456 (6.0MB)
   used     = 1081440 (1.031341552734375MB)
   free     = 5210016 (4.968658447265625MB)
   17.18902587890625% used
To Space:
   capacity = 6291456 (6.0MB)
   used     = 0 (0.0MB)
   free     = 6291456 (6.0MB)
   0.0% used
PS Old Generation
   capacity = 89653248 (85.5MB)
   used     = 16615680 (15.845947265625MB)
   free     = 73037568 (69.654052734375MB)
   18.533271655701753% used

18006 interned Strings occupying 2328928 bytes.

复制代码

From the above information, it can be seen the JVM heap memory currently used, including the young generation ( Edenarea, Fromregion, Toarea), and the old generation.

• View class name, number of objects, the size of the object occupy histogram

[root@test bin]# jmap -histo:live 13680|more

 num     #instances         #bytes  class name
----------------------------------------------
   1:         36536        6462912  [C
   2:         35557         853368  java.lang.String
   3:          7456         826968  java.lang.Class
   4:         20105         643360  java.util.concurrent.ConcurrentHashMap$Node
   5:          1449         469024  [B
   6:          6951         399280  [Ljava.lang.Object;
   7:          9311         297952  java.util.HashMap$Node
   8:          3122         274736  java.lang.reflect.Method
   9:          2884         269112  [I
  10:          6448         257920  java.util.LinkedHashMap$Entry
  11:          2994         255160  [Ljava.util.HashMap$Node;
  12:         15249         243984  java.lang.Object

.....
.....

复制代码

As described above, using -histooutput including the serial number, number of instances, the number of bytes occupied by the class name. Specified as follows:

1. instances column: Indicates how many instances of the current class.
2. bytes columns: Description of the current instance of the class total number of occupied bytes
3. class name column: is the name of the current class representation, class name read:
4. B represents a byte
5. C represents char
6. D means double
7. F means float
8. I on behalf of int
9. J represents the long
10. Z represents boolean
11. [Representative array, such as [I corresponds to the int []
12. Object with the [L + represents the class name

• The memory situations dump memory to a local file

[root@test bin]# jmap -dump:file=./heap.hprof 13680

复制代码

As described above, the stack will be written to the current directory where the heap.hproffile as to how to analyze this document, may be used jhat, but typically the actual development, rarely jhat memory dump file to be analyzed directly, so it is no longer They are told. More is the use of tools MATto visualize way to view, follow-up article will be on MATthe use of tools will be explained in detail.

6 thread stack query tooljstack

6.1 jstackDescription

This command prints the specified Java application thread stack for each Java frame, will print the full class names, method names, byte code index (BCI) and line number, can be used to detect deadlocks, thread standstill, the process consumes investigation cpu too high alarm issues.

6.2 jstackUse

Use -helpparameter access help, is also available on jstackthe official documentation : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstack.htmlhelp as follows:

[root@test bin]# jstack -help
Usage:
    jstack [-l] <pid>
        (to connect to running process)
    jstack -F [-m] [-l] <pid>

Options:
    -F  强制dump线程堆栈信息. 用于进程hung住， jstack <pid>命令没有响应的情况
    -m  同时打印java和本地(native)线程栈信息，m是mixed mode的简写
    -l  打印锁的额外信息
复制代码

Binding example java-monitor-example, the thread information can be printed (usually prints the contents of the file and then written to the analysis), as follows:

• Print this thread stack information

[root@test bin]# jstack 13680
2019-08-16 23:18:18
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.51-b03 mixed mode):
"http-nio-8083-Acceptor-0" #39 daemon prio=5 os_prio=0 tid=0x00007f7520698800 nid=0x359a runnable [0x00007f7508bb7000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
	- locked <0x00000000f8c85380> (a java.lang.Object)
	at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:448)
	at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:70)
	at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
	at java.lang.Thread.run(Thread.java:745)

复制代码

6.3 thread dump analysis

6.3.1 Thread State

java thread stack using jstackthe dump out, you can see the state of the thread, the thread state was divided into six kinds, you can refer to the official documentation : https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr034.htmlThe following

• NEW

It has created a new thread, but has not started (not yet started), jstackit will not print thread information of this state.

• RUNNABLE

Java virtual machine threads are run under the task of the state, but in fact it is just means that the thread is runnable (ready). For single-core CPU, multiple threads at the same time, can only run one thread, others need to wait for CPU scheduling.

• BLOCKED

The thread is blocked, waiting for a lock, a plurality of threads sharing a lock, a thread is used when the synchronization method of the lock to enter a synchronized block or method, and this thread needs to enter the sync block, the need to lock, cause the thread is blocked.

• WAITING

Wait state, the thread is in a wait state when executing the method of any of the three methods. 1. Object.waitA method, use and no timeout parameter; 2 Thread.joinmethod is not used 3. timeout parameter LockSupport.parkmethod. Thread is waiting state will wait for another thread to handle special behavior. A thread is in a wait state (the wait, typically in waiting for other threads to complete an operation (or notifyAll Notify). Note that, Object.wait()the method can only be called a synchronization code block. Called wait()the method will release the lock.

• TIMED_WAITING

Thread waits the specified time, for calling this method may lead to a thread in this state:. 1 Thread.sleep 2. The method of Object.waitmethod, with a time 3. The Thread.joinmethod 4. With time LockSupport.parkNanosmethod, with time 5. LockSupport.parkUntilMethod with time. Note that Thread.sleepthe method is called, it will not release the lock, still occupy system resources.

• TERMINATED

Thread suspended state, this thread has fully carried out its task.

This can be seen from the following changes in thread state:

6.3.2 Analysis of jstackthe thread stack content

From the front using the jstackdump out the information, we need to know the following information:

• "http-nio-8083-Acceptor-0" #39: Is the thread's name, therefore, we need to set their own general identification can name when creating a thread.
• daemon Does it mean that thread is a daemon thread
• prio Indicates the priority we set for the thread
• os_prio The priority of the corresponding operating system threads represented because not all operating systems support thread priorities, it may occur are set to 0
• tid Thread id
• nid Corresponding to the thread operating system native thread id, java each thread has a corresponding thread of the operating system, which is 16 hexadecimal, and therefore usually after the thread ID is acquired in the operating system, converted to hexadecimal need to correspond on.
• java.lang.Thread.State: RUNNABLERunning, it has been described above state of the thread, if WAITING state, in brackets shows the causes of waiting, such as parking instructions because LockSupport.park method results in a call waiting. The usual stack information, there will be a lock mark, as - locked <0x00000000f8c85380> (a java.lang.Object)said it is occupied by the lock.
• For thread pause, CPU usage and other issues, the focus can look at the waitthread state
• For the deadlock in the Dump out the thread stack snapshots can report directly to the Java level deadlock.

7 JVM Statistics tooljstat

7.1 jstatDescription

jstatNamely JVM Statistics Monitoring Tool, that JVM statistics monitoring tools, including monitoring class loading, memory, garbage collection, JIT compiler and other operating data, no graphics on the server, it is the tool of choice to run the positioning of virtual machine performance problems.

7.2 jstatUse

Use -helpparameter access help, is also available on jstatthe official documentation : https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstat.htmlhelp as follows:

[root@test bin]# jstat -help
jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]
Usage: jstat -help|-options
       jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]

Definitions:
  <option>      An option reported by the -options option
  <vmid>        Virtual Machine Identifier. A vmid takes the following form:
                     <lvmid>[@<hostname>[:<port>]]
                Where <lvmid> is the local vm identifier for the target
                Java virtual machine, typically a process id; <hostname> is
                the name of the host running the target Java virtual machine;
                and <port> is the port number for the rmiregistry on the
                target host. See the jvmstat documentation for a more complete
                description of the Virtual Machine Identifier.
  <lines>       Number of samples between header lines.
  <interval>    Sampling interval. The following forms are allowed:
                    <n>["ms"|"s"]
                Where <n> is an integer and the suffix specifies the units as
                milliseconds("ms") or seconds("s"). The default units are "ms".
  <count>       Number of samples to take before terminating.
  -J<flag>      Pass <flag> directly to the runtime system.
复制代码

As shown in the above, vmid, interval, countare the process ID, the print time interval (s or MS), the number of prints, wherein the optionmain parameters are the following (the command may be used jstat -optionto view):

• -class Statistical class loader behavior information, such as total number of classes loaded
• -compile Statistics HotSpot Just-in-Time compiler behavior
• -gc Statistical information jdk when gc heap
• -gccapacity Statistics different respective heap capacity situation generations
• -gccause Gc statistics, and (with -gcutil) and cause gc events
• -gcnew When statistics gc, the situation of the new generation
• -gcnewcapacity When statistics gc, the new generation heap capacity
• -gcold When statistics gc, situation of older areas
• -gcoldcapacity When statistics gc, tenured heap capacity
• -gcpermcapacity When statistics gc, permanent heap area capacity
• -gcutil When statistics gc, heap situation
• -printcompilation hotspot compiling statistical method

Generally, we use the -class, -gc, -gccauseand -gcutilmore, it is mainly used to analyze classes and heap usage and gc situation.

GC 7.3 The monitoring of the JVM

The example above works java-monitor-example, for example, which contains a function to test memory overflow (using an array loop to create objects until memory overflow). Use jstat -gc 13680 1000that is monitored every second, call /monitor/user/oomthe interface, the GC heap and see the changes. For the convenience of viewing, I put the output into sublime display, as shown below:

OOM error log output:

Output the contents of the above described each column as follows:

• S0C The current generation of young first survivor (s0) of total capacity (KB).
• S1C A first current younger generation survivor (s1) of the total capacity (KB).
• S0U s0 capacity (KB) has been used.
• S1U s1 capacity (KB) has been used.
• EC The current generation of young eden Chief capacity (KB).
• EU eden area capacity (KB) has been used.
• OC Total capacity Capacity (KB) older generations.
• OU The old generation used capacity (KB).
• MC Current Metaspace total capacity (KB).
• MU Metaspace currently used capacity (KB).
• CCSC Compressed class size capacity
• CCSU Compressed class used capacity
• YGC From when the application starts up to now, the total number of the young generation of young generation GC Events of occurrence.
• YGCT From when the application starts up to now, a total elapsed time the young generation Young generation garbage collection.
• FGC From when the application starts up to now, full GC events total number.
• FGCTWhen the application starts from now, Full sc .GCT total time from when the application starts up to now, the total garbage collection time GCT.GCT = YGCT + FGCT

As can be seen from the above output line 6, ECand EU, OCand OUrepresents the young generation, old generation memory have been exhausted (and capacity values are equal), the OOM occurs. At this time, it is necessary to take measures to increase the memory (-Xmx parameter) or find the cause of the OOM code changes.

8 summary

For the monitoring of java applications, the paper jdk provides a command-line tool itself provides is explained and introduced the use of a complete description of the view java application process, view the boot parameters, view memory condition, see the thread, the view memory statistics, etc. mainly jps, , jinfo, jmap, jstackfivejstat tools, and with examples, want to learn java developers to be able to master these techniques in the monitoring of java application, you can calmly face as OOM, high CPU, thread pause and other issues.

Reference material

• JDK Tools Reference document :https://docs.oracle.com/javase/8/docs/technotes/tools/unix/
• Sample codejava-monitor-example :https://github.com/mianshenglee/my-example/tree/master/java-monitor-example
• Java developers must master online issue of the investigation command :https://www.hollischuang.com/archives/1561
• Java comes with performance monitoring tools :http://www.tianshouzhi.com/api/tutorials/jvm/346

Related Reading

• "Java application monitoring (1) -java programmer should be aware of the application monitoring technology."
• "Java application monitoring (2) -java command of the secret"