Troubleshooting in Java service production environment

Preface
1. Operating parameters
- 1.1 View JVM parameters
- 1.2 View real-time GC status
2. Error troubleshooting
reference

Preface

In a production environment, we cannot immediately check the current running status and get error messages through breakpoint debugging, new logs, and visualization tools. At this time, we can use Java's own command line tools and related dump analysis tools as well as some small Techniques can greatly improve our efficiency in troubleshooting problems

1. Operating parameters

Listed below are some commonly used and very effective commands and parameters to view information about Java programs at runtime to help you understand the running status of the program. There are a large number of available functions provided by other parameters. Please refer to Oracle official documentation .

1.1 View JVM parameters

jps -lView all running Java programs, display the startup class name, and obtain the PID:

4706 org.apache.catalina.startup.Bootstrap
5023 sun.tools.jps.Jps

jinfo -flags [PID]Check the runtime process parameters and JVM parameters:

Attaching to process ID 28987, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.171-b11
Non-default VM flags: -XX:CICompilerCount=3 -XX:InitialHeapSize=132120576 -XX:MaxHeapSize=2092957696 -XX:MaxNewSize=697303040 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=44040192 -XX:OldSize=88080384 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC
Command line:  -Dspring.config.location=application.properties -Dspring.profiles.active=staging

java -XX:+PrintFlagsFinal -versionView the default JVM parameters of the current virtual machine

1.2 View real-time GC status

jstat -gc PID 1000 10Check gc information once every second, 10 times in total

S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT   
512.0  512.0   15.3   0.0    4416.0   1055.2   11372.0     7572.5   14720.0 14322.5 1664.0 1522.8     40    0.137   8      0.039    0.176

Outputs a lot of parameters. For explanations of each field, please see here .

You may encounter prompts during this period sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 24.181-b01. Target VM is 25.171-b11, because multiple versions are installed. Using which, ls -lyou can briefly locate the Java version that is the same as the currently executing Java program.

2. Error troubleshooting

2.1 Memory problem

2.1.1 View JVM memory

View JVM memory status:jmap -heap [pid]

It should be noted that when using CMS GC, the execution of jmap -heap may cause the JAVA process to hang.
Check the memory usage of objects in the JVM heap:jmap -histo [pid]
Export memory information in JVM:jmap -dump:format=b,file=[filename] [pid]

jhat is a tool that comes with SUN 1.6 and above for analyzing JVM heap DUMP files. Based on this tool, the memory usage of objects in JVM HEAP can be analyzed.

jhat -J-Xmx1024M [file](The file here refers to the memory data file exported by jmap -dump)

After execution, wait for the console to output start HTTP server on port 7000, and then use the browser to access IP: 7000.
eclipse Memory Analyzer: A plug-in provided by Eclipse for analyzing JVM heap Dump files. With this plug-in, you can check the memory usage of objects, reference relationships, analyze memory leaks, etc.

2.1.2 Analysis of OOM problems

Memory leak causing OOM? Memory usage is unusually high? This is a problem that often occurs in production environments. Java provides dump files for us to record what happened in the memory. We need to use some tools to obtain valuable information from them.

Export Dump file

Add these parameters to the Java program in advance and print the dump file.-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./
Use jmap with a running program:jmap -dump:format=b,file=[filename] [pid]

Analyze Dump files

If the Dump file is not too large, it can be transferred to http://heaphero.io/index.jsp for analysis.
The file is relatively large, and if you want to conduct a more systematic analysis, it is recommended to use MAT analysis. There are several common viewing methods as follows:
1. [Leak Suspects] on the home page can guess where the problem lies.
2. Click [Create a histogram from an arbitrary set of objects] to find the number of all objects
3. Right-click an object [Merge Shortest Paths to GC Roots] -> [exclude all phantom/weak/soft etc. references] to query which GC ROOT references a large number of objects.

2.2 Threading issues

The task does not exit for a long time? CPU load too high? It is very likely that some threads continue to execute without interruption due to infinite loops or deadlocks, but not reporting errors is the most annoying, so no error information can be seen in the log, and dump files cannot be used for analysis because it has nothing to do with memory. At this time, we need to use thread analysis tools to help us.

Export the jstack filejstack PID > 文件 . If it fails, please add -Fparameters. If it still fails, please use the user used when the Java program is started to execute jstack. The following is part of the output format of jstack:

      [线程名]                                                              [PID的16进制]
"http-nio-8080-Acceptor-0" #17 daemon prio=5 os_prio=0 tid=0x00007fac2c4bd000 nid=0x29f4 runnable [0x00007fac192f6000]
   java.lang.Thread.State: RUNNABLE（tomcat的工作线程正在运行，有NEW/RUNNABLE/BLOCKED/WAITING/TIMED_WATING/TERMINATED状态）
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
    - locked <0x00000000faf845a8> (a java.lang.Object)
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:682)
    at java.lang.Thread.run(Thread.java:748)

The output of jstack can see all the threads and their status. We can see which running threads we created ourselves. That is probably the thread that has been executing. At this time, the thread name is particularly important. Therefore, it is recommended to specify a meaningful thread name when creating a new thread. Of course, searching by PID is also very convenient.

Troubleshooting steps

topCheck which java program has the highest load
top -Hp PIDView the running status of all processes in this process
Record the high-load thread ID and printf "&x" PIDconvert it to hexadecimal
jstack PID> Documents
In the jstack file, use the thread ID converted to hexadecimal to query the thread running stack.
Understand what tasks the thread is performing from the stack, and determine the problem based on the business and code.

2.3 Java VisualVM Overview

It is too troublesome to use the command line to remotely monitor jvm. Oracle provides a new visualization in JDK1.6. JVM monitoring tool Java VisualVM. jvisualvm.exe is in the bin directory of the JDK.

After double-clicking to start Java VisualVM, you can see that there are three items: "Local", "Remote" and "Snapshot" in the "Application" column on the left side of the window.

Displayed under "Local" is the resource usage of Java programs running on localhost. If there is a Java program running locally, start Java VisualVM and you will see the corresponding program name. Click the program name to open the corresponding resource monitoring menu. Graphically Lists the statistical information of CPU, Heap, PermGen, classes, and threads occupied by the program in the form.

The resource usage of the Java program on the remote host listed under the "Remote" item, but the jstatd daemon needs to be running on the remote host

VisualVM is divided into 3 categories. It will automatically detect it locally and display it.

Double-click any node under Local and see the changes on the right. You can monitor the running status of CPU, memory, classes, threads, etc., and monitor server performance in real time.

Right click on VisualVM and we can see Thread Dump, Heap Dump

Doing Thread Dump is very fast and you can see the results immediately

Heap Dump takes a little time (you can see the number of objects in the current heap and the occupied proportion, which is very useful for OOM)

reference

https://coding.imooc.com/class/241.html
https://crossoverjie.top/2018/07/08/java-senior/JVM-Troubleshoot/
https://www.jianshu.com/p/1b1c998c4448