4. Starting from scratch JVM practical master - JVM monitoring and online problem location

foreword

Java programmers interviewing JVM is almost more difficult than asking questions. Questions such as JVM monitoring, online OOM, and CPU load 100% are often asked. Although it may not be our turn to deal with online problems in enterprises, whether it is for interviews or In order to cope with development, it is necessary to understand the JVM online problem handling.

Relatively speaking, whether it is to solve the fault problem or deal with performance bottlenecks, the general idea is roughly the same, that is: analyze data (logs), analyze and troubleshoot, locate problems, and solve problems. If we even use program execution data or logs If we can't get it, then we have no way to locate the problem.

Fortunately, Java has submitted JVM monitoring tools and related instructions to help us obtain JVM-related data to help us troubleshoot problems.

tool installation

We only know that there is a JVM, but its operation feels invisible to us, so we need tools to monitor its real-time status, just like the performance monitor of Windows, JDK also has its own Visualization tools. Java provides 2 monitoring tools:

• D:\opensource\jdk1.8\bin\jconsole.exe

• D:\opensource\jdk1.8\bin\jvisualvm.exe

jconsole

Enter jconsole through the cmd command line, and the following interface will pop up

. Select java to enter, and you can see the memory status, class loading status, thread status, etc. after entering.

Jvisualvm

We run cmd, enter jvisualvm, and start Java VisualVM.

The local menu on the left is the java process. After selecting a process, you can see the heap, class loading status, and ready-made status on the right.

jvisualvm installs GC plugin

The built-in jvisualvm does not monitor the GC garbage collection function, we need to install additional plug-ins:

Open Tools -> Plugins -> Select the "Available Plugins" page: We install a Visual GC here, so that we can see the memory recovery and the status of each generation. After ticking it, click Install, which is the regular next and agree to the agreement, etc. The network is not very stable, sometimes it may take a few more attempts. You can modify the plug-in center address in the settings:

Modify the address according to the following steps: Find the plug-in center

http://visualvm.github.io/pluginscenters.html

Find the corresponding JDK version:

http://visualvm.github.io/archive/uc/8u40/updates.htmlCopy

plug-in address:

Install the plugin:

Then find Visual GC in the available plug-ins

After the installation is complete, we close the current monitoring page, open it again, and you can see that there is an additional Visual GC page behind the Profiler.

Here we can see JIT activity time, class loading activity time, GC activity time and the situation of each generation.

It should be noted that the JDK version used by the current courseware is 1.8, which still comes with VisualVM. The version starting from 1.9 does not come with it, and needs to be downloaded additionally. The github address for downloading is:

https://visualvm.github.io/download.html

In addition, if the development tool uses Intellij IDEA, you can download a plug-in, VisualVM Launcher, and you can go directly to the above page through the plug-in startup, without looking for your own project in the left entry.

Of course, there are other tools, but this will be the main development of the all-in-one fault handling tool in the foreseeable future. So we will use this tool to analyze our JVM operation and optimize it later. To optimize, we also need Have a further understanding of the composition of the JVM

JVM monitoring command

In the production environment, we often encounter various strange performance problems. We can use the JVM monitoring commands provided by Java to achieve monitoring and viewing effects. The related commands are as follows

name	main effect
jps	View running Java processes
jstack	print thread snapshot
jmap	Export heap memory image file
to stand	View jvm statistics
jinfo	View and modify jvm configuration parameters in real time
Jhat	Used to analyze heapdump files

jps view progress

jps can list the running Java processes, and display the name of the main class (Main Class, the class where the main() function is located) of the virtual machine and the process id, and the parameters can be viewed through jps -help

options	effect
-q	Only output the process id
-m	Output the parameters passed to the main function of the main class
-l	Output the full class name of the main class, if the process executes the Jar package, output the name of the jar package
-v	The jvm parameter specified when the program starts

Case presentation:

jstack: print thread snapshot

Generally, in the production environment, if problems such as long-term pause, stuck, deadlock, and long request time occur, you can analyze and locate the problem by printing thread snapshots. The following is a deadlock code:

public class DeadlockExample {
    
    
    private static Object lock1 = new Object();
    private static Object lock2 = new Object();
     public static void main(String[] args) {
    
    
        Thread t1 = new Thread() {
    
    
            public void run() {
    
    
                synchronized (lock1) {
    
    
                    System.out.println("Thread 1 acquired lock 1");
                    try {
    
    
                        Thread.sleep(1000);
                        synchronized (lock2) {
    
    
                            System.out.println("Thread 1 acquired lock 2");
                        }
                    } catch (InterruptedException e) {
    
    
                        e.printStackTrace();
                    }
                }
            }
        };
         Thread t2 = new Thread() {
    
    
            public void run() {
    
    
                synchronized (lock2) {
    
    
                    System.out.println("Thread 2 acquired lock 2");
                    try {
    
    
                        Thread.sleep(1000);
                        synchronized (lock1) {
    
    
                            System.out.println("Thread 2 acquired lock 1");
                        }
                    } catch (InterruptedException e) {
    
    
                        e.printStackTrace();
                    }
                }
            }
        };
         t1.start();
         t2.start();
         try {
    
    
            t1.join();
            t2.join();
        } catch (InterruptedException e) {
    
    
            e.printStackTrace();
        }
    }
}

Then we try jps -l to view the process first

, get the process id, and use jstack to view each thread snapshot: jstack 27928

From the thread snapshot, you can see all the threads in the current process. Among them are the threads of our code, and the state is locked. At the same time, it prompts found 1 deadlock to find a deadlock, and gives the location where the deadlock occurred.

jmap: export heap snapshot

Execution jmap -histo pidcan print out the number of instances and memory usage of each class in the current heap, as follows, class name is the class name of each class ([B is byte type, [C is char type, [I is int type), bytes It is the memory size occupied by all examples of this class, and instances is the number of instances of this class:

execute jmap -dump to dump the heap memory snapshot to a specified file, such as executing

jmap -dump:format=b,file=/data/jvm/dumpfile_jmap.hprof PID , you can dump the snapshot of the current heap memory into the dumpfile_jmap.hprof file, and then analyze the memory snapshot.

We generally configure the production environment so that the virtual machine automatically generates a dump file after an OOM exception occurs

-XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=/Users

For example, there is an infinite loop of code and running it for a certain period of time will cause memory overflow

public class Main {
    
    
    public static void main(String[] args) {
    
    
        ArrayList arrayList = new ArrayList();
        int i = 0;
        while(true){
    
    
            arrayList.add(new Main());
            System.out.println(i++);
        }
    }
}

For the obvious effect, we set the heap to be smaller, and then set HeapDumpOnOutOfMemoryError on

-Xms2m
-Xmx2m
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=d:\

Set the above VM parameters for the current application, and the effect of executing the code is as follows

Then we find the hprof heap snapshot file, which can be loaded and analyzed through the jvisualvm tool.

Now the snapshot file, the file type needs to be selected as a heap snapshot.

After loading, you can directly see the error message of memory overflow

After clicking on the thread name main, you can directly locate the memory overflow location

jstat: monitor virtual machine information

jstat -gc pid 500 10: pid is the thread ID, and prints the status of the Java heap every 500 milliseconds (capacity, usage capacity, gc time, etc. of each area). Printing 10 times jstat can also monitor the memory size of each area and monitor class loading information from other angles.

Specifically You can google the detailed usage of jstat. The following is the result comparison

S0C: the size of the first survivor area
S1C: the size of the second survivor area
S0U: the used size of the first survivor area
S1U: the used size of the second survivor area
EC: the size of the Eden area
EU: the use of the Eden area Size
OC: Old Generation Size
OU: Old Generation Usage Size
MC: Method Area Size
MU: Method Area Usage Size
CCSC: Compressed Class Space Size CCSU
: Compressed Class Space Usage Size
YGC: Young Generation Garbage Collection Times
YGCT: Young Generation Garbage Collection Consumption Time
FGC: Old generation garbage collection times
FGCT: Old generation garbage collection time
GCT: Total garbage collection time
Unit: KB

jinfo : view process parameters

jinfo(Configuration Info for Java) Check the configuration parameters of the virtual machine, and can also be used to adjust the configuration parameters of the virtual machine.

In many cases, Java applications do not specify all Java virtual machine parameters. At this time, developers may not know the default value of a specific Java virtual machine parameter. In this case, it may be necessary to obtain the default value for a parameter by looking up the documentation. This search process can be very difficult. But with the jinfo tool, developers can easily find the current value of the Java virtual machine parameters.

jinfo can not only view the actual value of a certain Java virtual machine parameter at runtime, but even modify some parameters at runtime and make it take effect immediately. However, not all parameters support dynamic modification. Parameters can only be modified in real time with the flag marked manageable. In fact, this modification capability is extremely limited.

VM Flags:
Non-default VM flags: -XX:CICompilerCount=12 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=null -XX:InitialHeapSize=2097152 -XX:MaxHeapSize=209715200 -XX:MaxNewSize=69730304 -XX:MinHeapDeltaBytes=524288 -XX:OldSize=524288 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:-UseLargePagesIndividualAllocation -XX:+UseParallelGC
Command line: -Xms2m -Xmx200m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=d:\ -javaagent:D:\Program Files\JetBrains\IntelliJ IDEA 2022.2.1\lib\idea_rt.jar=54817:D:\Program Files\JetBrains\IntelliJ IDEA 2022.2.1\bin -Dfile.encoding=UTF-8

Through jinfo -flags pid: View the parameter value that has been assigned a value

jinfo can not only view the actual value of a certain Java virtual machine parameter at runtime, but even modify some parameters at runtime and make it take effect immediately. However, not all parameters support dynamic modification. Parameters can only be modified in real time with the flag marked manageable. You can use the java -XX:+PrintFlagsInitial | grep manageable command to view the manageable

modification format as follows:

• For Boolean type modification: jinfo -flag ±parameter pid
• For non-Boolean types: jinfo -flag parameter name=parameter value pid

For example, modify and print the GC log demonstration as follows: jinfo -flag +PrintGCDetails PID

//查看进程
C:\Users\Administrator>jps -l
20924 org.example.Main
...

//查看是否设置PrintGCDetails参数配置
C:\Users\Administrator>jinfo -flag PrintGCDetails 20924

//增加jvm参数：打印GC详情
C:\Users\Administrator>jinfo -flag +PrintGCDetails 20924

//查看是否设置PrintGCDetails参数配置
C:\Users\Administrator>jinfo -flag PrintGCDetails 20924
-XX:+PrintGCDetails
...

Solve the online CPU100% problem

Generally, 100% CPU is basically caused by an infinite loop of code. The core idea of ​​the investigation is to find the corresponding server, locate which codes in which thread of which process caused the problem, and briefly introduce the abnormal code examples at that time.

The first step is to find the process that consumes the most CPU: Use top -c to view the process, and then enter the big P to sort according to the CPU usage.

The second step is to find the thread that consumes the most CPU in the process: find the process with the highest CPU, find the process ID (PID), find the thread corresponding to this process through the command top -Hp PID, and then enter the big P to sort according to the CPU usage .

Get the first PID is the most time-consuming thread ID, and then use printf "%x\n" PID to convert the PID from decimal to hexadecimal (the reason for converting to hexadecimal is because in the stack, The thread id is expressed in hexadecimal.)

[root@VM-4-2-centos ~]# printf "%x\n" 13759
35bf

Next, we need to use jstack to print the stack information of the process, and then use grep to view the corresponding thread-related things. jstack process ID | grep "thread ID" -C5 --color

jstack 30979  | grep "35bf" -C5 --color

At this time, you can print out the code, and then match the nid from the printed thread snapshot, you can locate which thread is time-consuming, and at the same time quickly locate the code, you can see which method in which class caused the CPU 100 % of the reason.

Remote monitoring

If you are not very familiar with commands, it is a pain to use commands to monitor the JVM. JVisualvm provides the jmx remote function. The default is to provide RMI services through the ip address of localhost, which requires us to configure JVM parameters remotely to enable remote connections

-Xms256m 
-Xmx512m 
-Xss256m 
-XX:PermSize=512m 
-XX:MaxPermSize=1024m 

-Dcom.sun.management.jmxremote 
-Djava.rmi.server.hostname=服务器IP 
#远程服务的端口：
-Dcom.sun.management.jmxremote.port=9015 
#客户端 rmi通信端口
-Dcom.sun.management.jmxremote.rmi.port=9015 
#关闭ssl功能
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false 

Then we add the remote host in the local Jvisualvm

and then add the jmx remote connection

to set the remote connection parameters and cancel the ssl connection.
The article is over. If it is helpful to you, please give a good review. Your encouragement is my biggest motivation