Application suspended animation troubleshooting process and OutOfMemory problem location and processing

Application suspended animation troubleshooting process and OutOfMemory problem location and processing

Related keywords: java8, jmap, jstack, jstat, netstat, less, grep, jvm

Scenes

Review the troubleshooting process of OOM problems encountered before. The business details are not shown for now. The appearance: the application freezes and the interface request cannot respond. The main problem is that the business volume increases, one core business has byte[], and the heap setting is relatively small. Processing process, optimize the process of byte[], increase the size of the heap.

Troubleshooting process:
1.jmap outputs heap usage
2.jstat outputs GC information
3.jstack outputs thread stack snapshot
4.netstat checks the TCP connection status of the listening port
5. Checks whether there is an OutOfMemory error in the log

jmap

jmap: java memory map can obtain DUMP files (heap dump snapshots, binary files); it can also obtain memory-related information of the specified java process, including the usage of each area of ​​​​the java heap, statistical information of objects in the heap, and class loading information

Outputs the usage status of each area of ​​the current heap.

Output the JVM configuration of the current PID application and the usage of the young and old generations. We can understand the memory usage of the application and optimize our Java application as needed.

Syntax: jmap -heap pid

Attaching to process ID 29296, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.232-b09

using thread-local object allocation.
Parallel GC with 8 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = x
   MaxHeapFreeRatio         = x
   MaxHeapSize              = x
   NewSize                  = x
   MaxNewSize               = x
   OldSize                  = x
   NewRatio                 = x
   SurvivorRatio            = x
   MetaspaceSize            = x
   CompressedClassSpaceSize = x
   MaxMetaspaceSize         = x
   G1HeapRegionSize         = x

Heap Usage:
PS Young Generation
Eden Space:
   capacity = x
   used     = x
   free     = x
   x% used
From Space:
   capacity = x
   used     = x
   free     = x
   x% used
To Space:
   capacity = x
   used     = x
   free     = x
   x% used
PS Old Generation
   capacity = x
   used     = x
   free     = x
   x% used

x interned Strings occupying x bytes.

The meaning of each parameter in Heap Configuration:

MinHeapFreeRatio：堆可以被允许的最小空闲大小（以百分比表示）。
MaxHeapFreeRatio：堆中可以被允许的最大空闲空间大小（以百分比表示）。
MaxHeapSize：堆的最大尺寸。
NewSize：JVM 在创建新对象时将分配的内存区大小。
MaxNewSize：新生代的最大尺寸。
OldSize：老年代的初始大小。
NewRatio：新生代和老年代之间的比率。
SurvivorRatio：两个幸存者区域与 Eden 区域的大小比率。
MetaspaceSize：Metaspace 的初始大小（即非堆大小），用于存储加载类、方法等元数据。
CompressedClassSpaceSize：压缩类空间的大小。
MaxMetaspaceSize：Metaspace 的最大大小。
G1HeapRegionSize：G1 中的 Region 大小。每个 Region 的大小由 JVM 计算得出。

在 Heap Usage 部分，我们可以看到当前堆的使用情况，包括：

capacity：Java 堆的总容量。
used：Java 堆当前使用的总量。
free：Java 堆当前可用的数量。
42.22304174399364% used：Java 堆的当前使用率。

PS Young Generation: 年轻代
    Eden Space: eden区
    From Space: 幸存区1
    To Space: 幸存区2
PS Old Generation: 老年代

Output the 10 objects surviving and occupying the most memory in descending order

jmap -histo:live 29296 |head -n 10

Use this command to check whether there are abnormal objects that occupy too much memory and cause memory leaks

Output dump binary file

Syntax: jmap -dump:format=b,file=heap.bin

Output the dump binary file of the heap through the above command, and analyze the memory of our java program through jconsole of eclipse

JVM configuration parameters

-Xmx4048m  最大堆大小

-Xms4048m  初始堆大小

-Xmn2600m  设置年轻代的大小 建议设置为堆的3/8

-XX:+UseParallelGC  使用并行垃圾收集器，在多 CPU 系统上可以提高垃圾收集的效率。

-XX:-UseAdaptiveSizePolicy  并行收集器会自动选择年轻代区分大小和相应的Survivor区比例
 
-XX:SurvivorRatio=x   幸存区大小比率。

-XX:+HeapDumpOnOutOfMemoryError 在 OutOfMemoryError 异常发生时生成堆转储文件（heap dump）。

JVM configuration parameters can be divided into the following categories:

标准参数（- 开头）：这些参数主要控制 JVM 的基本行为，并且在所有的 JVM 实现中都支持。

非标准参数（-X 开头）：这些参数是实现特定的，不保证所有 JVM 实现都支持，且在未来的版本中可能会有所改变。

高级参数（-XX 开头）：这些参数被称为高级参数，它们提供了对 JVM 内部行为的细粒度调整，并允许用户绕过某些标准限制。这些参数也是实现特定的，并且在不同的 JVM 实现中可能有所不同。

下面是一些常用的 JVM 参数及其说明：

-Xms：初始堆大小，指定 JVM 启动时堆内存的初始大小。默认情况下，Java 虚拟机自适应地调整堆大小。

-Xmx：最大堆大小，指定 JVM 最大堆内存大小。如果堆超出该阈值，则 Java 虚拟机将抛出 OutOfMemoryError 异常。

-XX:PermSize：永久代初始值，默认值为 64 MB。

-XX:MaxPermSize：永久代最大值，默认值为 64 MB。

-XX:NewSize：新生代初始值，指定了启动时新生代的初始大小。

-XX:MaxNewSize：新生代最大值，指定了新生代的最大可用空间。

-XX:SurvivorRatio：幸存区大小比率。

-XX:+UseConcMarkSweepGC：启用 CMS 垃圾收集器，用于收集老年代内存空间的垃圾。

-XX:+UseParNewGC：启用 ParNew 垃圾收集器，用于收集新生代内存空间的垃圾。

-XX:+UseSerialGC：使用串行垃圾收集器，适用于小型应用程序和单 CPU 系统。

-XX:+UseParallelGC：使用并行垃圾收集器，在多 CPU 系统上可以提高垃圾收集的效率。

-XX:+HeapDumpOnOutOfMemoryError：在 OutOfMemoryError 异常发生时生成堆转储文件（heap dump）。

这些参数只是 JVM 参数中的一小部分，有些参数可能因版本、厂商或系统而异，使用前需要仔细查看文档或手册。

to stand

jstat: java virtual machine statistics monitoring tool

jstat -gccause pid

jstat -gccapacity pid

jstat -gc pid

options	illustrate
-class	Display information about class loading status
-compiler	Display JIT compiler statistics
-gc	Display information about garbage collection status
-gccapacity	Used to view the storage capacity of the new generation, the old generation and the permanent generation
-gccause	On the basis of -gc, display the cause of the last GC
-gcutile	Show overall garbage collection information

YGC: 从应用程序启动到采样的年轻代中GC的次数
FGC: 从应用程序启动到采样是old代GC的次数
FGCT: 从应用程序启动到采样是Old代GC所用的时间(s)
GCT: 从应用程序启动到采样时GC用的总时间(s)
GCC: 当前GC原因(No GC为当前没有执行GC)
LGCC: 最后一次GC原因

jstack

jstack generates a thread snapshot of the current java application

Output thread snapshot jstack pid > theadInfo.log

When the process does not respond, force the thread snapshot to be output, and it will display whether there is a deadlock jstack -F pid > theadInfo.log

RUNNABLE: 线程运行中和IO等待
BLOCKED:  线程在等待Monitor锁
TIMED_WAITING: 线程在等待唤醒，设置了超时时间
WAITING: 线程在无限等待唤醒
IN_NATIVE: 反应的是线程正在运行系统“本机”代码而不是java代码

In the log output by jstack:
1. Pay attention to whether there is a deadlock
2. Whether there are a lot of blocked in our own code, and analyze a single thread

netstat

netstat -anp |grep ':port'

Check the port that our service listens on, the established connection status, and see if there is any abnormal CLOSE_WAIT

TCP will have the following states, which correspond to each process of TCP three-way handshake, information transmission, and TCP four-way wave: listen, syn_sent, syn_revd, established, fin_wait_1, close_wait, fin_wait_2, last_ack, time_wait, closed

CLOSE_WAIT: It is formed by passively closing the connection and will occupy a network connection.
1. The client calls the close function to initiate two waves. After the server accepts it, it will enter the CLOSE_WAIT state. After the client receives the server's ACK, it will enter the FIN_WAIT_2 state; but the server has not yet initiated two waves and only completed four The connection is truly disconnected after the second wave of hands, and the two parties will release the corresponding connection resources at this time.
2. If the server does not call close after receiving two waves, then the server will have a large number of connections in the CLOSE_WAIT state, and each connection will occupy the server's resources, which will eventually lead to fewer and fewer available resources on the server.

TIME_WAIT: When the client actively closes the connection, after sending the last ACK, it will enter the TIME_WAIT state, and then stay for two maximum message survival times (2MLS) to enter the CLOSE state.

reference

https://blog.csdn.net/sulijin/article/details/124412379
https://www.nowcoder.com/discuss/388301958082293760