This is the second article in the JVM optimization series:
Through the knowledge of jvm garbage collection in the previous article, we understand how jvm handles memory allocation and garbage collection. Theory is a tool to guide practice. With theoretical guidance, when locating problems, knowledge and experience are the key foundation, and data can provide us with a basis.
In the common online problems, most of us will encounter the following problems:
- memory leak
- A process suddenly cpu soars
- thread deadlock
- Slow response...and other issues.
If you encounter the above problems, there are various local tools available offline to support viewing, but online, there are not so many local debugging tools to support, how can we locate the problem based on monitoring tools?
We generally locate based on data collection, and data collection is inseparable from the processing of monitoring tools, such as: running logs, exception stacks, GC logs, thread snapshots, heap snapshots, etc. Regular use of the right analytics and monitoring tools can speed up our ability to analyze data and locate and resolve problems. We will go into details below.
1. Common jvm monitoring tools & instructions
1. jps: jvm process status tool
jps [options] [hostid]If no hostid is specified, it defaults to the current host or server.
The command line parameter options are described as follows:
-q 不输出类名、Jar名和传入main方法的参数
- l 输出main类或Jar的全限名
-m 输出传入main方法的参数
- v 输出传入JVM的参数E.g:
2, jstat: jvm statistics monitoring tool
jstat is a command-line tool for viewing various running status information of virtual machines. It can display running data such as class loading, memory, garbage collection, jit compilation, etc. in the local or remote virtual machine process. It is the preferred tool for locating jvm performance online.
Command format:
jstat [ generalOption | outputOptions vmid [interval[s|ms] [count]] ]
generalOption - 单个的常用的命令行选项，如-help, -options, 或 -version。
outputOptions -一个或多个输出选项，由单个的statOption选项组成，可以和-t, -h, and -J等选项配合使用。Parameter options:
| Option |
Displays |
Ex |
| Statistics for viewing class loading |
jstat -class pid: Displays information such as the number of loaded classes and the space they occupy. |
|
| View statistics on just-in-time compiler compilations in HotSpot |
jstat -compiler pid: Displays information such as the number of VM real-time compilations. |
|
| View the statistics of the garbage collection of the heap in the JVM |
jstat -gc pid: can display gc information, check the number of gc, and time. The last five items are the number of young gc, the time of young gc, the number of full gc, the time of full gc, and the total time of gc. |
|
| View the storage capacity of the young generation, old generation and persistent generation |
jstat -gccapacity: can display the usage and size of three generations (young, old, perm) objects in VM memory |
|
| View garbage collection statistics (this is the same as the -gcutil option), if there is a garbage collection, it will also show the reason for the last and current garbage collection that occurred. |
jstat -gccause: show gc cause |
|
| Check the situation of young generation garbage collection |
jstat -gcnew pid: new object information |
|
| Used to view the storage capacity of the new generation |
jstat -gcnewcapacity pid: information of the new object and its occupancy |
|
| Used to view the GC occurrences in the old generation and persistent generation |
jstat -gcold pid: old object information |
|
| Used to view the capacity of the old generation |
jstat -gcoldcapacity pid: information of old objects and their occupancy |
|
| Used to view persistent generation capacity |
jstat -gcpermcapacity pid: perm object information and its occupancy |
|
| View the status of young, old and holding generation garbage collection |
jstat -util pid: statistics gc information statistics |
|
| Statistics of HotSpot Compilation Methods |
jstat -printcompilation pid: information about the current VM execution |
For example :
Check the gc status execution: jstat-gcutil 27777
3. jinfo: java configuration information
Command format:
jinfo[option] pidFor example: Get some jvm running and startup information of the current process.
4. jmap: java memory mapping tool
The jmap command is used to produce heap dump snapshots. Print out all the 'objects' in the memory of a java process (using pid) (eg: those objects generated, and their number).
Command format:
jmap [ option ] pid
jmap [ option ] executable core
jmap [ option ] [server-id@]remote-hostname-or-IP
Parameter options:
-dump:[live,]format=b,file=<filename> 使用hprof二进制形式,输出jvm的heap内容到文件=. live子选项是可选的，假如指定live选项,那么只输出活的对象到文件.
-finalizerinfo 打印正等候回收的对象的信息.
-heap 打印heap的概要信息，GC使用的算法，heap的配置及wise heap的使用情况.
-histo[:live] 打印每个class的实例数目,内存占用,类全名信息. VM的内部类名字开头会加上前缀”*”. 如果live子参数加上后,只统计活的对象数量.
-permstat 打印classload和jvm heap长久层的信息. 包含每个classloader的名字,活泼性,地址,父classloader和加载的class数量. 另外,内部String的数量和占用内存数也会打印出来.
-F 强迫.在pid没有相应的时候使用-dump或者-histo参数. 在这个模式下,live子参数无效.
-h | -help 打印辅助信息
-J 传递参数给jmap启动的jvm. E.g:
Use jmap -heap pid to view the process heap memory usage, including the GC algorithm used, heap configuration parameters, and heap memory usage in each generation:
Use jmap -histo[:live] pid to view the histogram of the number and size of objects in the heap memory.
5, jhat: jvm heap snapshot analysis tool
The jhat command is used in conjunction with jamp to analyze heap snapshots produced by map. jhat has a built-in mini http/Html server, which can be viewed in the browser. However, it is recommended not to use it as much as possible. Since there is a dumpt file, it can be pulled from the production environment and analyzed by local visualization tools, which not only reduces the pressure on the online server, but also can be analyzed in sufficient detail (such as MAT/jprofile/visualVm), etc. .
6. jstack: java stack trace tool
jstack is used to generate a thread snapshot of the current moment of the java virtual machine. A thread snapshot is a collection of method stacks that are being executed by each thread in the current Java virtual machine. The main purpose of generating a thread snapshot is to locate the cause of long-term pauses in threads, such as inter-thread deadlocks, infinite loops, and requests for external resources. Time to wait and so on.
Command format:
jstack [ option ] pid
jstack [ option ] executable core
jstack [ option ] [server-id@]remote-hostname-or-IP
parameter:
-F当’jstack [-l] pid’没有相应的时候强制打印栈信息
-l长列表. 打印关于锁的附加信息,例如属于java.util.concurrent的ownable synchronizers列表.
-m打印java和native c/c++框架的所有栈信息.
-h | -help打印帮助信息
pid 需要被打印配置信息的java进程id,可以用jps查询.Subsequent searches will be used in the example of the highest cpu cost.
2. Visualization tools
Common visualization tools for jvm monitoring, in addition to Jconsole and visualVm provided by jdk itself, there are jprofilter, perfino, Yourkit, Perf4j, JProbe, MAT, etc. provided by third parties. These tools have greatly enriched our way of positioning and optimizing JVM.
There are many tutorials on the Internet for the use of these tools, so I won't introduce them too much here. For VisualVm, it is recommended to use. In addition to being less intrusive to jvm, it is also developed by the jdk team. I believe that the functions will be more abundant and perfect in the future. For third-party monitoring tools, jprofilter provides the most complete functions and visualization. Currently, most IDEs support its plug-ins, which can be used together for debugging and performance tuning before going online.
In addition, for the heap information of the online dump, you should try to pull it offline for analysis with visualization tools, so that the analysis is more detailed. If some urgent problems must be monitored online, you can use the remote function of VisualVm to do it, which requires the use of the MAT function under tool.jar.
3. Application
1. The cpu soars
Sometimes online, there may be a problem that the CPU suddenly soars at a certain moment in the application. In this regard, we should be familiar with some instructions and quickly check the corresponding code.
1. Find the process that consumes the most CPU
instruction:
top
2. Find the thread that consumes the most CPU under the process
instruction:
top -Hp pid
3. Convert base
printf “%x\n” 15332 // 转换16进制（转换后为0x3be4） 4. Filter the specified thread and print the stack information
instruction
jstack pid |grep 'threadPid' -C5 --color
jstack 13525 |grep '0x3be4' -C5 --color // 打印进程堆栈 并通过线程id，过滤得到线程堆栈信息。
It can be seen that it is a reporting program that occupies too much CPU (the above example is only an example, and the CPU consumption itself is not high)
2. Thread deadlock
Sometimes deployment scenarios will have thread deadlock problems, but it is not common. At this point, we use jstack to check it out. For example, we now have a thread deadlocked program, causing some operations to wait.
1. Find the java process id
instruction:
top 或者 jps 
2. View the thread snapshot information of the java process
instruction:
jstack -l pid
As you can see from the output, there is a thread deadlock, and that line of code appears. This allows for quick troubleshooting.
3. OOM memory leak
The OOM exception in the java heap is a common memory overflow exception in practical applications. Generally, we first analyze the dumped heap dump snapshot through a memory mapping analysis tool (such as MAT) to confirm whether there is a problem with the objects in the memory.
Of course, there are many reasons for OOM, and it is not a case of insufficient application resources in the heap. It is also possible that too many resources are applied for and not released, or the system resources are exhausted due to frequent applications. For these three cases, I need to investigate one by one.
Three cases of OOM:
1. The application resource (memory) is too small and not enough.
2. Too many application resources are not released.
3. Too many application resources and exhausted resources. For example: too many threads, too large thread memory, etc.
1. Troubleshoot the application for application resources.
指令:jmap -heap 11869 Check the allocation size and usage of the new generation and old generation heap memory to see if the allocation itself is too small.
From the above investigation, it is found that there is no problem with the memory requested by the program.
2. Check gc
Especially in the case of fgc, the memory situation of each generation.
指令:jstat -gcutil 11938 1000 每秒输出一次gc的分代内存分配情况，以及gc时间
3. Find the most memory-intensive object
指令: jmap -histo:live 11869 | more
In the above output information, the maximum memory object is only 161kb, which belongs to the normal range. If an object occupies a large space, such as more than 100Mb, you should focus on analyzing why it is not released.
Note that the above command:
jmap -histo:live 11869 | more
执行之后，会造成jvm强制执行一次fgc，在线上不推荐使用，可以采取dump内存快照，线下采用可视化工具进行分析，更加详尽。
jmap -dump:format=b,file=/tmp/dump.dat 11869
或者采用线上运维工具，自动化处理，方便快速定位，遗失出错时间。
4. Confirm whether resources are exhausted
- pstree View the number of process threads
- netstat to view the number of network connections
or use:
- ll /proc/${PID}/fd | wc -l // number of open handles
- ll /proc/${PID}/task | wc -l (same effect as pstree -p | wc -l) //Number of open threads
The above are some common jvm command applications.
The application of a tool is not a panacea, it can cure all diseases. The solution of the problem often requires the combination of multiple tools to better locate the problem. No matter what analysis tool is used, the most important thing is to be familiar with the advantages of each tool. and disadvantages. In this way, we can learn from each other and use them together.
-----------------------------------------------------------------------------
If you want to see more interesting and original technical articles, scan and follow the official account.
Focus on personal growth and game development, and promote the growth and progress of the domestic game community.
