Continue to talk about optimization must know jvm series - monitoring tool

Personal blog navigation page (click on the right link to open a personal blog): Daniel take you on technology stack 

Previous through the jvm garbage collection knowledge, we understand the jvm memory allocation and garbage collection is how to deal with. Theory is a tool to guide practice, with theoretical guidance, positioning of the time, knowledge and experience is the key basis, the data can provide the basis for us.

In a common online problem when most of us will experience the following problems:

• Memory leak
• A process cpu sudden surge
• Thread deadlock
• Slow to respond ... and so on other issues.

If you encounter this problem above, there can be a variety of online tools to support local view, but to line up, not so much to support local debugging tools, how do we based monitoring tool to locate the problem?

We usually based on data collected location, and data collection is inseparable from the process monitoring tools, such as: running logs, exception stack, GC logs, thread snapshots, heap snapshots. Often using the appropriate analysis and monitoring tools can speed up our data analysis, positioning speed to solve the problem. Below we will explain the details.

 

A, jvm common monitoring tools & instructions

 

1, jps: jvm process status tool

jps [options] [hostid]

 If you do not specify it defaults to the current host hostid or a server. 

 Command line parameter options are as follows: 

-q 不输出类名、Jar名和传入main方法的参数

- l 输出main类或Jar的全限名

-m 输出传入main方法的参数

- v 输出传入JVM的参数

E.g:

 

2, jstat: jvm statistics monitoring tool

jstat is the command to run the virtual machine status information for a variety of knowledge-line tool. It can display local or remote virtual machine in the process of class loading, memory, garbage collection, jit compiler and other operating data, it is the tool of choice for online positioning jvm performance.

Format:

jstat [ generalOption | outputOptions vmid [interval[s|ms] [count]] ]

generalOption - 单个的常用的命令行选项，如-help, -options, 或 -version。

outputOptions -一个或多个输出选项，由单个的statOption选项组成，可以和-t, -h, and -J等选项配合使用。

Parameter options:

Option	Displays	Ex
class	View statistics for class loading conditions	jstat -class pid: displaying information on the number of loaded class, and occupied space.
compiler	View statistics in real-time compiler to compile case HotSpot	jstat -compiler pid: VM display real-time information such as the number of compilation.
gc	View JVM garbage collection situation heap of statistics	jstat -gc pid: gc information can be displayed to view the number of gc, and time. The last five, respectively, the number of young gc, young gc time, the number of full gc, full gc time, total time of gc.
gccapacity	View storage capacity is the new generation, the old generation and permanent generation	jstat -gccapacity: may be displayed, using VM memory three generations (young, old, perm) occupied by the object size and
gccause	View statistics garbage collection (this option and -gcutil as), if garbage collection occurs, it will display the last time and the reasons for garbage collection is currently taking place.	jstat -gccause: gc show reason
gcnew	View the situation of the new generation garbage collection	jstat -gcnew pid: information new object
gcnewcapacity	View case for a new generation of storage capacity	jstat -gcnewcapacity pid: Information and footprint of new objects
gcold	View the case for the old generation and permanent generation occurrence of GC	jstat -gcold pid: Information old objects
gcoldcapacity	Older Generation for viewing capacity	jstat -gcoldcapacity pid: information and its footprint of old objects
gcpermcapacity	For viewing the permanent generation capacity	jstat -gcpermcapacity pid: information and its footprint perm objects
gcutil	Situation View the new generation, on behalf of the holders of the old generation and garbage collection	jstat -util pid: Statistical Information Statistics gc
printcompilation	HotSpot compilation of statistical methods	jstat -printcompilation pid: VM execution of current information

For example :

View gc implementation: jstat-gcutil 27777

 

3, jinfo: java configuration information

Format:

jinfo[option] pid

For example: Get jvm running and starting some of the current process.

 

4, jmap: java memory mapping tool

jmap order for the production of heap dump snapshot. Print out a java process (using pid), all the 'objects' in the memory (such as: produce those objects, and their number).

Format:

jmap [ option ] pid

jmap [ option ] executable core

jmap [ option ] [server-id@]remote-hostname-or-IP

 

Parameter options:

-dump:[live,]format=b,file=<filename> 使用hprof二进制形式,输出jvm的heap内容到文件=. live子选项是可选的，假如指定live选项,那么只输出活的对象到文件. 

-finalizerinfo 打印正等候回收的对象的信息.

-heap 打印heap的概要信息，GC使用的算法，heap的配置及wise heap的使用情况.

-histo[:live] 打印每个class的实例数目,内存占用,类全名信息. VM的内部类名字开头会加上前缀”*”. 如果live子参数加上后,只统计活的对象数量. 

-permstat 打印classload和jvm heap长久层的信息. 包含每个classloader的名字,活泼性,地址,父classloader和加载的class数量. 另外,内部String的数量和占用内存数也会打印出来. 

-F 强迫.在pid没有相应的时候使用-dump或者-histo参数. 在这个模式下,live子参数无效. 

-h | -help 打印辅助信息 

-J 传递参数给jmap启动的jvm. 

E.g:

Use jmap -heap pid view the process heap memory usage, including the use of GC algorithm, heap configuration parameters, and each generation heap memory usage:

Use jmap -histo [: live] pid to see the number of objects in the heap memory, the size of the histogram.

5, jhat: jvm heap snapshot analysis tools

jhat command jamp with the use of fast storage used to analyze heap snapshot map production. jhat built a miniature http / Html server, you can find the view in the browser. However, the proposed try to do, since there dumpt file, you can pull from the production environment down, and then analyzed by local visualization tools, both to reduce the line pressure on the server, there can be analyzed in sufficient detail (such as MAT / jprofile / visualVm), etc. .

6, jstack: java stack trace tool

jstack thread used to generate java virtual machine snapshot of the current time. A snapshot is a method in the current thread java virtual machine each executing thread stack collection, the main purpose of generating a snapshot of the thread is to locate the thread appears long pause, such as inter-thread deadlocks, infinite loop, long requested external resources caused by time to wait so on.

Format:

jstack [ option ] pid

jstack [ option ] executable core

jstack [ option ] [server-id@]remote-hostname-or-IP

 

parameter:

-F当’jstack [-l] pid’没有相应的时候强制打印栈信息

-l长列表. 打印关于锁的附加信息,例如属于java.util.concurrent的ownable synchronizers列表.

-m打印java和native c/c++框架的所有栈信息.

-h | -help打印帮助信息

pid 需要被打印配置信息的java进程id,可以用jps查询.

Find the most cpu-consuming subsequent examples will be used.

 

Second, visualization tools

Jvm common visualization tools for monitoring, in addition to Jconsole and visualVm jdk itself provides, as well as third-party jprofilter, perfino, Yourkit, Perf4j, JProbe, MAT and so on. These tools have greatly enriched our positioning and optimization jvm way.

Using these tools, there are many online tutorials provided here will not introduce too much. For VisualVm, the comparison is recommended that in addition to the jvm invasive lower than, or jdk own development team, I believe the future will be more feature rich and perfect. jprofilter for third-party monitoring tool that provides functionality and visualization the most complete, most current ide support its plug-ins for debugging and performance tuning before the on-line can be used together.

In addition to the dump heap line information, the line should try to pull to visualization tools for analyzing, so that a more detailed analysis. If for some urgent issues need to be online monitoring, remote function can be used to VisualVm, which requires the use MAT functions in tool.jar.

 

Third, the application

 

1, cpu soar

Sometimes some point, the application may be a problem at some point there will be a sudden surge in cpu online. Which we should be familiar with some instructions, quick investigation and the corresponding code.

 

1. Locate the most CPU-intensive process

instruction:

top

 

2. Locate this process consuming the most cpu thread

instruction:

top -Hp pid

3. Conversion hex

printf “%x\n” 15332 // 转换16进制（转换后为0x3be4） 

4. Filter the specified thread, the print stack information

instruction

jstack pid |grep 'threadPid'  -C5 --color 

jstack 13525 |grep '0x3be4'  -C5 --color  //  打印进程堆栈 并通过线程id，过滤得到线程堆栈信息。

A reporting program may be seen, the cpu occupancy excessive (above examples are exemplary only, not high cpu cost itself)

 

2, thread deadlock 

Sometimes there will be deployment scenario thread deadlock problems, but not common. At this time, we used to see under what jstack. For example, we now have a thread deadlock of the program, resulting in some operations in waiting.

 

1. Find the java process id

instruction:

top 或者 jps 

 

2. Check the java process threads snapshot information

instruction:

jstack -l pid

You can see from the output, there is a thread deadlock occurred, and pointed out that the line of code that appears. So you can quickly troubleshoot problems.

 

3, OOM memory leaks

OOM in java heap exceptions are the practical application of the common memory overflow exception. Generally, we are the first out of the analysis of the dump heap dump snapshots of memory mapping analysis tools (such as MAT), to confirm whether the object in memory problems.

There are of course many reasons for the emergence of the OOM, not the application heap resources shortage situation. It is also possible to apply too many resources are not released, or frequent frequent applications, system resources are exhausted. For the three cases I need one investigation.

 

OOM three cases:

1. Application of resources (memory) is too small, not enough.

2. Apply for too many resources, there is no release.

3. apply too many resources, resource exhaustion. For example: too many threads, the thread too much memory and so on.

 

1. Troubleshoot application application resource problems.

指令:jmap -heap 11869 

 View the new generation, the size distribution of the old generation heap memory usage and see if the distribution itself is too small.

From the above investigation found no problem with the application program memory.

2. investigation gc

Fgc is especially the case, where each generational memory.

指令:jstat -gcutil 11938 1000 每秒输出一次gc的分代内存分配情况，以及gc时间

3. Find the most expensive memory object

指令: jmap -histo:live 11869 | more

Said output information, only 161KB maximum memory objects, within the normal range. If an object is a large space, such as more than 100Mb, analysis should focus on the reasons for not releasing.

 

Note that, the command:

jmap -histo:live 11869 | more

执行之后，会造成jvm强制执行一次fgc，在线上不推荐使用，可以采取dump内存快照，线下采用可视化工具进行分析，更加详尽。

jmap -dump:format=b,file=/tmp/dump.dat 11869 

或者采用线上运维工具，自动化处理，方便快速定位，遗失出错时间。

 

4. Confirm whether the resources are exhausted

• pstree see the number of process threads
• netstat view the number of network connections

Or to use:

• ll / proc / $ {PID} / fd | wc -l // number of handles open
• ll / proc / $ {PID} / task | wc -l (the same effect pstree -p | wc -l) // number of threads open

 

These are some of the common commands jvm application.

Application of a tool is not the master key, cure, problem-solving is often the need to combine a variety of tools to better locate the problem, regardless of the analysis tools, the most important thing is familiar with the advantages of each tool and weaknesses. So as to learn from each other, with the use.

Attached Java / C / C ++ / machine learning / Algorithms and Data Structures / front-end / Android / Python / programmer reading / single books books Daquan:

(Click on the right to open there in the dry personal blog): Technical dry Flowering
===== >> ① [Java Daniel take you on the road to advanced] << ====
===== >> ② [+ acm algorithm data structure Daniel take you on the road to advanced] << ===
===== >> ③ [database Daniel take you on the road to advanced] << == ===
===== >> ④ [Daniel Web front-end to take you on the road to advanced] << ====
===== >> ⑤ [machine learning python and Daniel take you entry to the Advanced Road] << ====
===== >> ⑥ [architect Daniel take you on the road to advanced] << =====
===== >> ⑦ [C ++ Daniel advanced to take you on the road] << ====
===== >> ⑧ [ios Daniel take you on the road to advanced] << ====
=====> > ⑨ [Web security Daniel take you on the road to advanced] << =====
===== >> ⑩ [Linux operating system and Daniel take you on the road to advanced] << = ====

There is no unearned fruits, hope you young friends, friends want to learn techniques, overcoming all obstacles in the way of the road determined to tie into technology, understand the book, and then knock on the code, understand the principle, and go practice, will It will bring you life, your job, your future a dream.