Author: Transfer from Original Church Java: https://www.javatang.com/archives/2017/10/26/08572060.html
Fault Analysis and Performance Optimization series JVM
Failure analysis and performance optimization one JVM Series: jstack positioning thread stack information
JVM failure analysis and performance optimization of the two series: jstack generated Thread Dump log structure elucidation
three JVM failure analysis and performance optimization series: the use of command and VM jstat thread analysis
JVM failure analysis and performance optimization series on: jstack generated thread Dump log thread state
JVM failure analysis and performance optimization series of five: common thread Dump log case Study
six JVM failure analysis and performance optimization series: JVM Heap Dump use (heap dump file) generation and MAT of
JVM failure analysis and performance optimization series of seven: use the Histogram and Dominator Tree MAT locate the source overflow
Contents [ hide ]
Symptoms and Solutions
Here are solutions to some common symptoms that is corresponding to:
CPU utilization is high, the response is very slow
According to "Java memory leak analysis one Series: jstack positioning thread stack information" in the said method, first find the CPU-intensive process, and then navigate to the corresponding thread, and finally analyze the information corresponding to the stack.
At the same time repeatedly using the above method, and then were analyzed, where to find the cause of the problem from the code. If the thread is pointed to "VM Thread" or directly from the code can not find the reason, it is necessary for memory analysis, a specific article below.
CPU occupancy rate is not high, but the response is very slow
In the process of performing the entire request multiple Thread Dump then compared to obtain BLOCKEDa state list of threads, because the thread is typically stopped at the I / O, database or network connection place.
Focus Overview
In Thread Dump file, the state of the thread into two: Native the Thread Status and the JVM the Thread Status , specific meaning can refer to the previous article. Below are some common threads need to focus state at the time of log analysis:
System thread status deadlock
Threads are deadlocked, it will take up a lot of system resources.
The system thread status is waiting for monitor entry or in Object.wait ()
As stated in an article, the system thread is in this state that it is waiting to enter a critical section, then the state JVM thread are usually java.lang.Thread.State: BLOCKED.
If the number of threads in this state, it may be a global lock that blocked a large number of threads. If multiple print Thread Dump information in the short term, found waiting for monitor entrya thread state more and more, not decreasing trend could mean some threads in the critical zone for too long a time, so that more and more new thread late late inaccessible.
The system thread status is waiting on condition
The system thread is in this state that it is waiting for another condition occurs to wake himself, or that he issued the sleep () method. The status JVM thread is typically java.lang.Thread.State: WAITING (parking) (wake-up wait condition) or java.lang.Thread.State: TIMED_WAITING (parking or Sleeping) (waiting for a timed wake-up condition).
If the number of threads in this state, indicating that these threads went to get a third-party resources, such as network resource or database read operation of a third party, for a long time can not get a response, resulting in a large number of thread into a wait state. Thus, this system is described in a database network bottlenecks or read operation time is too long.
The system thread status is blocked
The thread is blocked, need to be determined according to the actual situation.
case study
Let's get to solve the problem of decomposition by several cases.
waiting for monitor entry 和 java.lang.Thread.State: BLOCKED
"DB-Processor-13" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at beans.ConnectionPool.getConnection(ConnectionPool.java:102)
- waiting to lock <0xe0375410> (a beans.ConnectionPool)
at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111)
at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43)
"DB-Processor-14" daemon prio=5 tid=0x003edf98 nid=0xca waiting for monitor entry [0x000000000825f020]
java.lang.Thread.State: BLOCKED (on object monitor)
at beans.ConnectionPool.getConnection(ConnectionPool.java:102)
- waiting to lock <0xe0375410> (a beans.ConnectionPool)
at beans.cus.ServiceCnt.getTodayCount(ServiceCnt.java:111)
at beans.cus.ServiceCnt.insertCount(ServiceCnt.java:43)
"DB-Processor-3" daemon prio=5 tid=0x00928248 nid=0x8b waiting for monitor entry [0x000000000825d080]
java.lang.Thread.State: RUNNABLE
at oracle.jdbc.driver.OracleConnection.isClosed(OracleConnection.java:570)
- waiting to lock <0xe03ba2e0> (a oracle.jdbc.driver.OracleConnection)
at beans.ConnectionPool.getConnection(ConnectionPool.java:112)
- locked <0xe0386580> (a java.util.Vector)
- locked <0xe0375410> (a beans.ConnectionPool)
at beans.cus.Cue_1700c.GetNationList(Cue_1700c.java:66)
at org.apache.jsp.cue_1700c_jsp._jspService(cue_1700c_jsp.java:120)State of the system is above the thread waiting for monitor entry, this thread is described by synchronized (obj) {} apply to enter the critical section, but the corresponding obj Monitor owned by another thread, the state JVM threads is java.lang.Thread.State: BLOCKED (on object monitor), resource timeout thread waits for instructions.
The following waiting to lock <0xe0375410>description of threads waiting for 0xe0375410the lock address (trying to obtain 0xe0375410 lock), if it is found in the log has a large number of threads are waiting to 0xe0375410lock it, this time need to find the thread that acquired the lock in the log locked <0xe0375410>, as the above example is "DB-Processor-14" thread, so you can follow it up. The above example is because access to the database operation due to wait too long, this time on the need to modify the configuration information for the database connection.
If two threads are locked to each other the other thread lock, thus causing the 死锁phenomenon, as in the following example:
waiting on condition 和 java.lang.Thread.State: TIMED_WAITING
"RMI TCP Connection(idle)" daemon prio=10 tid=0x00007fd50834e800 nid=0x56b2 waiting on condition [0x00007fd4f1a59000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000acd84de8> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)JVM thread state is java.lang.Thread.State: TIMED_WAITING (parking), the thread described in the state of waiting for the timing, parking means in the thread is suspended.
waiting on conditionIt requires a combination of the stack parking to wait for <0x00000000acd84de8> ( a java.util.concurrent.SynchronousQueue $ TransferStack) analysis together. First of all, this thread is definitely waiting for the occurrence of a condition to wake up to themselves. Secondly, SynchronousQueue not a queue, but the mechanism of transfer of information between threads, when we put an element into the SynchronousQueue in time must have another thread is waiting to accept the transfer of tasks, so it is waiting for the conditions in this thread .
in Object.wait() 和 java.lang.Thread.State: TIMED_WAITING
"RMI RenewClean-[172.16.5.19:28475]" daemon prio=10 tid=0x0000000041428800 nid=0xb09 in Object.wait() [0x00007f34f4bd0000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000aa672478> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00000000aa672478> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
at java.lang.Thread.run(Thread.java:662) JVM thread state in this case is java.lang.Thread.State: TIMED_WAITING (on object monitor)described a thread invokes java.lang.Object.wait(long timeout)a method to enter the waiting state.
The thread state "Wait Set" is waiting in Object.wait(), when the thread acquires the Monitor enter the critical section, the thread continues to run if the condition is found not satisfied, it is the calling object (usually the synchronized object) of the wait () method, giving up the Monitor, enter "Wait Set" queue. Only when another thread calls notify () or notifyAll () method on the object, "Wait Set" queue threads before they get the chance to compete, but only one thread Monitor to get the object, return to the running state.
Also important to note is that, is the first locked <0x00000000aa672478> then Waiting ON <0x00000000aa672478> , is so, can be demonstrated by the following code:
static private class Lock { };
private Lock lock = new Lock();
public Reference<? extends T> remove(long timeout) {
synchronized (lock) {
Reference<? extends T> r = reallyPoll();
if (r != null) return r;
for (;;) {
lock.wait(timeout);
r = reallyPoll();
// ……
}
}线程在执行的过程中，先用 synchronized 获得了这个对象的 Monitor（对应 locked <0x00000000aa672478>），当执行到 lock.wait(timeout); 的时候，线程就放弃了Monitor的所有权，进入 "Wait Set" 队列（对应 waiting on <0x00000000aa672478>）。
前面几篇文章详细说明了如何分析Thread Dump文件，除此之外还可以通过分析JVM堆内存信息来进一步找到问题的原因。
参考资料：
性能分析之-- JAVA Thread Dump 分析综述
三个实例演示 Java Thread Dump 日志分析
如何分析Java虚拟机死锁
各种 Java Thread State 第一分析法则
How to Analyze Java Thread Dumps | 中文版