Java thread dump summary

When the server hangs, crashes or is under performance, it is necessary to capture the server's thread stack (Thread Dump) for subsequent analysis.

Thread dump provides a snapshot of the currently active threads. It provides the stack of all Java threads in the JVM Tracking information             

1. How to get Thread dump
  1) Execute jps in javaHome/bin directory (get thread pid)
   2) jstack -l pid > deadlock.txt
will produce deadlock.txt file in bin directory

(you can use
linux command ps aux What is the difference between |grep java and ps -ef|grep java?
ps aux is to display the java process in BSD format. The
displayed items are: USER , PID , %CPU , %MEM , VSZ , RSS , TTY , STAT , START , TIME , COMMAND
ps -ef is to display the java process in a standard format. The
displayed items are: UID , PID , PPID , C , STIME , TTY , TIME , CMD
)
2. How to see thread CPU
  1) top -H
  2) top –H -p pid (specific)
  There is no top command under the window, please refer to http://blog.csdn.net/hexin373/article/details/8846919
as follows:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                            
4280 nbg-syst  18   0 3608m 2.0g  21m R 93.6 25.9   5004:49 java                                               
4279 nbg-syst  18   0 3608m 2.0g  21m R 92.6 25.9   4876:40 java                                               
4281 nbg-syst  18   0 3608m 2.0g  21m R 92.6 25.9   3892:54 java                                                
4282 nbg-syst  18   0 3608m 2.0g  21m R 91.2 25.9   4954:40 java                                               
4244 nbg-syst  15   0 3608m 2.0g  21m S  3.3 25.9 168:34.04 java                                                      
PID所在的列即是对应的线程ID，这是十进制的。

最后：找到耗费CPU高的线程及对应的源代码
Take the PID 4280 of the first line that consumes the most CPU above, and convert it to hexadecimal to get 0x10b8. Then search for 0x10b8 in the thread dump log, and you will find the following information:
"Stack.ClientSelector-1" daemon prio=10 tid=0x000000004baeec00 nid= 0x10b8 runnable [0x0000000053169000..0x0000000053169c90] Thread status dump file



in jstack Dump log file Here,
the thread states worth paying attention to are:
Deadlock, Deadlock (focus)
executing, Runnable  
waiting for resources, Waiting on condition (focus)
waiting to acquire a monitor, Waiting on monitor entry (focus)
pause, Suspended
object waiting , Object.wait() or TIMED_WAITING
blocks, Blocked (focus) 
stops, Parked
comprehensive demonstration 1: Waiting to lock and Blocked
examples are as follows:
"RMI TCP Connection(267865)-172.16.5.25" daemon prio=10 tid=0x00007fd508371000 nid =0x55ae waiting for monitor entry [0x00007fd4f8684000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:201)
- waiting to lock <0x00000000acf4d0c0> (a org.apache.log4j.Logger)
at org .apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java :234)
at com.tuan.core.common.lang.cache.remote.SpyMemcachedClient.get(SpyMemcachedClient.java:110)
……
1) The thread state is Blocked, blocking state. Explain that the thread waits for the resource to time out!
2) "waiting to lock <0x00000000acf4d0c0>" means that the thread is waiting to lock this 0x00000000acf4d0c0 address (in English, it can be described as: trying to obtain 0x00000000acf4d0c0 lock).
3) Find the string 0x00000000acf4d0c0 in the dump log, and find that a large number of threads are waiting to lock this address. If you can find out who got the lock in the log (eg locked < 0x00000000acf4d0c0 >), you can follow the clues.
4) "waiting for monitor entry" indicates that this thread has entered the critical section through synchronized(obj) {...} application, thus entering the "Entry Set" queue in Figure 1 below, but the monitor corresponding to the obj is owned by other threads , so the thread waits in the Entry Set queue.
5) In the first line, "RMI TCP Connection(267865)-172.16.5.25" is the Thread Name. tid refers to the Java Thread id. nid refers to the id of the native thread. prio is the thread priority. [0x00007fd4f8684000] is the starting address of the thread stack.

The meaning and precautions of the thread status in the Dump file The
meaning is as follows:

Deadlock: Deadlock thread, generally refers to the situation where multiple threads call each other and enter into mutual resource occupation, resulting in the situation that the thread cannot be released.
Runnable: Generally means that the thread is in the execution state, the thread occupies resources, is processing a request, may be passing SQL to the database for execution, may be operating on a file, may be converting data types, etc.
Waiting on condition: Waiting for a resource, or waiting for a condition to occur. The specific reasons need to be analyzed in combination with stacktrace.
If the stack information is clearly application code, it proves that the thread is waiting for a resource. Generally, when a large number of resources are read, and the resource uses a resource lock, the thread enters the waiting state and waits for the reading of the resource.
Or, it is waiting for the execution of other threads, etc.
If it is found that a large number of threads are in the Wait on condition, from the thread stack, they are waiting for network reads and writes, which may be a symptom of a network bottleneck. The thread cannot execute because of network congestion.
One situation is that the network is very busy, almost consuming all the bandwidth, and there is still a large amount of data waiting for the network to read and write;
another situation may be that the network is idle, but due to routing and other issues, packets cannot arrive normally.
Another common situation where the Wait on condition occurs is that the thread is sleeping and will be woken up when the time to wait for sleep is up.
Blocked: The thread is blocked, which means that during the execution of the current thread, the required resources have been waiting for a long time but have not been obtained, and are marked as blocked by the thread manager of the container, which can be understood as a thread waiting for the resource to time out.
Waiting for monitor entry and in Object.wait(): Monitor is the main method in Java to achieve mutual exclusion and cooperation between threads. It can be regarded as an object or Class lock. Every object has and only has one monitor. As can be seen from Figure 1 below, each Monitor can only be owned by one thread at a certain time. This thread is "Active Thread", while other threads are "Waiting Thread", which are in two queues "Entry Set" respectively. ” and “Wait Set”. The thread state waiting in "Entry Set" is "Waiting for monitor entry", and the thread state waiting in "Wait Set" is "in Object.wait()".

Comprehensive demonstration 2: Waiting on condition and TIMED_WAITING
examples are as follows:
"RMI TCP Connection(idle)" daemon prio=10 tid=0x00007fd50834e800 nid=0x56b2 waiting on condition [0x00007fd4f1a59000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000acd84de8> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
1) timed_waiting in "TIMED_WAITING (parking)" refers to the waiting state, but The time is specified here, and the waiting state is automatically exited when the specified time is reached; parking refers to the thread being suspended.

2) "waiting on condition" needs to be combined with "parking to wait for <0x00000000acd84de8> (a java.util.concurrent.SynchronousQueue$TransferStack)" in the stack. First, the thread must be waiting for a condition to occur to wake itself up. Secondly, SynchronousQueue is not a queue, but a mechanism for transferring information between threads. When we put an element into SynchronousQueue, another thread must be waiting for the task to be handed over, so this is the condition that this thread is waiting.
3) Nothing else can be seen.
Comprehensive demonstration 3: The example of in Obejct.wait() and TIMED_WAITING

is as follows:
"RMI RenewClean-[172.16.5.19:28475]" daemon prio=10 tid=0x0000000041428800 nid=0xb09 in Object.wait() [0x00007f34f4bd0000]
   java.lang. Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000aa672478> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked < 0x00000000aa672478> (a java.lang.ref.ReferenceQueue$Lock)
at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
at java.lang.Thread.run(Thread.java:662)
1) "TIMED_WAITING (on object monitor)", for this example, it enters the waiting state because the thread calls java.lang.Object.wait(long timeout).

2) The waiting thread state in "Wait Set" is " in Object.wait() ". When the thread obtains the Monitor and enters the critical section, if it finds that the conditions for the thread to continue running are not met, it calls the wait() method of the object (usually the synchronized object), abandons the Monitor, and enters the "Wait Set" queue . Only when other threads call notify() or notifyAll() on the object, the threads in the "Wait Set" queue get the opportunity to compete, but only one thread obtains the object's Monitor and returns to the running state.

3) RMI RenewClean is part of DGCClient. DGC refers to Distributed GC, which is distributed garbage collection.

4) Please note that it is first locked <0x00000000aa672478>, and then waiting on <0x00000000aa672478>. The reason why it is locked first and then equals an object, please see its code implementation below:
static private class Lock { };
private Lock lock = new Lock( );
public Reference<? extends T> remove(long timeout)
{
    synchronized (lock) {
        Reference<? extends T> r = reallyPoll();
        if (r != null) return r;
        for (;;) {
            lock. wait(timeout);
            r = reallyPoll();
            …
       }
}
That is, during the execution of the thread, the Monitor of this object is first obtained by synchronized (corresponding to locked <0x00000000aa672478>); when the execution reaches lock.wait(timeout);, the thread gives up the ownership of the Monitor and enters the "Wait Set" queue (corresponds to waiting on <0x00000000aa672478> ).
5) From the stack information, it is clearing the remote references to remote objects, the lease of the reference has arrived, and the distributed garbage collection is cleaning up one by one.





Thread info block:
1. "Timer-0" daemon prio=10tid=0xac190c00 nid=0xaef in Object.wait() [0xae77d000]
2. java.lang.Thread.State: TIMED_WAITING (on object monitor)
3. atjava. lang.Object.wait(Native Method)
4. -waiting on <0xb3885f60> (a java.util.TaskQueue) ###Continue to wait
5. atjava.util.TimerThread.mainLoop(Timer.java:509)
6. -locked <0xb3885f60> (a java.util.TaskQueue) ###locked
7. atjava.util.TimerThread.run(Timer.java:

* Thread type: daemon
* Priority: 10, the default is 5
* jvm thread id: tid=0xac190c00, the unique identifier of the jvm internal thread (obtained by java.lang.Thread.getId(), usually implemented by auto-increment.)
* Corresponding system thread id (NativeThread ID): nid=0xaef, which corresponds to the thread pid viewed by the top command, but one is in decimal and the other is in hexadecimal. (Through the command: top -H -p pid, you can view all thread information of the process)
* Thread status: in Object.wait().
* Starting stack address: [0xae77d000]
* Java thread statck trace: yes 2- 7 lines of information. This is by far the most important data, and the Java stack trace provides most of the information to pinpoint the source of the problem.

For thread dump information, the main concern is the state of the thread and its execution stack. Now explain these two key parts:
1) Detailed explanation of Java thread statck trace:
The stack information should be interpreted in reverse: the program executes the 7th line first, then the 6th line, and so on.
- locked <0xb3885f60> (a java.util.ArrayList)
- waiting on <0xb3885f60> (a java.util.ArrayList)
That is to say, the object is locked first, locks the object 0xb3885f60, and then releases the object lock and enters the waiting state.
Why does this happen? Take a look at the following java code example to understand:
synchronized(obj) { 
       ......... 
       obj.wait(); 
       ......... 
} 
Resources:
http://blog.csdn.net/hanxingwang0806/article/details/41849681
http://blog.csdn.net/rachel_luo/article/details/8920596
http://blog.csdn.net/gobitan/article/details/5474062