Thread Dump分析

摘自：http://blog.csdn.net/wanyanxgf/article/details/6944987

thread dump获取1. 发送信号
* In Unix, use "kill -3 <pid>" where pid is the Process ID of the JVM.（kill 信号列表）
* In Windows, press CTRL+BREAK on the window where the JVM is running.

2. jstack -l pid. 

(1). jstack入口

[java] view plain copy print ?

1. public class JStack {  
2.             public static void main(String[] args) throws Exception {  
3.                ...  
4.                     String pid = args[optionCount];  
5.                     String params[];  
6.                     if (locks) {  
7.                         params = new String[] { "-l" };  
8.                     } else {  
9.                         params = new String[0];  
10.                     }  
11.                     runThreadDump(pid, params);  
12.                 ...  
13.             }  
14.             // Attach to pid and perform a thread dump    
15.             private static void runThreadDump(String pid, String args[])  
16.                     throws Exception {  
17.                 VirtualMachine vm = null;  
18.                 try {  
19.                     vm = VirtualMachine.attach(pid);  
20.                 } catch (Exception x) {  
21.                    ...  
22.                 }  
23.                 // Cast to HotSpotVirtualMachine as this is implementation specific   
24.                 // method.   
25.                 InputStream in = ((HotSpotVirtualMachine) vm)  
26.                         .remoteDataDump((Object[]) args);  
27.                 // read to EOF and just print output    
28.                 byte b[] = new byte[256];  
29.                 int n;  
30.                 do {  
31.                     n = in.read(b);  
32.                     if (n > 0) {  
33.                         String s = new String(b, 0, n, "UTF-8");  
34.                         System.out.print(s);  
35.                     }  
36.                 } while (n > 0);  
37.                 in.close();  
38.                 vm.detach();  
39.             }  

public class JStack {
            public static void main(String[] args) throws Exception {
               ...
                    String pid = args[optionCount];
                    String params[];
                    if (locks) {
                        params = new String[] { "-l" };
                    } else {
                        params = new String[0];
                    }
                    runThreadDump(pid, params);
                ...
            }
            // Attach to pid and perform a thread dump 
            private static void runThreadDump(String pid, String args[])
                    throws Exception {
                VirtualMachine vm = null;
                try {
                    vm = VirtualMachine.attach(pid);
                } catch (Exception x) {
                   ...
                }
                // Cast to HotSpotVirtualMachine as this is implementation specific
                // method.
                InputStream in = ((HotSpotVirtualMachine) vm)
                        .remoteDataDump((Object[]) args);
                // read to EOF and just print output 
                byte b[] = new byte[256];
                int n;
                do {
                    n = in.read(b);
                    if (n > 0) {
                        String s = new String(b, 0, n, "UTF-8");
                        System.out.print(s);
                    }
                } while (n > 0);
                in.close();
                vm.detach();
            }

(2).执行方式
LinuxVirtualMachine.execute

[java] view plain copy print ?

1. /** 
2.     * Execute the given command in the target VM. 
3.     */  
4.    @Override  
5.    InputStream execute(String cmd, Object... args) throws AgentLoadException, IOException  
6.    {  
7.       ...  
8.       int s = socket();  
9.       // connect to target VM   
10.       try {  
11.          connect(s, p);  
12.       }  
13.       catch (IOException x) {  
14.          close(s);  
15.          throw x;  
16.       }  
17.       IOException ioe = null;  
18.       // connected - write request，发送command到target jvm   
19.       // <ver> <cmd> <args...>   
20.       try {  
21.          writeString(s, PROTOCOL_VERSION);  
22.          writeString(s, cmd);  
23.          for (int i = 0; i < 3; i++) {  
24.             if (i < args.length && args[i] != null) {  
25.                writeString(s, (String) args[i]);  
26.             }  
27.             else {  
28.                writeString(s, "");  
29.             }  
30.          }  
31.       }  
32.       catch (IOException x) {  
33.          ioe = x;  
34.       }  
35.       // Create an input stream to read reply.   
36.       SocketInputStream sis = new SocketInputStream(s);  
37.       ....  
38.       // Return the input stream so that the command output can be read.   
39.       return sis;  
40.    }  

/**
    * Execute the given command in the target VM.
    */
   @Override
   InputStream execute(String cmd, Object... args) throws AgentLoadException, IOException
   {
      ...
      int s = socket();
      // connect to target VM
      try {
         connect(s, p);
      }
      catch (IOException x) {
         close(s);
         throw x;
      }
      IOException ioe = null;
      // connected - write request，发送command到target jvm
      // <ver> <cmd> <args...>
      try {
         writeString(s, PROTOCOL_VERSION);
         writeString(s, cmd);
         for (int i = 0; i < 3; i++) {
            if (i < args.length && args[i] != null) {
               writeString(s, (String) args[i]);
            }
            else {
               writeString(s, "");
            }
         }
      }
      catch (IOException x) {
         ioe = x;
      }
      // Create an input stream to read reply.
      SocketInputStream sis = new SocketInputStream(s);
      ....
      // Return the input stream so that the command output can be read.
      return sis;
   }

3. jvisualvm中来thread dump.

thread dump解析

头部信息

时间，jvm信息

[java] view plain copy print ?

1. {code}  
2. 2011-11-02 19:05:06  
3. Full thread dump Java HotSpot(TM) Server VM (16.3-b01 mixed mode):  
4. {code}  

{code}
2011-11-02 19:05:06
Full thread dump Java HotSpot(TM) Server VM (16.3-b01 mixed mode):
{code}

 线程info信息块

[java] view plain copy print ?

1. {code}  
2. "Checkpointer" daemon prio=10 tid=0x68ce1c00 nid=0x7c11 in Object.wait() [0x68b5c000]  
3.    java.lang.Thread.State: WAITING (on object monitor)  
4.         at java.lang.Object.wait(Native Method)  
5.         - waiting on <0x740ad988> (a java.lang.Object)  
6.         at java.lang.Object.wait(Object.java:485)  
7.         at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)  
8.         - locked <0x740ad988> (a java.lang.Object)  
9.         at java.lang.Thread.run(Thread.java:619)  
10. {code}  

{code}
"Checkpointer" daemon prio=10 tid=0x68ce1c00 nid=0x7c11 in Object.wait() [0x68b5c000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x740ad988> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:485)
        at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)
        - locked <0x740ad988> (a java.lang.Object)
        at java.lang.Thread.run(Thread.java:619)
{code}

[java] view plain copy print ?

1. "Checkpointer" daemon prio=10 tid=0x68ce1c00 nid=0x7c11 in Object.wait() [0x68b5c000]  

"Checkpointer" daemon prio=10 tid=0x68ce1c00 nid=0x7c11 in Object.wait() [0x68b5c000]

* 线程名称:Checkpointer
* 线程类型：daemon
* 优先级:10，默认是5
* jvm线程id:jvm内部线程的唯一标识,0x68ce1c00
* 对应系统线程id:和top命令查看的pid对应，不过一个是10进制，一个是16进制。0x7c11
* 线程状态：Object.wait().
* 起始栈地址

线程状态详解

Runnable
_The thread is either running or ready to run when it gets its CPU turn._
不解释。

Wait on condition
_The thread is either sleeping or waiting to be notified by another thread._
该状态出现在线程等待某个条件的发生或者sleep。
_最常见的情况是线程在等待网络的读写，比如当网络数据没有准备好读时，线程处于这种等待状态，而一旦有数据准备好读之后，线程会重新激活，读取并处理数据。_

 Waiting for Monitor Entry and in Object.wait()

_The thread is waiting to get the lock for an object (some other thread may be holding the lock). This happens if two or more threads try to execute synchronized code. Note that the lock is always for an object and not for individual methods._
当一个线程申请进入临界区时，获取到monitor，线程将处于 “Runnable”的状态，否则，线程 DUMP会显示处于 “waiting for monitor entry”。
当线程获得了 Monitor，进入了临界区之后，如果发现线程继续运行的条件没有满足，它则调用对象（一般就是被 synchronized 的对象）的 wait() 方法，放弃了 Monitor，进入 “Wait Set”队列。只有当别的线程在该对象上调用了 notify() 或者 notifyAll() ， “ Wait Set”队列中线程才得到机会去竞争，但是只有一个线程获得对象的 Monitor，恢复到运行态。在 “Wait Set”中的线程， DUMP中表现为： in Object.wait()。
例：

[java] view plain copy print ?

1. <SPAN style="BACKGROUND-COLOR: rgb(255,255,255)"><SPAN style="COLOR: #ff6666">{code}  
2. "Timer-0" daemon prio=10 tid=0x695c3000 nid=0x7c00 in Object.wait() [0x69468000]  
3.    java.lang.Thread.State: TIMED_WAITING (on object monitor)  
4.         at java.lang.Object.wait(Native Method)  
5.         - waiting on <0x744f2850> (a java.util.TaskQueue)          ###继续wait  
6.         at java.util.TimerThread.mainLoop(Timer.java:509)  
7.         - locked <0x744f2850> (a java.util.TaskQueue)              ###已经lock到0x744f2850  
8.         at java.util.TimerThread.run(Timer.java:462)  
9. {code}</SPAN></SPAN>  

{code}
"Timer-0" daemon prio=10 tid=0x695c3000 nid=0x7c00 in Object.wait() [0x69468000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x744f2850> (a java.util.TaskQueue)          ###继续wait
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x744f2850> (a java.util.TaskQueue)              ###已经lock到0x744f2850
        at java.util.TimerThread.run(Timer.java:462)
{code}

参见：http://jameswxx.iteye.com/blog/1041173

[java] view plain copy print ?

1. {code}  
2.    java.lang.Thread.State: WAITING (on object monitor)  
3. <P style="PADDING-BOTTOM: 2px; MARGIN: 4px 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; PADDING-TOP: 2px">{code}</P>  

{code}
   java.lang.Thread.State: WAITING (on object monitor)
{code}

线程状态运行：

WAITING||State || Description||

|blocked|This thread tried to enter a synchronized block, but the lock was taken by another thread. This thread is blocked until the lock gets released.|

|blocked (on thin lock)|This is the same state as blocked, but the lock in question is a thin lock.||waiting|This thread called Object.wait() on an object. The thread will remain there until some other thread sends a notification to that object.|

|sleeping|This thread called java.lang.Thread.sleep().||parked|This thread called java.util.concurrent.locks.LockSupport.park().||suspended|The thread's execution was suspended by java.lang.Thread.suspend() or a JVMTI agent call.|

[java] view plain copy print ?

1. {code}  
2.         at java.lang.Object.wait(Native Method)  
3.         - waiting on <0x740ad988> (a java.lang.Object)        ###等待堆地址为0x740ad988的java.lang.Object对象的锁  
4.         at java.lang.Object.wait(Object.java:485)  
5.         at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)  
6.         - locked <0x740ad988> (a java.lang.Object)            ###hold住堆地址为0x740ad988的java.lang.Object对象的锁  
7.         at java.lang.Thread.run(Thread.java:619)  
8. {code}  

{code}
        at java.lang.Object.wait(Native Method)
        - waiting on <0x740ad988> (a java.lang.Object)        ###等待堆地址为0x740ad988的java.lang.Object对象的锁
        at java.lang.Object.wait(Object.java:485)
        at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)
        - locked <0x740ad988> (a java.lang.Object)            ###hold住堆地址为0x740ad988的java.lang.Object对象的锁
        at java.lang.Thread.run(Thread.java:619)
{code}

thread dump使用

首先，thread dump是个瞬时数据，需要多个采样进行对比才能更好的发现问题

cpu飙高，load高，响应很慢

方案：
* 一个请求过程中多次dump

* 对比多次dump文件的runnable线程，如果执行的方法有比较大变化，说明比较正常。如果在执行同一个方法，就有一些问题了。

cpu使用率不高但是响应很慢

方案：
* 进行dump，查看是否有很多thread struck在了i/o、数据库等地方，定位瓶颈原因。 

请求无法响应

方案：
* 多次dump，对比是否所有的runnable线程都一直在执行相同的方法，如果是的，恭喜你，锁住了！

死锁例子：

[java] view plain copy print ?

1. public class DeadLock {  
2.     public static void main(String[] args) {  
3.         final Object obj_1 = new Object(), obj_2 = new Object();  
4.         Thread t1 = new Thread("t1"){  
5.             @Override  
6.             public void run() {  
7.                 synchronized (obj_1) {  
8.                     try {  
9.                         Thread.sleep(3000);  
10.                     } catch (InterruptedException e) {}  
11.                     synchronized (obj_2) {  
12.                         System.out.println("thread t1 done.");  
13.                     }  
14.                 }  
15.             }  
16.         };  
17.         Thread t2 = new Thread("t2"){  
18.             @Override  
19.             public void run() {  
20.                 synchronized (obj_2) {  
21.                     try {  
22.                         Thread.sleep(3000);  
23.                     } catch (InterruptedException e) {}  
24.                     synchronized (obj_1) {  
25.                         System.out.println("thread t2 done.");  
26.                     }  
27.                 }  
28.             }  
29.         };  
30.         t1.start();  
31.         t2.start();  
32.     }  
33. }  

public class DeadLock {
    public static void main(String[] args) {
        final Object obj_1 = new Object(), obj_2 = new Object();
        Thread t1 = new Thread("t1"){
            @Override
            public void run() {
                synchronized (obj_1) {
                    try {
                        Thread.sleep(3000);
                    } catch (InterruptedException e) {}
                    synchronized (obj_2) {
                        System.out.println("thread t1 done.");
                    }
                }
            }
        };
        Thread t2 = new Thread("t2"){
            @Override
            public void run() {
                synchronized (obj_2) {
                    try {
                        Thread.sleep(3000);
                    } catch (InterruptedException e) {}
                    synchronized (obj_1) {
                        System.out.println("thread t2 done.");
                    }
                }
            }
        };
        t1.start();
        t2.start();
    }
}

thread dump

[java] view plain copy print ?

1. "t2" prio=10 tid=0x09d0d000 nid=0x1f17 waiting for monitor entry [0x71dad000]  
2.    java.lang.Thread.State: BLOCKED (on object monitor)  
3.         at DeadLock$2.run(DeadLock.java:30)  
4.         - waiting to lock <0x9f4c29b0> (a java.lang.Object)  
5.         - locked <0x9f4c29b8> (a java.lang.Object)  
6.    Locked ownable synchronizers:  
7.         - None  
8. "t1" prio=10 tid=0x09d22c00 nid=0x1f16 waiting for monitor entry [0x71dfe000]  
9.    java.lang.Thread.State: BLOCKED (on object monitor)  
10.         at DeadLock$1.run(DeadLock.java:15)  
11.         - waiting to lock <0x9f4c29b8> (a java.lang.Object)  
12.         - locked <0x9f4c29b0> (a java.lang.Object)  
13.    Locked ownable synchronizers:  
14.         - None  
15.   
16. Found one Java-level deadlock:  
17. =============================  
18. "t2":  
19.   waiting to lock monitor 0x09c32f18 (object 0x9f4c29b0, a java.lang.Object),  
20.   which is held by "t1"  
21. "t1":  
22.   waiting to lock monitor 0x09c32360 (object 0x9f4c29b8, a java.lang.Object),  
23.   which is held by "t2"  
24.   
25. Java stack information for the threads listed above:  
26. ===================================================  
27. "t2":  
28.         at DeadLock$2.run(DeadLock.java:30)  
29.         - waiting to lock <0x9f4c29b0> (a java.lang.Object)  
30.         - locked <0x9f4c29b8> (a java.lang.Object)  
31. "t1":  
32.         at DeadLock$1.run(DeadLock.java:15)  
33.         - waiting to lock <0x9f4c29b8> (a java.lang.Object)  
34.         - locked <0x9f4c29b0> (a java.lang.Object)  
35.   
36.   
37. Found 1 deadlock.  

"t2" prio=10 tid=0x09d0d000 nid=0x1f17 waiting for monitor entry [0x71dad000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at DeadLock$2.run(DeadLock.java:30)
        - waiting to lock <0x9f4c29b0> (a java.lang.Object)
        - locked <0x9f4c29b8> (a java.lang.Object)
   Locked ownable synchronizers:
        - None
"t1" prio=10 tid=0x09d22c00 nid=0x1f16 waiting for monitor entry [0x71dfe000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at DeadLock$1.run(DeadLock.java:15)
        - waiting to lock <0x9f4c29b8> (a java.lang.Object)
        - locked <0x9f4c29b0> (a java.lang.Object)
   Locked ownable synchronizers:
        - None

Found one Java-level deadlock:
=============================
"t2":
  waiting to lock monitor 0x09c32f18 (object 0x9f4c29b0, a java.lang.Object),
  which is held by "t1"
"t1":
  waiting to lock monitor 0x09c32360 (object 0x9f4c29b8, a java.lang.Object),
  which is held by "t2"

Java stack information for the threads listed above:
===================================================
"t2":
        at DeadLock$2.run(DeadLock.java:30)
        - waiting to lock <0x9f4c29b0> (a java.lang.Object)
        - locked <0x9f4c29b8> (a java.lang.Object)
"t1":
        at DeadLock$1.run(DeadLock.java:15)
        - waiting to lock <0x9f4c29b8> (a java.lang.Object)
        - locked <0x9f4c29b0> (a java.lang.Object)


Found 1 deadlock.

查找占用cpu最多的线程信息

方案：
# dump thread
# 找到导致cpu高的线程:top -H -p pid，对应的线程id是十进制的。(top命令详解见：http://www.w3pop.com/learn/view/p/1/o/1/doc/linux_cmd_top/)
# 十进制转十六进制
# 从dump中找到对应的线程


详细参见：http://www.longtask.com/blog/?tag=jstack