top -c
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6965 root 20 0 57.3g 27g 1.0g S 303.5 58.4 3243:41 /usr/local /...
23853 root 20 0 1614m 1.1g 15m S 4.6 2.4 4363:17 java...
22384 guest 15 0 12740 1164 820 R 0.3 0.0 0:00.08 top -c
2 root RT -5 0 0 0 S 0.0 0.0 0 :21.97 [migration/0]
PID:6965
(2) Find the thread with the highest CPU usage
top -Hp 6965 -d 1 -n 1
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16392 root 16 0 57.4g 27g 1.0g R 55.3 58.4 0:30.98 java
8757 root 25 0 57.4g 27g 1.0g R 53.6 58.4 103:54.27 java
top command parameter description:
-p PID Monitor only the ID of the specified process, PID is a numerical value;
-c Display the command line, not just the command name
-h When the system consists of multiple CPUs, the status information of individual CPUs is hidden, and only the average status value is displayed
-d N Displays the interval between two refresh times, such as -d 5, which means that the interval between two refreshes is 5 seconds;
(3) Print the stack information of JAVA process 6965 which occupies the highest CPU.
jstack 6965 > /root/mss/dump.txt
( 4) Convert the thread 16392 that accounts for the highest CPU into hexadecimal and go to the document to find the corresponding thread 4008.
Use the command
printf "%x\n" 16392
to find the corresponding hexadecimal thread id: 4008
"Reference Handler" daemon prio=10 tid =0x27ed0400 nid=0x1f1c in Object.wait() [0
x281ff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait( Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x0e6a5188> (a java.lang.ref.Reference$Lock)
参考文章：
http://lxiaodao.iteye.com/blog/1413774
来自 <http://blog.csdn.net/hfhwfw/article/details/37930013>
----------------------------------------------------------------------
top -Hp 22751 -d 1 -n 1
top - 10:03:35 up 153 days, 13:55, 1 user, load average: 3.05, 2.95, 2.86
Tasks: 336 total, 3 running, 333 sleeping, 0 stopped, 0 zombie
Cpu(s): 12.5%us, 5.3%sy, 0.1%ni, 81.9%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 132044872k total, 127819736k used, 4225136k free, 142604k buffers
Swap: 0k total, 0k used, 0k free, 109023720k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23274 work 20 0 29.6g 2.4g 9904 R 92.9 1.9 6547:42 java
23203 work 20 0 29.6g 2.4g 9904 R 91.0 1.9 6552:36 java
23130 work 20 0 29.6g 2.4g 9904 R 89.0 1.9 6541:45 java
22978 work 20 0 29.6g 2.4g 9904 S 2.0 1.9 7: 02.52 java
22982 work 20 0 29.6g 2.4g 9904 S 2.0 1.9 6: 48.97 java
23275 work 20 0 29.6g 2.4g 9904 S 2.0 1.9 8: 35.93 java
22751 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 00.00 java
22752 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 01.87 java
22753 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 03.21 java
22754 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 03.23 java
22755 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 03.24 java
22756 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 03.23 java
22757 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 03.24 java
22758 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 03.23 java
22759 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 03.21 java
22760 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 03.23 java
22761 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 00.00 java
22762 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 00.00 java
22763 work 20 0 29.6g 2.4g 9904 S 0.0 1.9 0: 00.00 java
printf "% x \ n" 23274
5aea
printf "% x \ n"
23203 5aa3
printf "% x \ n" 23130
5a5a
jstack -l 22751 > jstack22751.txt
"pool-3-thread-3" #315 prio=5 os_prio=0 tid=0x00007f777368f000 nid=0x5aea runnable [0x00007f7451e96000] java.lang.Thread.State: RUNNABLE at com.baidu.rigel.nbcps.handler.ElasticSearchTask.run(ElasticSearchTask.java:75) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x000000068013e060> (a java.util.concurrent.ThreadPoolExecutor$Worker) "pool-3-thread-2" #245 prio=5 os_prio=0 tid=0x00007f7772bba800 nid=0x5aa3 runnable [0x00007f7453ffe000] java.lang.Thread.State: RUNNABLE at com.baidu.rigel.nbcps.handler.ElasticSearchTask.run(ElasticSearchTask.java:75) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x0000000680762fd0> (a java.util.concurrent.ThreadPoolExecutor$Worker) "pool-3-thread-1" #175 prio=5 os_prio=0 tid=0x00007f77720e0800 nid=0x5a5a runnable [0x00007f76f9422000] java.lang.Thread.State: RUNNABLE at com.baidu.rigel.nbcps.handler.ElasticSearchTask.run(ElasticSearchTask.java:75) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Locked ownable synchronizers: - <0x0000000680209828> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Found in the code:
public void run() {
this.timestamp = System.currentTimeMillis();
while (this.active) {
long timeNow = System.currentTimeMillis();
if (null != this.queue && this.queue.size() > 0) {
try {
// return null if timeout is 50ms
T obj = queue.poll(TIMEOUT_MILLISECONDS, TimeUnit.MILLISECONDS);
if (null != obj) {
this.putList.add(obj);
}
} catch (Exception e) {
this.log.info(this.indexName + " Es batch thread sleep interrupted");
// Throwing an interrupt exception first exits the loop and stops the batch operation
if (!this.active) {
break;
}
}
}
this.queue.size() > 0 This condition will cause the cpu to be too high to remove
change to
if (null != this.queue)