[Reprint] Java performance testing tool - a record high troubleshoot Linux server CPU utilization practices by jstack

Java performance testing tool - a record high troubleshoot Linux server CPU utilization practices by jstack

https://www.jianshu.com/p/d4e31301ba2e

 

I. Description of the problem

Configuring a Linux server is 4-core 16G, the war package is deployed to tomcat, start tomcat, found that memory usage is not high, but the CPU has been up to 100%; browser, enter the relevant url can not access the project, and the tomcat process has been there is, what configuration program no problem ah, confused ...... view the performance status of the server by top command is as follows:

[root@aws-java-MAM ec2-user]# top
top - 18:49:21 up 45 min,  2 users,  load average: 0.97, 0.50, 0.19 Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie Cpu0 : 95.1%us, 0.0%sy, 0.0%ni, 4.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 5.9%us, 0.0%sy, 0.0%ni, 94.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 1.6%us, 0.5%sy, 0.0%ni, 97.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 1.1%us, 0.0%sy, 0.0%ni, 98.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16330912k total, 2374776k used, 13956136k free, 20172k buffers Swap: 0k total, 0k used, 0k free, 370244k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2141 root 20 0 7684m 1.7g 22m S 103.0 11.0 3:35.95 java 1423 root 20 0 449m 23m 2884 S 0.0 0.1 0:01.05 salt-minion 1959 root 20 0 98280 3860 2920 S 0.0 0.0 0:00.00 sshd 2278 root 20 0 98280 3836 2900 S 0.0 0.0 0:00.00 sshd 1387 postfix 20 0 81536 3436 2544 S 0.0 0.0 0:00.00 qmgr 1379 root 20 0 81284 3424 2520 S 0.0 0.0 0:00.01 master 1386 postfix 20 0 81364 3380 2508 S 0.0 0.0 0:00.00 pickup 1303 ntp 20 0 30720 2124 1508 S 0.0 0.0 0:00.00 ntpd 1986 root 20 0 105m 2000 1544 S 0.0 0.0 0:00.03 bash 1413 zabbix 20 0 42448 1972 1300 S 0.0 0.0 0:00.13 zabbix_agentd 

PID can be seen as a java process 2141 11% of the cost of memory, is not high, but the CPU reaches 103%, where there should be an infinite loop or thread is blocked to live, resulting in high CPU utilization (in various configurations personally feel should be is no problem), how to locate specific thread or program position it? Thought jstack tool.

Two, jstack Profile

jstack is java virtual machine comes with a stack trace tool. Java stack information for printing a given java process ID or core file or remote debugging services, use the command tool can be jstack thread stack information, according to the thread stack information, we can go to check for problems arise Java programs, such as detection deadlocks, and troubleshooting information output deadlock.

Third, the way the investigation

1, the first list of threads displayed by the following command

ps -mp pid -o THREAD,tid,time

[root@aws-java-MAM ec2-user]# ps -mp 2141 -o THREAD,tid,time |sort -rn USER %CPU PRI SCNT WCHAN USER SYSTEM TID TIME root 94.3 - - - - - - 00:04:04 root 85.9 19 - - - - 2185 00:03:21 root 3.7 19 - futex_ - - 2153 00:00:09 root 3.6 19 - futex_ - - 2152 00:00:09 root 2.8 19 - futex_ - - 2162 00:00:07 root 1.3 19 - futex_ - - 2154 00:00:03 root 0.9 19 - futex_ - - 2176 00:00:02 root 0.7 19 - futex_ - - 2147 00:00:02 root 0.7 19 - futex_ - - 2146 00:00:02 root 0.7 19 - futex_ - - 2145 00:00:02 root 0.7 19 - futex_ - - 2144 00:00:02 root 0.2 19 - futex_ - - 2148 00:00:00 root 0.2 19 - futex_ - - 2143 00:00:00 root 0.1 19 - inet_s - - 2209 00:00:00 root 0.0 19 - poll_s - - 2181 00:00:00 root 0.0 19 - poll_s - - 2179 00:00:00 root 0.0 19 - futex_ - - 2188 00:00:00 root 0.0 19 - futex_ - - 2183 00:00:00 root 0.0 19 - futex_ - - 2182 00:00:00 root 0.0 19 - futex_ - - 2180 00:00:00 root 0.0 19 - futex_ - - 2178 00:00:00 root 0.0 19 - futex_ - - 2175 00:00:00 

From here we can see the most CPU intensive thread ID 2185 is occupied CPU85.9%, TIME is the corresponding time duration of 3 minutes 21 seconds, sort -rn command is the reverse order of the mean.

2, thread ID need to be converted to hexadecimal format

printf "%x\n" tid

[root@aws-java-MAM ec2-user]# printf "%x\n" 2185 889 

3 stack information, print thread

jstack pid | grep time -A 100

[root@aws-java-MAM ec2-user]# jstack 2141 |grep 889 -A 100
"Common.timer" #39 daemon prio=5 os_prio=0 tid=0x00007ffbc4398800 nid=0x889 runnable [0x00007ffc49b19000]
   java.lang.Thread.State: RUNNABLE
    at com.skymusic.service.cache.Cache$1.run(Cache.java:59)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)

"Timer-1" #37 daemon prio=5 os_prio=0 tid=0x00007ffbc4373800 nid=0x887 in Object.wait() [0x00007ffc49c1a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.util.TimerThread.mainLoop(Timer.java:552)
    - locked <0x00000006c8201a40> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:505)

"pool-4-thread-2" #36 prio=5 os_prio=0 tid=0x00007ffbc41f9800 nid=0x886 waiting on condition [0x00007ffc49f1b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000006cb893460> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"AMQP Connection 172.31.44.144:5672" #35 prio=5 os_prio=0 tid=0x00007ffbc41f8000 nid=0x885 runnable [0x00007ffc4a01c000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    - locked <0x00000006cb893a28> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)
    at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:91)
    at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:164)
    - locked <0x00000006cb893a08> (a java.io.DataInputStream)
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:596)
    at java.lang.Thread.run(Thread.java:745)

"pool-2-thread-2" #34 prio=5 os_prio=0 tid=0x00007ffbc41f6800 nid=0x884 waiting on condition [0x00007ffc4a11d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000006cbca2378> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

"AMQP Connection 172.31.41.147:5672" #33 prio=5 os_prio=0 tid=0x00007ffbc41ef000 nid=0x883 runnable [0x00007ffc4a21e000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    - locked <0x00000006cbcb8b60> (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)
    at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:91)
    at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:164)
    - locked <0x00000006cbcb8b40> (a java.io.DataInputStream)
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:596)
    at java.lang.Thread.run(Thread.java:745)

"Timer-0" #32 daemon prio=5 os_prio=0 tid=0x00007ffbf8f94800 nid=0x882 in Object.wait() [0x00007ffc4aad1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000006cbcc2d48> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:552)
    - locked <0x00000006cbcc2d48> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:505)

"Thread-7" #30 daemon prio=5 os_prio=0 tid=0x00007ffbf8f90800 nid=0x880 waiting on condition [0x00007ffc4b0f3000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000006c8160e18> (a java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
    at com.dj.boot.Boot.init(Boot.java:108)
    at com.dj.boot.Boot.dev(Boot.java:35)
    at com.dj.filter.MD5Filter$1.run(MD5Filter.java:65)
    at java.lang.Thread.run(Thread.java:745)

"startQuertz_QuartzSchedulerThread" #29 prio=5 os_prio=0 tid=0x00007ffbf8f77800 nid=0x87f in Object.wait() [0x00007ffc4b1f4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:410)
    - locked <0x00000006c7dbf6b0> (a java.lang.Object)

"startQuertz_Worker-10" #28 prio=5 os_prio=0 tid=0x00007ffbf8f6c000 nid=0x87e in Object.wait() [0x00007ffc4b2f5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
[root@aws-java-MAM ec2-user]# 

-A behind parameter is the number of rows, the row number display the stack information, the display 100 is 100 lines. In which the section below caught my attention:

"Thread-7" #30 daemon prio=5 os_prio=0 tid=0x00007ffbf8f90800 nid=0x880 waiting on condition [0x00007ffc4b0f3000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00000006c8160e18> (a java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
    at com.dj.boot.Boot.init(Boot.java:108)
    at com.dj.boot.Boot.dev(Boot.java:35)
    at com.dj.filter.MD5Filter$1.run(MD5Filter.java:65)
    at java.lang.Thread.run(Thread.java:745)

Here's a run method MD5Filter class, from a thread to initialize a Boot configuration (new business other colleagues, do not understand), then WATTING been here, so find the relevant colleagues looked and found new business logic problems, after colleagues changed the CPU on the down.

4, and then summarize the methods and techniques under investigation CPU failure

1) top command

View real-time CPU usage, you can also view the most recent CPU usage.

2) PS command

Process status monitoring command; you can view the current processes and CPU usage threads in the process, which belongs to the current state of the sample data.

3) jstack

View a process of the current thread stack operation. According to the current operating state of the output of a process you can locate the command of all threads running code, and whether the deadlock.

4）pstack

View a process of the current thread stack operation.