A server deploys several tomcat instances, that is, several vertically segmented Java site services and several Java microservices, and suddenly receives an abnormal CPU alarm for operation and maintenance.
Q: How to locate which service process is causing the CPU overload, which thread is causing the CPU overload, and which code is causing the CPU overload?
Step 1. Find the process that consumes the most CPU
Tools: top
method:
-
Execute top -c to display a list of process running information
-
Type P (uppercase p), processes are sorted by CPU usage
Icon:
As shown above, the PID of the process that consumes the most CPU is 10765
Step 2: Find the thread that consumes the most CPU
Tools: top
method:
-
top -Hp 10765 , displays a list of thread running information of a process
-
Type P (uppercase p), threads are sorted by CPU usage
Icon:
As shown in the figure above, in process 10765, the PID of the thread that consumes the most CPU is 10804
Step 3: Convert the thread PID to hexadecimal
tool: printf
Method: printf "%x" 10804
Icon:
As shown in the figure above, the hexadecimal corresponding to 10804 is 0x2a34. Of course, a calculator can be used in this step.
The reason why it needs to be converted to hexadecimal is because the thread id is represented in hexadecimal in the stack.
Step 4: Check the stack to find out what the thread is doing
Tools: pstack/jstack/grep
Method: jstack 10765 | grep '0x2a34' -C5 --color
-
print process stack
-
By thread id, filter to get thread stack
Icon:
As shown in the figure above, the thread name "AsyncLogger-1" corresponding to the thread with high CPU consumption is found, and the stack of the code being executed by the thread is seen.