Online CPU 100% troubleshooting ideas

Suppose that several Java site services and several Java microservices are deployed on the server and suddenly receive an abnormal CPU operation and maintenance alarm. How to locate which service process is causing CPU overload.

Which thread overloads the CPU and which code overloads the CPU?

The brief steps are as follows:

(1) Find the process that consumes the most CPU;

(2) Find the thread that consumes the most CPU;

(3) Check the stack, locate what the thread is doing, locate the corresponding code;

Step 1: Find the most CPU consuming process

Tool : topMethod :

  • Execution top -c, display a list of process running information
  • Type P (uppercase p), the processes are sorted by CPU usage

Illustration: image
As shown above, the PID of the most CPU-intensive process is 10765.

Step 2: Find the thread that consumes the most CPU tool : top method :

  • top -Hp 10765, Display a list of thread running information of a process
  • Type P (uppercase p), threads are sorted by CPU usage

Icon:image

As shown above, in the process 10765, the PID of the thread that consumes the most CPU is 10804.
Step 3: Check the stack, locate the thread, and locate the corresponding code.
First, convert the PID of the thread to hexadecimal. Tool : printf method : printf "% x \ n" 10804 icon:
image
As shown above, the hexadecimal corresponding to 10804 is 0x2a34, of course, you can use a calculator in this step. The reason why it needs to be converted to hexadecimal is because the thread id is expressed in hexadecimal in the stack. Next, look at the stack and find out what the thread is doing.
Tool : jstack
Method :jstack 10765 | grep '0x2a34' -C5 --color

  • Print process stack
  • Filter the thread stack by thread id

Illustration:
image
As shown above, the thread name "AsyncLogger-1" corresponding to the thread with high CPU consumption is found, and the stack where the thread is executing code is seen.
Finally, according to the information in the stack, find the corresponding code and get it!

Published 41 original articles · Liked 14 · Visitors 10,000+

Guess you like

Origin blog.csdn.net/Yunwei_Zheng/article/details/104018739