The key point is to check which thread of the process occupies too much CPU, and then use gdb to attach to the process, debug the thread, and see if there is an infinite loop or deadlock. The steps are as follows:
First use ps + grep to find out the damn process pid, such as 1706
Then top -H -p 1706, (top and then shift+H can see a certain thread, there is a hint in the upper left corner: thread on means the thread can be viewed) All threads of the process are listed, and see which thread pid takes up the most , Write down the corresponding thread number, such as: 1723
1.gdb attach to the process number (1706)
2. (Still in gdb) The results of info threads are roughly as follows:
(gdb) info threads 8 Thread 0x7f9fa9366700 (LWP 1716) 0x0000003cec00b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 7 Thread 0x7f9fa8965700 (LWP 1720) 0x0000003cec00b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 6 Thread 0x7f9fa7f64700 (LWP 1721) 0x0000003cec00f4b5 in sigwait () from /lib64/libpthread.so.0 5 Thread 0x7f9fa7563700 (LWP 1722) 0x0000003cec00b98e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x7f9fa6b62700 (LWP 1723) 0x0000003cec00b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 3 Thread 0x7f9fa6161700 (LWP 1724) 0x0000003cebce9163 in epoll_wait () from /lib64/libc.so.6 2 Thread 0x7f9fa1159700 (LWP 1887) 0x0000003cebce9163 in epoll_wait () from /lib64/libc.so.6 * 1 Thread 0x7f9fa95ad820 (LWP 1706) 0x0000003cec00b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0Find the thread corresponding to the thread number (LWP1723) is the thread number we just noted
3. (Still in gdb) thread thread number is switched to thread (4)-The serial number displayed in info threads here needs to use the line program number that gdb can recognize, that is, execute: thread 4 switches to the thread number we just noted: The corresponding thread of 1723 is as follows:
(gdb) thread 4 [Switching to thread 4 (Thread 0x7f9fa6b62700 (LWP 1723))]#0 0x0000003cec00b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.04. (Still in gdb) bt view thread call stack
(gdb) bt
#0 0x0000003cec00b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f9fa9f7144d in IceUtil::Cond::waitImpl (this=0x263f4c8, mutex=...) at ../../include/IceUtil/Cond.h:215 #2 0x00007f9fa9f9a4b1 in IceUtil::Monitor::wait (this=0x263f4c8) at ../../include/IceUtil/Monitor.h:152 #3 0x00007f9fa9fd7567 in IceInternal::EndpointHostResolver::run (this=0x263f480) at EndpointI.cpp:161 #4 0x00007f9fa9b1b975 in startHook (arg=0x263f480) at Thread.cpp:413 #5 0x0000003cec0079d1 in start_thread () from /lib64/libpthread.so.0 #6 0x0000003cebce8b6d in clone () from /lib64/libc.so.65. From the information output above, you can basically check the code break corresponding to the thread, whether there is an infinite loop, etc. If it is a deadlock, you need to check the current thread stack several times, or check the stack of all threads, there will always be something Some threads are inconsistent with other threads, and then correspond to the code to locate and solve