1. Tuesday after the new measure needs to mention running into the night, received a warning message, the CPU load is too high a production environment, and then troubleshoot the problem first, operation and maintenance of expansion, there is a problem on the machine to restart the line off the assembly line, CPU usage is normal, normal service response.
2. start troubleshooting, the one reserved for the machine in question for troubleshooting,
The first step, see related logs, no obvious abnormalities. Then top command to view cpu resource usage, jps -lm java process to find the corresponding number after 9021, top -H -p9021 see threads occupancy, cpu utilization rate of 93%
Record high usage rate of the specific thread id: 9023,9024,9027,9029 in linux, the thread is lightweight processes
By then view jstack thread stack information, jstack 9021> jstack_9021.txt, then the decimal 9023,9024,9027,9029 converted into hexadecimal (0x233f, 0x2340, 0x2343, 0x2345)
Finally, the cat jstack_9021.txt | grep -C 20 0x233f order to find a specific thread information, found that the cpu is played GC thread, and then jmap first look at the overall memory usage, Xmx configuration pricey, there is not enough memory,
Then jstat -gcutil 9021 3000 20 GC recovery of view, every time interval 3000ms printing, print 20 times and found Eden District and Old areas are depleted, FullGc very slow and takes a long time, which can determine a memory leak occurs.
jmap -histo 9021 print memory usage, found mainly introduce significant new third-party RSA bag SignContent object because this object contains a Base64 encoded byte Base64 decode byte PDF file and picture. In the collection, the end of the request did not lead to the release.