For a project I was responsible for recently, the CPU usage has been floating above 90% after the project was started for a period of time. After the inspection, there was no infinite loop. I was very puzzled.
The server configuration is a single core of Alibaba Cloud's server E5-2630 with 2GB of memory. The operating system is Windows Server 2008 (x64) and the Java virtual machine is Tomcat v7.0.
First, the CPU usage of mysql.exe is too high. First, use the show processlist command to find the most frequent sql statements. Then the key problem is that there are no other indexes in this database except the primary key. The guess is because of the parameter setting of the data. So the following adjustments were made
1. Increase MySQL's tmp_table_size to 200M
2. Add an index to the where filter field of the most frequently used statement
After the increase is completed, the CPU usage of the MySQL process drops significantly, and the memory is stable at around 300MB without a significant increase. Database tuning is complete.
Then the java.exe process takes up a lot of CPU, and there is no infinite loop, so I downloaded Microsoft's process explorer tool to monitor java.exe, found its pid number xxxx, and then checked the details and found that two of them The cpu occupied by the thread is particularly high, and then write down the tid numbers of these two threads. Let's find a way to find the running logs of these two threads.
First, try to use jstack that comes with jdk to grab the log, but an error is reported, like this
C:\Users\Administrator>jstack -l 8736 8736: Insufficient memory or insufficient privileges to attach The -F option can be used when the target process is not responding
报 Insufficient memory or insufficient privileges to attach
The -F option can be used when the target process is not responding
In fact, adding the -F parameter is useless. The root cause is a permission problem. The reason for this error is that my tomcat process is automatically started by calling the system's start service, and my jstack is started by itself on the command line. Although it seems to be under an administrator user, in fact, the tomcat process is not no.
So I stopped the tomcat service, and then manually clicked startup.bat in the bin directory of tomcat to start tomcat, and then called it again
jstack -l 3254 >c:\stack.log
command to output the process log of java.exe to a file, you will find that the jstack command is easy to use.
Then use a tool such as notpad++ to open the log file, convert the pid recorded above into hexadecimal, and use the converted thread number to search in the log file. I was lucky and directly located the specific method. It is found that this is a general method, which is a method of querying the corresponding entity class in the database according to the attribute.
The method name is findByProperties(xxxx, start, limit);
At first, I didn't find the exception of this method. Later, I found that my colleague accidentally assigned the value of start to 0, so that every time the user clicks the next page, the value from 0 to the destination page is actually loaded. Eventually, the cpu load exploded.
After changing start to a real variable, the problem is solved.