Problem solving: record the process of solving the high CPU of Java application at a time

Problem scenario

There is a project online. During the operation, it CPUwill gradually increase over time, and will eventually take up all of it CPU, causing the application to fail to respond and cause a malfunction. This blog post is mainly aimed at solving problems in this situation and sorting out the resolution steps.

Problem environment

software	version
Centos	6.4
JDK	1.6

problem causes

1. The top command to view the process

We first use the topcommand to view the corresponding process, the effect is as shown in the figure below, we can know that the current program has occupied 10a CPUcore, which is a relatively high CPUoccupancy rate.

2. The top command to view the thread situation inside a specific process

Next, we use the following command to access the inside of the corresponding process to see which thread is occupying the CPUresource, as follows:

top -H -p 24544

The result is as shown in the figure below:

we can know from the figure above that 10the CPUoccupancy rate of one thread has been reached 96%. We randomly selected a thread, here we choose pidfor the 24937thread, and converted to 16进制, the following screenshot:

3. Use the jstack command to get the stack information of the process

After that, we use the following command to dump the stack information of the process as follows:

 jstack 24544 > 24544_error.log

4. Search in the stack information according to the hexadecimal pid in the second step to obtain the specific thread situation

According to the above and get 16进制coding, 24544_error.logsearching look, you can find a thread, as shown below:

5. View the specific code situation

According to the thread situation obtained in step 4, find the code location corresponding to the project. The red box in the figure below is where the blocked code is:

At first glance, it feels like HashMapan infinite loop problem. The blogger checked the XssFilter.invalidMapassignment and found that in XssFilterthe filtering method of the filter, the pair XssFilter.invalidMapwas initialized, as shown in the figure below

This will cause the link to be initialized every time it passes through the filter XssFilter.invalidMap. From the code, you can see that it XssFilter.invalidMapis an HashMapobject, which is a non-thread-safe class. In each call XssFilter.invalidMapto traverse, if other threads happen to be initialized, an infinite loop will occur. In the case of high concurrency, the incidence of such an endless loop situation will be very high. Therefore, when the project is running online, this situation usually occurs during the peak period, which leads to a CPUhigh spike , and then makes the project inaccessible.

solution

Now that it is HashMapan infinite loop problem, just HashMaplock the initialization. After consulting the original developer, this initialization is enough. I provide a solution here, using double check lock:

if (null == invalidMap) {
    
    
    synchronized (invalidMap) {
    
    
        if (null == invalidMap) {
    
    
            // 初始化invalidMap 
        }
    }
}

This only needs to be initialized once, and there will be no repeated initialization.

result

After updating the patch, the project is currently running normally. The problem is solved! ! !

to sum up

When writing code, you should think more about efficiency and safety. This blog post mainly describes the idea of ​​the whole solution process. Generally, according to this idea, many problems can be solved basically.

Ask for praise

If my article is helpful to everyone, you can click like or favorite at the bottom of the article;
if there is a good discussion, you can leave a message;
if you want to continue to view my future articles, you can click Follow
You can scan the following QR code to follow me 'S public account: Fengye Zhixuege, check out my latest share!