Problem solving: Record the resolution process of a Java program memory leak (proxool memory leak)

Problem scenario

One day the front-end response of the program running online was slow, but the back-end log was still outputting normally. In response to this situation, this blog is mainly to explain the process of problem solving.

Problem environment

software	version
Centos	6.4
JDK	1.6
proxool	0.9.0RC3

problem causes

1. Use the top command to view the process

2. Use the jstat command to view the current GC situation

 jstat -gcutil  13062 5000

As can be seen from the above figure, the project is now in a state of frequent GC, and the memory is basically occupied.

3. Use the jmap command to print the maximum 20 objects currently alive

 jmap -histo:live 13062 | head -20

As you can see from the above figure, the database-related classes currently occupy a relatively large space. Among them is proxool. At this time, it is guessed that the problem is caused by proxool.

4. Use jmap to save the current stack information

jmap -dump:format=b,file=heap-dump.bin 13062 

5. Use the MAT tool to analyze the head.dump

Use the MAT tool to open the stack information exported in step 4, and then analyze it. The analysis results are as follows:

From the figure, we can know that Finalizer occupies the largest space, reaching 2.9GB.

6. Use the jstack command to get the stack information of the process

After that, we use the following command to dump the stack information of the process, as follows:

 jstack 13062 > 13062 _error.log

And search for the word Finalizer, the results are as follows:

7. Locking issues

This is a well-known proxoolmemory leak problem. When the object is JVMreclaimed WrappedConnection, because the proxy class rewrites the finalizemethod, the WrappedConnectionmethod is thrown into the reference queue to wait for the finalizerthread to execute the finalizemethod. There finalizeis no additional implementation itself, but the proxy class will do one before executing the method. isClosedetermination, and jdbc oraclethe class is used to achieve a synchronizemodification of the isCloseresulting business logic from the pool out of the connection will be used when the finalizethread contention the lock, once the business logic in a busy state if finalizerthe frequency of a thread of execution is greatly reduced, at this time The reference in the queue will still exist, and the object will still live in the heap.

solution

Now that you know the reason, overwrite the org.logicalcobwebs.proxool.WrappedConnectionclass and add the following code:

then recompile and submit it to the project, and restart.

result

After upgrading the patch, restart the project. On the next day, I dumped the stack information and checked it. There was no more memory occupied by Finalizer. The problem is solved.

to sum up

This blog post is mainly to record the various commands used in the solution process. Familiar use can solve many problems.

Reference link

Proxool solve problems connected oracle memory overflow
proxool pressure measurement problems encountered tune of

Ask for praise

If my article is helpful to everyone, you can click like or favorite at the bottom of the article;
if there is a good discussion, you can leave a message;
if you want to continue to view my future articles, you can click Follow
You can scan the following QR code to follow me 'S public account: Fengye Zhixuege, check out my latest share!