A JAVA memory leak investigation

foreword

Some blog posts I wrote at work before were not synchronized to the blog. This article may have been written more than a year ago, hoping to give readers some inspiration.

First, let's introduce the background of the project. Our project is deployed to tomcat. There are several Java projects running under one tomcat. We find that when we redeploy (throw the war package directly into the webapps directory) the project often occurs for a long time. And the follow-up solution is generally to restart tomcat.
This is a troubleshooting attempt I made.

1. First open the tomcat manager interface.

The specific steps are:

1. /tomcat/conf/tomcat-users.xml add:

<role rolename="manager-gui"/>
<user username="tomcat" password="s3cret" roles="manager-gui"/>

2. CATALINA_HOME/webapps/manager/META-INF/context.xml, comment remote connection limit:

<Context antiResourceLocking="false" privileged="true" >
<!--
<Valve className="org.apache.catalina.valves.RemoteAddrValve" allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" />
-->
</Context

3. Access remote host: 8080 Click Manager App:

4. Redeploy a project and click the Find leaks button:

5. Check to see if there is a memory leak:

2. View server status

View memory status:

3. Configure JMX

no password

1. edit /tomcat/bin/setenv.sh

 #!/bin/sh
 JAVA_OPTS="$JAVA_OPTS  -Dcom.sun.management.jmxremote
 -Dcom.sun.management.jmxremote.ssl=false
 -Djava.rmi.server.hostname=120.25.205.78                                                  -Dcom.sun.management.jmxremote.authenticate=false
 -Dcom.sun.management.jmxremote.port=18889"
 
 #JAVA_OPTS="$JAVA_OPTS  -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management. j mxremote.password.file=../conf/jmxremote.password -Dcom.sun.management.jmxremote.access.file=../conf/jmxremote.access -Dcom.sun.manag em ent.jmxremote.port=18889"   

Note that first hostname must be IP, and secondly it cannot be 127.0.0.1. If it is a virtual machine or the like, you can use the intranet address, which is similar to 192.168.1.3 or the like. Fill in the external network address of the cloud server and the like.

2. Restart tomcat.

with password

1. edit /tomcat/bin/setenv.sh

 #!/bin/sh
 JAVA_OPTS="$JAVA_OPTS  -Dcom.sun.management.jmxremote
 -Dcom.sun.management.jmxremote.ssl=false
 -Djava.rmi.server.hostname=xxx                                                 -Dcom.sun.management.jmxremote.authenticate=true
 -Dcom.sun.management. j mxremote.password.file=../conf/jmxremote.password 
 -Dcom.sun.management.jmxremote.access.file=../conf/jmxremote.access
 -Dcom.sun.management.jmxremote.port=18889"

2. Create/edit /tomcat/conf/jmxremote.password account permissions

monitorRole readonly                                                                      controlRole readwrite

3. Create/edit /tomcat/conf/jmxremote.access account password

 monitorRole tomcat
 controlRole tomcat

4. Modify read and write permissions:

chmod 600 jmxremote.access 

chmod 600 jmxremote.password 

5. restart tomcat

Failure Analysis

You can use Telnet to see if the port is available. If you can't use it, look at the firewall and the like.

4. Configure jstat

1. Confirm whether the environment variable JAVA_HOME exists. It has something to do with the jdk installation method.

export | grep JAVA_HOME

2. There is an output to represent this variable, so the next step is to create a jstatd.all.policy file.

mkdir -p /etc/java/
cd /etc/java/
(sudo) vim jstatd.all.policy

3. Edit the file jstatd.all.policy, you can see that the reason to confirm whether there is JAVA_HOME is for the reference of the file content.

grant codebase "file:${java.home}/../lib/tools.jar" { 
   permission java.security.AllPermission; 
};

4. start up

jstatd -J-Djava.security.policy=jstatd.all.policy -J Djava.rmi.server.hostname=xxx &

5. Analyze Tomcat logs

question:

 A web application appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak.

solve:

• upgrade mysql-connector-java

• introduce

  @Component
  public class ShutdownListener implements ServletContextListener {
        
        
      @Override
      public void contextInitialized(ServletContextEvent sce) {
        
        
  
      }
  
      @Override
      public void contextDestroyed(ServletContextEvent sce) {
        
        
          AbandonedConnectionCleanupThread.checkedShutdown();
      }
  }
  

  Reference: https://bugs.mysql.com/bug.php?id=69526

Hot deployment phenomenon

Use visualvm to observe and deploy a project with memory leaks: