We are using Java 8 Spring Boot 2 for our micro-service app. Running a load test locally noticed the heap consuming memory but never releasing back. I am running the app with G1 garbage collector and also did a manual GC from JVisualVM, but still the allocated memory never gets released.
I took the heap dump and analysed it, and I can clearly see the large byte array created by System Class Loader listed as leak suspect. I see the byte array instance holds my HTTP request to endpoint "/test". But the load test already completed and threads went back down to where they were before running load test.
Not sure why byte array loaded by system class loader contains all these elements and taking all this heap for no reason.
/test endpoint is the only method in a @RestController class
@RequestMapping(value = "/test", method = RequestMethod.GET)
@CrossOrigin(origins = "*")
public void test() {
logger.info("Testing1...");
}
Below are the Spring Boot application.properties related to server:
server.port=8090
server.tomcat.max-threads=200
server.tomcat.accept-count=100
server.tomcat.min-spare-threads=20
server.error.whitelabel.enabled=false
server.max-http-header-size=2097152
Tomcat caches a number of objects to make it go faster. With the setting server.max-http-header-size=2097152 you made one of those cached objects claim 2 MB of memory and keep it. In this case it is the Http11OutputBuffer and you can see here that it claims (in your case) the 2 MB of memory. The Http11OutputBuffer is used by the Http11Processor which you can see here.
The documentation has the following to say about processorCache:
The protocol handler caches Processor objects to speed up performance. This setting dictates how many of these objects get cached. -1 means unlimited, default is 200. If not using Servlet 3.0 asynchronous processing, a good default is to use the same as the maxThreads setting. If using Servlet 3.0 asynchronous processing, a good default is to use the larger of maxThreads and the maximum number of expected concurrent requests (synchronous and asynchronous).
So my suggestion is to set the server.max-http-header-size to something more reasonable, e.g. 8KB (the default) and slowly double that when testing shows you really need that (related: Tomcat throws "400 Bad request" when total header size is larger than server.max-http-header-size).