The operation and maintenance called and said that a service process in production was hung up, so he restarted it first. Operation and maintenance monitoring shows that the heap memory usage of Jvm has always been high, greater than 90%. Checked that there is indeed a problem with the allocation of jvm memory, which is smaller than normal needs. Check the erro log of the process, "Out of Memory Error" is impressive, then adjust the heap memory allocation, restart the matter.
a few days later ....
Operations calls. . Your service process hangs up again...
After observing the operation and maintenance monitoring log, the heap memory usage is normal. Check the error log of the process. Still "Out of Memory Error". This means that it is not that the heap memory is insufficient, it may be the stack memory. Take a closer look,
pool -5374 . The thread pool has been opened to 5374. The following threads are ranked to 6. At least there are already 30000+ threads. This is sure that there is a problem with the use of the thread pool, and it is leaked and not released. Running out of stack memory.
I searched for Threadpool in the whole project. Sure enough, the new threadpool in the method does not shut down after running. The fundamental problem is solved.
1. out of memory is not just insufficient heap memory, although it is more common.
2. ThreadPool should be used with caution, improper use is a big killer. Remember to shutdown when used locally.
newFixedThreadPool
The threads in the pool will exist
until it is explicitly {@link ExecutorService#shutdown shutdown}.