After a few days of analysis, I hope to summarize what I have learned.
System version: Windows Server 2008 R2 Standard
System type: 64bit
Memory: 32GB
Program: Solr is deployed on the system, and then 5 threads are written to continuously query solr.
Problem phenomenon: The physical memory in the task manager has been increasing, and finally reached 99%. But the memory occupied by the process adds up to less than 10G.
analyze:
Step 1: Suspect java program memory overflow.
Tools: jvisualvm and eclipse mat
jvisualvm detects whether there is a memory overflow. If there is a memory overflow, you can use jmap to export the dump file, and then use mat to analyze it. mat can analyze the memory occupied by each class. There are a lot of mat use information on the Internet, you can check it yourself.
I used jvisualvm that comes with jdk, and checked the memory with jvisualvm.exe in the jdk bin directory. As shown below:
Through the above figure, it is found that the heap will be collected, so there is no memory overflow. In order to be familiar with the jmap and mat tools, I exported the dump file myself and analyzed it with mat.
Step 2: Further Analysis
If the program has no memory overflow problem, what is the memory occupied?
Analysis tools: RamMap and VMMap
RamMap can analyze the memory usage as a whole. VMMap can be accurate to a certain process ID. They can all check what files have been mapped from disk to memory.
I used RAMMAP analysis and found that a large amount of memory was occupied by Mapped File. Clicking Empty--Empty Working Sets will release the memory. At this time, the memory usage of the task manager dropped to 28%, but the memory was exhausted again after a while (there is a principle here , there is information on the Internet, this method is not advisable, and it affects the performance of the system).
Memory release:
Through rammap and vmmap analysis, it is found that the system loads the solr index file into the memory, and my index file has 90G. I checked a lot of information about this mapped file. It's probably like this: when a large number of IO operations are performed under the windows server, in order to improve performance, the system will map the files on the disk to the memory by default, but there is no memory limit. If the disk file is too large, it will cause memory exhaustion , which is also a windows server 2008 bug.
What if you limit this memory cap? There are two methods on the Internet: SetSystemFileCacheSize, and installing the Microsoft Windows Dynamic Cache Service patch provided by Microsoft. This is available on the Internet, and it is quite simple. You can check it.
Through the above two methods, still did not solve my problem. So, I plan to try it under linux, and I will add to the blog later.