Memory consumption problem when JAVA parses XML

There was a problem

In a recent project, there is a required function to read data from XML passed in from the outside, and then write it into the database.
After writing, I ran Tomcat on the local computer, read the data in the XML normally, and after sorting it out, inserted it into the database and saved it. However, when running online, there is often a problem that Tomcat crashes directly because of this function.

At first, I thought it was an online environment, and there were a large number of XML files imported from the outside. There was a problem when reading the list of XML files, because a recursive method was used to find all XML files in the multi-layer directory. Later, it was found that the directory hierarchy Not deep, recursion shouldn't be a problem.

After that, hundreds of XML files (about 7MB each) were created on the local computer for testing, and they could still be read and written to the database normally. During one round of testing, I accidentally looked at the memory usage of the local computer, and found that during the running process, the memory usage rose from 7.8G to 9.4G, and then remained stable until the end of the run, and then dropped back to 7.8G.

At this time, the problem is obvious. The process of reading XML files consumes too much memory. The total heap memory of online Tomcat is only 1GB, which must not be able to support it.

This aroused my curiosity. Here, an XML is only 7MB. How much memory can it consume when reading it?

So I opened JV (jvisualvm.exe in the jdk/bin directory). Put a breakpoint on the place where the program reads the XML file, and the monitoring results are as follows:


I found that the maximum space of Eden Space (new generation space) is 340M (in order to be the same as the online environment, the total heap memory of Tomcat on the local computer is also set to 1GB by me. , according to Tomcat’s default rules, the Eden space is 1/3 of the size), reading an XML file directly uses 150M, and the occupied Eden memory reaches 190M, and another one is directly full, so as long as the GC release is a little slower, it will run directly It doesn’t move anymore (my local computer GC is fast, so it can still run normally, that is, there are more GC times)

After that, I checked the information and found that the dom reading method I used is relatively memory-intensive, but this method can modify the XML content (but I don’t need this function for this requirement), and the other SAX consumes less memory.

So I changed the XML reading method, and finally let the online Tomcat (heap memory 1GB) can read a large number of XML files normally. The following are two ways to read 1000 XML monitoring results locally, and the number of GC times is doubled:

Let’s talk about it below. The modified reading method uses the JAXBContext (javax.xml.bind.JAXBContext) that comes with JAVA.
1. First, you need an object with the same structure as XML. If it is a multi-layer node, just Object references are required.

@XmlRootElement(name = "ROOT") //注解法指定根元素/对象与元素绑定，大小写需要一致；也可以换设置法
public static class Root{
    
    
	BookVo bookVo; //java对象中的属性名可以和节点名不一致
	public BookVo getBookEle() {
    
     //对象必做有get/set，且需要和XML节点名对应，<ROOT><bookEle>...</bookEle></ROOT>
		return viosurveilVo;
	}
	public void setBookEle(BookVo bookVo) {
    
    
		this.viosurveilVo = viosurveilVo;
	}
}
public static BookVo{
    
    ...}

//解析指定xml文件
private Root analysisXML(String xmlFilePath) {
    
    
	try {
    
    
		JAXBContext jaxbContext = JAXBContext.newInstance(Root.class);
		Unmarshaller unmarshaller = jaxbContext.createUnmarshaller();
		StreamSource source = new StreamSource(new File(xmlFilePath));
		JAXBElement<Root> jaxbElement = unmarshaller.unmarshal(source, Root.class); //指定法，指定根元素/对象与元素绑定
		Root root = jaxbElement.getValue(); //直接得到可操作JAVA对象
		//Root root = (Root) unmarshaller.unmarshal(new File(xmlFilePath)); //使用@注解法，只需要这句，不需要上面3句
		//ViosurveilVo viosurveilVo = root.getViosurveil();
		return root;
	}catch (Exception e) {
    
    
		return null;
	}
}