problem:
xml parsing throws an exception, the exception information is as follows:
javax.servlet.ServletException: javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:268)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at org.apache.struts2.dispatcher.ServletDispatcherResult.doExecute(ServletDispatcherResult.java:164)
at org.apache.struts2.dispatcher.StrutsResultSupport.execute(StrutsResultSupport.java:191)
at com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:372)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:276)
at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at app.base.interceptor.AuthenticationInterceptor.intercept(AuthenticationInterceptor.java:65)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:256)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:168)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:265)
at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:76)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:125)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:253)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:100)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:141)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:145)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:140)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:193)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)
at org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54)
at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:567)
at org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:81)
at org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:652)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:612)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:503)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.xml.parsers.FactoryConfigurationError: Provider org.apache.xerces.jaxp.DocumentBuilderFactoryImpl not found
at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:127)
at org.apache.jasper.xmlparser.ParserUtils.parseXMLDocument(ParserUtils.java:128)
at org.apache.jasper.xmlparser.ParserUtils.parseXMLDocument(ParserUtils.java:194)
at org.apache.jasper.compiler.TagLibraryInfoImpl.parseTLD(TagLibraryInfoImpl.java:236)
at org.apache.jasper.compiler.TagLibraryInfoImpl.<init>(TagLibraryInfoImpl.java:183)
at org.apache.jasper.compiler.Parser.parseTaglibDirective(Parser.java:386)
at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:450)
at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1400)
at org.apache.jasper.compiler.Parser.parse(Parser.java:130)
at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:255)
at org.apache.jasper.compiler.ParserController.parse(ParserController.java:137)
at org.apache.jasper.compiler.Parser.processIncludeDirective(Parser.java:296)
at org.apache.jasper.compiler.Parser.parseIncludeDirective(Parser.java:333)
at org.apache.jasper.compiler.Parser.parseDirective(Parser.java:442)
at org.apache.jasper.compiler.Parser.parseElements(Parser.java:1400)
at org.apache.jasper.compiler.Parser.parse(Parser.java:130)
at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:255)
at org.apache.jasper.compiler.ParserController.parse(ParserController.java:103)
at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:185)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:354)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:334)
at org.apache.jasper.compiler.Compiler.compile(Compiler.java:321)
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:592)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:328)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
... 78 more
Caused by: java.lang.ClassNotFoundException: org/apache/xerces/jaxp/DocumentBuilderFactoryImpl
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at javax.xml.parsers.FactoryFinder.getProviderClass(FactoryFinder.java:123)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:178)
at javax.xml.parsers.FactoryFinder.newInstance(FactoryFinder.java:147)
at javax.xml.parsers.FactoryFinder.find(FactoryFinder.java:219)
at javax.xml.parsers.DocumentBuilderFactory.newInstance(DocumentBuilderFactory.java:121)
... 103 moreThis problem will break out from time to time. There are 3 servers in the cluster, 2 for user access, 1 for timed tasks, and 2 servers for user access. This problem basically occurs once every other week. Once the problem occurs, the system The function will be paralyzed in a large area and cannot be used normally; basically every time this problem occurs on one of the 2 servers, the first few times are solved by restarting the server, but this problem frequently occurs and affects the system The use of, so I feel that the root cause is investigated;
Troubleshoot:
First of all, the method of investigation is naturally the same. First, Baidu reports an error message. There are some opinions about this error on the Internet, such as jar package conflicts. Some say delete xercesImpl.jar, use JDK's own xml parsing, etc.; others say The running code is processed by setSysterProperty(); however, after analysis at the time, no conflicting jar package mentioned on the Internet was found, and the error report was not the application code, but the code in other jar packages. Therefore, Baidu Did not find a solution.
At the same time, other colleagues helped solve this problem. 3 colleagues with more than 10 years of experience and 2 colleagues with 7 years of experience did not find the cause of the problem. So far, the problem has been put on hold, but the problem still occurs at least once a week, but the function continues. Troubleshoot the problem, check the source code, check the last part of the abnormal information after analysis, find the source code location, the source code is as follows (the project is running jdk1.7, and I am writing this blog, personal computer jdk1.8, so the abnormal information and the following code The number of lines will not correspond, but it does not affect what I express):
static <T> T find(Class<T> type, String fallbackClassName)
throws FactoryConfigurationError
{
final String factoryId = type.getName();
dPrint("find factoryId =" + factoryId);
// Use the system property first
try {
String systemProp = ss.getSystemProperty(factoryId);
if (systemProp != null) {
dPrint("found system property, value=" + systemProp);
return newInstance(type, systemProp, null, true);
}
}
catch (SecurityException se) {
if (debug) se.printStackTrace();
}
// try to read from $java.home/lib/jaxp.properties
try {
if (firstTime) {
synchronized (cacheProps) {
if (firstTime) {
String configFile = ss.getSystemProperty("java.home") + File.separator +
"lib" + File.separator + "jaxp.properties";
File f = new File(configFile);
firstTime = false;
if (ss.doesFileExist(f)) {
dPrint("Read properties file "+f);
cacheProps.load(ss.getFileInputStream(f));
}
}
}
}
final String factoryClassName = cacheProps.getProperty(factoryId);
if (factoryClassName != null) {
dPrint("found in $java.home/jaxp.properties, value=" + factoryClassName);
return newInstance(type, factoryClassName, null, true);
}
}
catch (Exception ex) {
if (debug) ex.printStackTrace();
}
// Try Jar Service Provider Mechanism
T provider = findServiceProvider(type);
if (provider != null) {
return provider;
}
if (fallbackClassName == null) {
throw new FactoryConfigurationError(
"Provider for " + factoryId + " cannot be found");
}
dPrint("loaded from fallback value: " + fallbackClassName);
return newInstance(type, fallbackClassName, null, true);
}After analyzing the above source code, it can be seen that the fully qualified name of the implementation class is first read from the systemProperty property. If it cannot be read (it has not been set), then try to read it from the jax.properties under lib in jdk. If you can’t read it, look for it from the jar package introduced under the web application (META-INF/services/javax.xml.parsers.DocumentBuilderFactory). If you still can’t find it in the end, it will use the com.sun that comes with jdk. org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl, but based on the above analysis, no problem was found. The corresponding jar package also exists in the project. Logically speaking, the third method should be used to obtain the fully qualified name of the implementation class. However, through the exception analysis, the result is that the first type is gone. After the first type, it must be where the setSystemProperty() is called to set the corresponding property value, but why does it report an error after the first type is gone? It is speculated that the class loader cannot load the corresponding class, but which class loader is called here, and why can't it be loaded? With this question, decompile rt.jar under JRE in jdk, and modify the methods in javax.xml.parsers.FactoryFinder and related classes, and increase the log output at each key point to facilitate the next time the problem occurs. Analyze the cause of the problem accurately; but at the same time, there is also a second consideration, which is to wrap the exception with try catch when the first type is used, so that even if the first type reports an error, the program will continue to execute downward, but it will not be found in this way The root cause, and also need to modify the jdk source code, so the first method is adopted to find the root cause of the problem.
Sure enough, an error occurred again not long after. After checking the log information this time, I found that my guess was indeed verified. The class loader could not load the corresponding class. Of course, not all of the first ones were not loaded. However, when the class is loaded as StandardClassLoader, it cannot be loaded, and WebAppClassLoader can be loaded. Since the added log outputs the class loader information, it is known that the former is the parent of the latter, and tomcat is involved here. The class search order of the container and the search order of the jdk class are now known. The class loading order of the jdk is known. At the time, the tomcat container did not know it. After a preliminary understanding of Baidu, the cause of the problem was generally understood.
But the next step is to find a solution. The direct reason is where the setSystemProperty() is called in the project to set the corresponding property value, but it is like looking for a needle in a haystack. How should I find it? I have searched the code in the project, and I have not found it. Exception, finally decided to find the answer from the jar package;
Therefore, unzip all the jar packages in the project, then compress them into a file, decompile this file, then unzip this file, and finally use the global search method of all files in the EditPlus folder, and finally find the problem, in a certain In the class of the function module, the code is executed in the static static code block, so as long as the function is clicked and the class is loaded, this code will be executed. So far, the root cause of the problem has been found, but I still don’t know how to solve it and why I didn't report an error before, why did I suddenly start to report an error? Fortunately, after finding the root cause, you can test and verify it. By operating this functional module, it is found that only the two servers accessed by the user will report an error. The server running the scheduled task will not report an error. Think of the two servers. It is the newly added server that you added to the cluster at the end of the year. It may be that the configuration is different, and finally locked to tomcat is different. After comparison, it is found that the tomcat version has become higher, which causes the problem of different class loaders and errors in some cases.
solve:
Finally, by comparing the server running the timed task with the 2 servers accessed by the user, it is found that the version of the tomcat is different, and the server version accessed by the two users is higher. Therefore, the tomcat on the timed task is migrated to these 2 servers as a whole On the server, so far the problem has been perfectly solved.
to sum up:
In the process of solving this problem, the biggest gain is the in-depth understanding of the class loading mechanism of the class loader, such as the formulation of the loading path corresponding to different class loaders, the search order of tomcat loading classes, and the priority of loading from the application for WebAppClassLoader, and then in Pass up, load down from the top layer.
This blog is mainly used to record personal project experience, so there is not too much accurate expression and explanation, I can understand it, if there are friends who have the same problem, you can comment if you don’t understand, I see Will answer.