Storm工程在初始化Spring环境时的问题

问题描述

某个工程在启动时报如下错误

Worker is dead on 10.237.65.30:6804,20181213154018, at 2018-12-13 15:40
org.springframework.beans.factory.BeanDefinitionStoreException: IOException parsing XML document from class path resource [applicationContext-act.xml]; nested exception is java.io.IOException: Stream closed , at 2018-12-13 15:40
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:410)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:613)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:514)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:93)
at com.xxx.SpringContext.init(SpringContext.java:14)(业务类)

或者是这种

org.springframework.beans.factory.BeanDefinitionStoreException: Unexpected exception parsing XML document from class path resource [applicationContext-act.xml]; nested exception is java.lang.IllegalStateException: zip file closed , at 2018-12-13 16:30
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:414)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336)
at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:181)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:217)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:188)
at org.springframework.beans.factory.support.AbstractBeanDefinitionReader.loadBeanDefinitions(AbstractBeanDefinitionReader.java:252)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:127)
at org.springframework.context.support.AbstractXmlApplicationContext.loadBeanDefinitions(AbstractXmlApplicationContext.java:93)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:129)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:613)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:514)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.(ClassPathXmlApplicationContext.java:93)

问题以一定概率出现（最头痛的情况），但是能够正常启动。报的文件也不固定。因为xml里又import了多个xml。
其实也许这是一个无伤大雅的问题，但是我们不能这样的无视。谁知道哪天故障了系统崩溃这就成为你背锅的理由呢。作为程序员，我们确实不能交付一个用户有感知的BUG的系统并对这个BUG一无所知。
业务所提交的代码是这个样子的。

public class SpringContext {

    public static ApplicationContext context;

    public static synchronized  void init() {
        if (context == null) {
            context = new ClassPathXmlApplicationContext(new String[]{
                    "applicationContext-act.xml"
            });
        }
    }

在每一个bolt的启动前初始化context。利用JAVA的static模式保证单例模式。并加了锁。看样子真是一点问题都没有。

问题排查

上述代码严格来说并非没有问题。因为指令重排的原因，context可能在初始化完成前就已经发布。可能会有线程读到别的线程没有初始化完成的context。不过这无法解释为什么会出现流错误。
Storm会在nimbus端初始化bolt对象，然后序列化到各个节点。在执行bolt的pepare。同时把jar报作为一个文件传递给不同的节点。不同的worker上，其所谓的流，都是指自己的文件系统的该XML文件（Spring配置）。
我们也并非不熟悉StreamClosed这个异常。多个线程索引同一个inputstream时，当某一个thread在执行完之后，把这个inputstream关闭了；而此时正在从这个input stream流中读取信息的线程就会抛出 java.io.IOException: Stream closed 异常。

public
class InflaterInputStream extends FilterInputStream {
    private boolean closed = false;
    /**
     * Check to make sure that this stream has not been closed
     */
    private void ensureOpen() throws IOException {
        if (closed) {
            throw new IOException("Stream closed");
        }
    }
    public void close() throws IOException {
        if (!closed) {
            if (usesDefaultInflater)
                inf.end();
            in.close();
            closed = true;
        }
    }

很遗憾，我们没办法本地复现这个错误。我们也缺乏重写JDK加入日志的勇气。依然没有头绪。