MySQL Statement Cancellation Timer问题

线上日志最近报这样的错误:

"MySQL Statement Cancellation Timer" daemon prio=10 tid=0x00007f0cf403c800 nid=0x6e42 
in Object.wait() [0x00007f0ce4f4e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000c6b0f0d8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

而且每次查看日志也都一直还在，可能还会一直增加，并且全部是守护线程

**WAITING (on object monitor)**应该是等待获取对象，就像是排除一样，等待队列中是否有任务，有任务就获取任务处理

先看看TimerThread是什么吧，

class TimerThread extends Thread {
    boolean newTasksMayBeScheduled = true;


    private TaskQueue queue;

    TimerThread(TaskQueue queue) {
        this.queue = queue;
    }

    public void run() {
        try {
            mainLoop();
        } finally {
            // Someone killed this Thread, behave as if Timer cancelled
            synchronized(queue) {
                newTasksMayBeScheduled = false;
                queue.clear();  // Eliminate obsolete references
            }
        }
    }


    private void mainLoop() {
        while (true) {
            try {
                TimerTask task;
                boolean taskFired;
                synchronized(queue) {
                    // Wait for queue to become non-empty
                    while (queue.isEmpty() && newTasksMayBeScheduled)
                        queue.wait();
                    if (queue.isEmpty())
                        break; // Queue is empty and will forever remain; die

                    // Queue nonempty; look at first evt and do the right thing
                    long currentTime, executionTime;
                    task = queue.getMin();
                    synchronized(task.lock) {
                      //如果队列中的任务状态确实是已经取消的，则从队列中移除
                        if (task.state == TimerTask.CANCELLED) {
                            queue.removeMin();
                            continue;  // No action required, poll queue again
                        }
                        currentTime = System.currentTimeMillis();
                        executionTime = task.nextExecutionTime;
                        if (taskFired = (executionTime<=currentTime)) {
                            if (task.period == 0) { // Non-repeating, remove
                                queue.removeMin();
                                task.state = TimerTask.EXECUTED;
                            } else { // Repeating task, reschedule
                                queue.rescheduleMin(
                                  task.period<0 ? currentTime   - task.period
                                                : executionTime + task.period);
                            }
                        }
                    }
                    if (!taskFired) // Task hasn't yet fired; wait
                        queue.wait(executionTime - currentTime);
                }
                if (taskFired)  // Task fired; run it, holding no locks
                    task.run();
            } catch(InterruptedException e) {
            }
        }
    }
}

TimerThread是jdk中timer中的一个内部类，等待timer队列中的任务，作用是执行，调度task,移除队列中已经取消的和无效的(spent non-repeating)task

再看下这个线程是在哪里被初始化的吧,在mysql-connector-java-xxx.jar中的类ConnectionImpl中的getCancelTimer方法，大概在362行左右:

public Timer getCancelTimer() {
   synchronized (getConnectionMutex()) {
       if (this.cancelTimer == null) {
           boolean createdNamedTimer = false;

           // Use reflection magic to try this on JDK's 1.5 and newer, fallback to non-named timer on older VMs.
           try {
               Constructor<Timer> ctr = Timer.class.getConstructor(new Class[] { String.class, Boolean.TYPE });

               this.cancelTimer = ctr.newInstance(new Object[] { "MySQL Statement Cancellation Timer", Boolean.TRUE });
               createdNamedTimer = true;
           } catch (Throwable t) {
               createdNamedTimer = false;
           }

           if (!createdNamedTimer) {
               this.cancelTimer = new Timer(true);
           }
       }

       return this.cancelTimer;
   }
}

在这里插入图片描述
可以看到哪里调用getCancelTimer方法，大部分都是mysql-connector-java-xxx.jar中的执行sql查询(query)的statement中调用,抽取StatementImpl.java中的executeQuery的方法:

public java.sql.ResultSet executeQuery(String sql) throws SQLException {
   synchronized (checkClosed().getConnectionMutex()) {
       //ommit....
       String oldCatalog = null;

       try {
           if (locallyScopedConn.getEnableQueryTimeouts() && this.timeoutInMillis != 0 && locallyScopedConn.versionMeetsMinimum(5, 0, 0)) {
               timeoutTask = new CancelTask(this);
               locallyScopedConn.getCancelTimer().schedule(timeoutTask, this.timeoutInMillis);
           }

           if (!locallyScopedConn.getCatalog().equals(this.currentCatalog)) {
               oldCatalog = locallyScopedConn.getCatalog();
               locallyScopedConn.setCatalog(this.currentCatalog);
           }

           //
           // Check if we have cached metadata for this query...
           //

          //omitted...

           if (timeoutTask != null) {
               if (timeoutTask.caughtWhileCancelling != null) {
                   throw timeoutTask.caughtWhileCancelling;
               }

               timeoutTask.cancel();

               locallyScopedConn.getCancelTimer().purge();

               timeoutTask = null;
           }

           synchronized (this.cancelTimeoutMutex) {
               if (this.wasCancelled) {
                   SQLException cause = null;

                   if (this.wasCancelledByTimeout) {
                       cause = new MySQLTimeoutException();
                   } else {
                       cause = new MySQLStatementCancelledException();
                   }

                   resetCancelledState();

                   throw cause;
               }
           }
       } finally {
           this.statementExecuting.set(false);

           if (timeoutTask != null) {
               timeoutTask.cancel();

               locallyScopedConn.getCancelTimer().purge();
           }

           if (oldCatalog != null) {
               locallyScopedConn.setCatalog(oldCatalog);
           }
       }

       //ommitted...
       return this.results;
   }
}

可以发现，如果开启了QueryTimeouts,并且timeoutInMillis！=0的时候，一旦发现sql查询超出timeoutInMillis的时间，就会创建timeoutTask

数据库连接池配置

再看项目中数据库连接池配置，使用的tomcat-jdbc连接池:

<bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource" destroy-method="close">
   <property name="poolProperties">
       <bean class="org.apache.tomcat.jdbc.pool.PoolProperties">
           <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
           <property name="url" value="jdbc:mysql://localhost:3306"/>
           <property name="username" value="root"/>
           <property name="password" value="root"/>
           <property name="jmxEnabled" value="false"/>
           <property name="testWhileIdle" value="false"/>
           <property name="initialSize" value="1"/>
           <property name="maxActive" value="60"/>
           <property name="maxIdle" value="50"/>
           <property name="minIdle" value="15"/>
           <property name="defaultAutoCommit" value="false"/>
           <property name="maxWait" value="6000"/>
           <property name="removeAbandoned" value="true"/>
           <property name="removeAbandonedTimeout" value="60"/>
           <property name="testOnBorrow" value="true"/>
           <property name="testOnReturn" value="false"/>
           <property name="validationQuery" value="SELECT 1"/>
           <property name="validationInterval" value="60000"/>
           <!-- <property name="validationQueryTimeout" value="5"/> -->
           <property name="timeBetweenEvictionRunsMillis" value="300000"/>
           <property name="minEvictableIdleTimeMillis" value="1800000"/>
           <property name="jdbcInterceptors"
                     value="org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer"/>
       </bean>
   </property>
</bean>

查看tomcat-jdbc连接池文档: http://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html

Attribute	Description
validationQueryTimeout	(int) The timeout in seconds before a connection validation queries fail. This works by calling java.sql.Statement.setQueryTimeout(seconds) on the statement that executes the validationQuery. The pool itself doesn’t timeout the query, it is still up to the JDBC driver to enforce query timeouts. A value less than or equal to zero will disable this feature. The default value is -1.

这个validationQueryTimeout参数就是sql设置查询超时时间的，因为xml中设置了5秒，直接注释了就行，测试之后发现好像就没有出现这个问题了，至少目前没有。。。

下一步

确定引起sql查询超时的sql，可能需要开启mysql的慢日志查询