The blogger posted a blog in the previous blog post about the problem of flink writing to mysql or Oracle with high performance. Although the writing performance has been improved, but in the development process of other projects, they encountered connection connection The problem of failure.
The blogger’s usage scenario is like this:
The blogger’s project is a real-time push project. For each successful push, a piece of mysql data is inserted. Considering that the push to the user at night may disturb the user, so the user will not be pushed from 22 to 07, so in this During the gap period, the mysql connection is not used, so the connection connection failure will be reported. At this point, the high-performance write mysql solution mentioned by the blogger will not work, because that way cannot set the connection failure time and Detected, in the case of c3p0 is not ideal, this time the blogger used the DBCP connection pool.
Maven used
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-dbcp2</artifactId>
<version>2.5.0</version>
</dependency>
The tools I use are as follows
package com.chinaTelecom.mySqlUtiles;
import org.apache.commons.dbcp2.BasicDataSource;
import javax.sql.DataSource;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
public class DBCPUtils {
//建立连接的驱动驱动名称
public static final String DRIVER_CLASS_NAME = "***";
//数据库链接数据哭的url
public static final String URL = "***";
//链接的数据库账号
public static final String USERNAME = "***";
//链接的数据库密码
public static final String PASSWORD = "***";
//初始化时链接池的数量
private static final int INITIAL_SIZE = 10;
//的到链接实例
private static BasicDataSource dataSource = new BasicDataSource();
//初始化链接参数
static{
dataSource.setDriverClassName(DRIVER_CLASS_NAME);
dataSource.setUrl(URL);
dataSource.setUsername(USERNAME);
dataSource.setPassword(PASSWORD);
dataSource.setInitialSize(INITIAL_SIZE); //初始化连接:连接池启动时创建的初始化连接数量
dataSource.setMaxTotal(10);
dataSource.setMaxIdle(6); //最大空闲连接:连接池中容许保持空闲状态的最大连接数量,超过的空闲连接将被释放,如果设置为负数表示不限制
dataSource.setMinIdle(4); //最小空闲连接:连接池中容许保持空闲状态的最小连接数量,低于这个数量将创建新的连接,如果设置为0则不创建
dataSource.setMinEvictableIdleTimeMillis(1000 * 60 * 60 * 10); //连接在池中保持空闲而不被空闲连接回收器线程(如果有)回收的最小时间值，单位毫秒
dataSource.setTestOnBorrow(true); //是否在从池中取出连接前进行检验,如果检验失败,则从池中去除连接并尝试取出另一个.
dataSource.setTestOnCreate(true); //指明对象在创建后是否需要验证是否有效，如果对象验证失败，则触发对象创建的租借尝试将失败。
dataSource.setTestWhileIdle(true); //指明对象是否需要通过对象驱逐者进行校验（如果有的话），假如一个对象验证失败，则对象将被从池中释放。和timeBetweenEvictionRunsMillis配合使用。
dataSource.setTimeBetweenEvictionRunsMillis(100000); //在空闲连接回收器线程运行期间休眠的时间值，以毫秒为单位，即每100秒做一次idle连接的检测，检测语句为validationQuery。
dataSource.setValidationQuery("select 1"); //SQL查询，用来验证从连接池取出的连接，验证连接是否有效，查询必须是一个SQL SELECT并且必须返回至少一行记录。
dataSource.setNumTestsPerEvictionRun(10); //每次空闲连接回收器线程运行时检查的连接数量
}
//提供获得数据源
public static DataSource getDateSource(){
return dataSource;
}
//提供获得链接
public static Connection getConnection() throws SQLException {
return dataSource.getConnection();
}
//测试connection是否正常
public static void main(String[] args) throws SQLException {
Connection connection = DBCPUtils.getConnection();
PreparedStatement ps = connection.prepareStatement("select * from phoneNum");
ResultSet rs = ps.executeQuery();
while (rs.next()){
String string = rs.getString(2);
System.out.println(string);
}
}
}
Parameter interpretation :
testOnBorrow=false means that when using the connection in the connection pool, no detection is performed. This is also a recommended configuration. If set to true, it will affect the performance of the database.
testWhileIdle=true indicates whether the connection is tested by the idle connection collector. If the test fails, the connection will be removed from the pool and used in conjunction with timeBetweenEvictionRunsMillis.
validationQuery=select 1 SQL query, used to verify the connection taken from the connection pool, to verify whether the connection is valid, the query must be a SQL SELECT and must return at least one row of records.
timeBetweenEvictionRunsMillis=100000 The sleep time value during the running of the idle connection collector thread, in milliseconds, that is, an idle connection check is performed every 100 seconds, and the check statement is a validationQuery.
minEvictableIdleTimeMillis=600000 The connection is kept idle in the pool without being idle connection collector thread, that is, the storage time of idle connection is 600 seconds (10 minutes)
There is no problem. Idle thread detection is performed every 100 seconds. The detection time of the application is 600 seconds, which is also less than the 1800 seconds set by the database. Why is there an abnormal connection failure? And the reported connection failure time is getting longer and longer.
Analysis :
Look at the dbcp source code and see that there is also a parameter numTestsPerEvictionRun, the number of connections checked each time the idle connection collector thread runs. This parameter configures the number of idle threads to be detected every 100 seconds. We did not configure this parameter, but the default value.
protected int numTestsPerEvictionRun = 3;
The configuration in the source code is 3, which means that only 3 idle threads are detected each time by default.
According to the worst case, the largest idle thread is 800, and the judgment of 3 idle threads is executed every 100 seconds. It takes (800/3) * 100 seconds = 26666 seconds to determine the status of all idle threads, and the database timeout is 1800 seconds, which means that if the pool is full (or there are a lot of connections in the pool), until all idle threads have been tested, the application has a high probability of fetching failed connections. The earliest timeout connection should be after 30 minutes, which is 1800 seconds in the database configuration
Solution :
The recommended configuration for everyone on the Internet is to set numTestsPerEvictionRun=maxIdle, so that all invalid idles can be removed from the thread pool after a single test, avoiding the problem that the connection pool connection has failed after a peak.
In view of the fact that the invalid connection will be removed slowly, no configuration modification operation was done at that time, and then the numTestsPerEvictionRun parameter will be configured to the same value as maxIdle, which can avoid this problem.