Mybatis database connection pool source code analysis
TSMYK Java technology programming
The first address of this article is the personal blog address:
https://my.oschina.net/mengyuankan/blog/2664784
This article will introduce from the following aspects
- related articles
- Preface
- Class Diagram
- Factory class implementation
- Database connection realization
- Implementation of connection pool
- Get a connection from the connection pool (flow chart)
- Put the connection into the connection pool (flow chart)
related articles
Source code analysis of Mybatis parsing configuration files Source code analysis of
Mybatis type conversion
Preface
When using Mybatis, the database connection generally uses third-party data source components, such as C3P0, DBCP and Druid, etc. In fact, Mybatis also has its own data source implementation, which can connect to the database and has the function of connection pool. Take a look at an implementation principle of the data source and connection pool implemented by Mybatis.
Class Diagram
The realization of Mybatis data source is mainly under the datasource package:
Our common data source components all implement the Javax.sql.DataSource interface, Mybatis also implements this interface and provides two implementation classes UnpooledDataSource and PooledDataSource. One uses connection pool and the other does not use connection pool. In addition, for these two classes, Mybatis also provides two factory classes to create objects, which is an application of the factory method pattern. First, let's look at a class diagram of them:
Regarding the above-mentioned classes, PooledDataSource and UnpooledDataSource are the main logic of the data source implementation. The code is more complicated. Let's look at it later. Now let's take a look at the two factory classes.
DataSourceFactory
Let's take a look at the DataSourceFactory class first. This class is the top-level interface of the JndiDataSourceFactory and UnpooledDataSourceFactory factory classes. Only two methods are defined, as shown below:
public interface DataSourceFactory {
// 设置 DataSource 的相关属性，一般在初始化完成后进行设置
void setProperties(Properties props);
// 获取数据源 DataSource 对象
DataSource getDataSource();
}UnpooledDataSourceFactory
UnpooledDataSourceFactory is mainly used to create the UnpooledDataSource object, it will initialize the UnpooledDataSource object in the construction method, and complete the configuration of the UnpooledDataSource object in the setProperties method
public class UnpooledDataSourceFactory implements DataSourceFactory {
// 数据库驱动前缀
private static final String DRIVER_PROPERTY_PREFIX = "driver.";
private static final int DRIVER_PROPERTY_PREFIX_LENGTH = DRIVER_PROPERTY_PREFIX.length();
// 对应的数据源，即 UnpooledDataSource
protected DataSource dataSource;
public UnpooledDataSourceFactory() {
this.dataSource = new UnpooledDataSource();
}
// 对数据源 UnpooledDataSource 进行配置
@Override
public void setProperties(Properties properties) {
Properties driverProperties = new Properties();
// 创建 DataSource 相应的 MetaObject
MetaObject metaDataSource = SystemMetaObject.forObject(dataSource);
// 遍历 properties 集合，该集合中存放了数据源需要的信息
for (Object key : properties.keySet()) {
String propertyName = (String) key;
// 以 "driver." 开头的配置项是对 DataSource 的配置，记录到 driverProperties 中
if (propertyName.startsWith(DRIVER_PROPERTY_PREFIX)) {
String value = properties.getProperty(propertyName);
driverProperties.setProperty(propertyName.substring(DRIVER_PROPERTY_PREFIX_LENGTH), value);
} else if (metaDataSource.hasSetter(propertyName)) { // 该属性是否有 set 方法
// 获取对应的属性值
String value = (String) properties.get(propertyName);
// 根据属性类型进行类型的转换，主要是 Integer, Long, Boolean 三种类型的转换
Object convertedValue = convertValue(metaDataSource, propertyName, value);
// 设置DataSource 的相关属性值
metaDataSource.setValue(propertyName, convertedValue);
} else {
throw new DataSourceException("Unknown DataSource property: " + propertyName);
}
}
// 设置 DataSource.driverProerties 属性值
if (driverProperties.size() > 0) {
metaDataSource.setValue("driverProperties", driverProperties);
}
}
// 返回数据源
@Override
public DataSource getDataSource() {
return dataSource;
}
// 类型转
private Object convertValue(MetaObject metaDataSource, String propertyName, String value) {
Object convertedValue = value;
Class<?> targetType = metaDataSource.getSetterType(propertyName);
if (targetType == Integer.class || targetType == int.class) {
convertedValue = Integer.valueOf(value);
} else if (targetType == Long.class || targetType == long.class) {
convertedValue = Long.valueOf(value);
} else if (targetType == Boolean.class || targetType == boolean.class) {
convertedValue = Boolean.valueOf(value);
}
return convertedValue;
}
}JndiDataSourceFactory relies on the DataSource configured by the user in the JNDI server, which is not necessary here.
PooledDataSourceFactory
PooledDataSourceFactory is mainly used to create PooledDataSource objects. It inherits the UnpooledDataSource class. The method of setting DataSource parameters reuses the setProperties method in UnpooledDataSource, except that the data source returns a PooledDataSource object.
public class PooledDataSourceFactory extends UnpooledDataSourceFactory {
public PooledDataSourceFactory() {
this.dataSource = new PooledDataSource();
}
}The above are the factory classes used by Mybatis to create the data source. Let's look at the main implementation of the data source.
Data source realization
UnpooledDataSource
UnpooledDataSource does not use the connection pool to create a database connection, each time a database connection is obtained, a new connection is created and returned;
public class UnpooledDataSource implements DataSource {
// 加载 Driver 类的类加载器
private ClassLoader driverClassLoader;
// 数据库连接驱动的相关配置
private Properties driverProperties;
// 缓存所有已注册的数据库连接驱动
private static Map<String, Driver> registeredDrivers = new ConcurrentHashMap<String, Driver>();
private String driver;
private String url;
private String username;
private String password;
// 是否自动提交
private Boolean autoCommit;
// 事物隔离级别
private Integer defaultTransactionIsolationLevel;
// 静态块，在初始化的时候，从 DriverManager 中获取所有的已注册的驱动信息，并缓存到该类的 registeredDrivers集合中
static {
Enumeration<Driver> drivers = DriverManager.getDrivers();
while (drivers.hasMoreElements()) {
Driver driver = drivers.nextElement();
registeredDrivers.put(driver.getClass().getName(), driver);
}
}
public UnpooledDataSource() {
}
public UnpooledDataSource(String driver, String url, String username, String password) {
this.driver = driver;
this.url = url;
this.username = username;
this.password = password;
}
}Next, let's look at the method to get the connection:
// 获取一个新的数据库连接
@Override
public Connection getConnection(String username, String password) throws SQLException {
return doGetConnection(username, password);
}
// 根据 properties 获取一个新的数据库连接
private Connection doGetConnection(Properties properties) throws SQLException {
// 初始化数据库驱动
initializeDriver();
// 通过 DriverManager 来获取一个数据库连接
Connection connection = DriverManager.getConnection(url, properties);
// 配置数据库连接的 autoCommit 和隔离级别
configureConnection(connection);
// 返回新连接
return connection;
}
// 初始化数据库驱动
private synchronized void initializeDriver() throws SQLException {
// 如果当前的驱动还没有注册，则进行注册
if (!registeredDrivers.containsKey(driver)) {
Class<?> driverType;
try {
if (driverClassLoader != null) {
driverType = Class.forName(driver, true, driverClassLoader);
} else {
driverType = Resources.classForName(driver);
}
// 创建驱动
Driver driverInstance = (Driver)driverType.newInstance();
// 向 JDBC 的 DriverManager 注册驱动
DriverManager.registerDriver(new DriverProxy(driverInstance));
// 向本类的 registeredDrivers 注册驱动
registeredDrivers.put(driver, driverInstance);
} catch (Exception e) {
throw new SQLException("Error setting driver on UnpooledDataSource. Cause: " + e);
}
}
}
// 设置数据库连接的 autoCommit 和隔离级别
private void configureConnection(Connection conn) throws SQLException {
if (autoCommit != null && autoCommit != conn.getAutoCommit()) {
conn.setAutoCommit(autoCommit);
}
if (defaultTransactionIsolationLevel != null) {
conn.setTransactionIsolation(defaultTransactionIsolationLevel);
}
}The above code is the main implementation logic of the UnpooledDataSource class. Every time a connection is obtained, a new connection is created from the database to return, and because the creation of a database connection is a time-consuming operation, and the database connection is a very precious resource. Creating one connection for each acquisition may cause system bottlenecks and slow down the response speed. At this time, a database connection pool is needed. Mybatis also provides its own database connection pool implementation, which is the PooledDataSource class.
Implementation of connection pool
PooledDataSource
PooledDataSource is a more complex class. It creates a new database connection using UnpooledDataSource. It does not manage java.sql.Connection objects, but instead manages PooledConnection objects. The real database connection objects and objects are encapsulated in PooledConnection. Its proxy object; in addition, because it is a connection pool, it also needs to manage the state of the connection pool, such as how many connections are idle and how many connections can be created. At this time, a class is needed to manage the connection pool objects. That is the PoolState object; first look at a UML diagram of PooledDataSource:
PooledConnection
Let’s take a look at the PooledConnection class first. It is mainly used to manage database connections. It is a proxy class that implements the InvocationHandler interface.
class PooledConnection implements InvocationHandler {
// close 方法
private static final String CLOSE = "close";
// 记录当前的 PooledConnection 对象所在的 PooledDataSource 对象，
// 该 PooledConnection 对象是从 PooledDataSource 对象中获取的，
// 当调用 close 方法时会将 PooledConnection 放回该 PooledDataSource 中去
private PooledDataSource dataSource;
// 真正的数据库连接
private Connection realConnection;
// 数据库连接的代理对象
private Connection proxyConnection;
// 从连接池中取出该连接的时间戳
private long checkoutTimestamp;
// 该连接创建的时间戳
private long createdTimestamp;
// 该连接最后一次被使用的时间戳
private long lastUsedTimestamp;
// 用于标识该连接所在的连接池，由URL+username+password 计算出来的hash值
private int connectionTypeCode;
// 该连接是否有效
private boolean valid;
// 创建连接
public PooledConnection(Connection connection, PooledDataSource dataSource) {
this.hashCode = connection.hashCode();
this.realConnection = connection;
this.dataSource = dataSource;
this.createdTimestamp = System.currentTimeMillis();
this.lastUsedTimestamp = System.currentTimeMillis();
this.valid = true;
this.proxyConnection = (Connection) Proxy.newProxyInstance(Connection.class.getClassLoader(), IFACES, this);
}
// 废弃该连接
public void invalidate() {
valid = false;
}
// 判断该连接是否有效，
// 1.判断 valid 字段
// 2.向数据库中发送检测测试的SQL，查看真正的连接还是否有效
public boolean isValid() {
return valid && realConnection != null && dataSource.pingConnection(this);
}
// setter / getter 方法
}Next, look at the invoke method. This method is the real proxy logic of the proxyConnection proxy object. It will proxy the close method and check the connection before calling the real connection.
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
String methodName = method.getName();
// 如果执行的方法是 close 方法，则会把当前连接放回到 连接池中去，供下次使用，而不是真正的关闭数据库连接
if (CLOSE.hashCode() == methodName.hashCode() && CLOSE.equals(methodName)) {
dataSource.pushConnection(this);
return null;
} else {
try {
// 如果不是 close 方法，则 调用 真正的数据库连接执行
if (!Object.class.equals(method.getDeclaringClass())) {
// 执行之前，需要进行连接的检测
checkConnection();
}
// 调用数据库真正的连接进行执行
return method.invoke(realConnection, args);
} catch (Throwable t) {
throw ExceptionUtil.unwrapThrowable(t);
}
}
}PoolState
The PoolState class is mainly used to manage the state of the connection pool, such as which connections are idle, which are active, and how many connections can be created. This class only defines some properties to control the state of the connection pool, and there is no method.
public class PoolState {
// 该 PoolState 属于哪个 PooledDataSource
protected PooledDataSource dataSource;
// 来用存放空闲的 pooledConnection 连接
protected final List<PooledConnection> idleConnections = new ArrayList<PooledConnection>();
// 用来存放活跃的 PooledConnection 连接
protected final List<PooledConnection> activeConnections = new ArrayList<PooledConnection>();
// 请求数据库连接的次数
protected long requestCount = 0;
// 获取连接的累计时间
protected long accumulatedRequestTime = 0;
// checkoutTime 表示从连接池中获取连接到归还连接的时间
// accumulatedCheckoutTime 记录了所有连接的累计 checkoutTime 时长
protected long accumulatedCheckoutTime = 0;
// 连接超时的连接个数
protected long claimedOverdueConnectionCount = 0;
// 累计超时时间
protected long accumulatedCheckoutTimeOfOverdueConnections = 0;
// 累计等待时间
protected long accumulatedWaitTime = 0;
// 等待次数
protected long hadToWaitCount = 0;
// 无效的连接数
protected long badConnectionCount = 0;
// setter / getter 方法
}PooledDataSource
PooledDataSource It is a simple, synchronized, thread-safe database connection pool. After
knowing that UnpooledDataSource is used to create new connections to the database, PooledConnection is used to manage connections in the connection pool, and PoolState is used to manage the state of the connection pool. A logic of PooledDataSource,
there are mainly the following methods in this class:
- Method to get database connection popConnection
- Put the connection back to the connection pool method pushConnection,
- The method pingConnection to check whether the database connection is valid,
- There is also the method forceCloseAll to close all connections in the connection pool.
Next, let's see how these methods are implemented. Before looking at it, let's take a look at some of the attributes defined by this class:
public class PooledDataSource implements DataSource {
// 连接池的状态
private final PoolState state = new PoolState(this);
// 用来创建真正的数据库连接对象
private final UnpooledDataSource dataSource;
// 最大活跃的连接数，默认为 10
protected int poolMaximumActiveConnections = 10;
// 最大空闲连接数，默认为 5
protected int poolMaximumIdleConnections = 5;
// 最大获取连接的时长
protected int poolMaximumCheckoutTime = 20000;
// 在无法获取到连接时，最大等待的时间
protected int poolTimeToWait = 20000;
// 在检测一个连接是否可用时，会向数据库发送一个测试 SQL
protected String poolPingQuery = "NO PING QUERY SET";
// 是否允许发送测试 SQL
protected boolean poolPingEnabled;
// 当连接超过 poolPingConnectionsNotUsedFor 毫秒未使用时，会发送一次测试 SQL 语句，测试连接是否正常
protected int poolPingConnectionsNotUsedFor;
// 标志着当前的连接池，是 url+username+password 的 hash 值
private int expectedConnectionTypeCode;
// 创建连接池
public PooledDataSource(String driver, String url, String username, String password) {
dataSource = new UnpooledDataSource(driver, url, username, password);
expectedConnectionTypeCode = assembleConnectionTypeCode(dataSource.getUrl(), dataSource.getUsername(), dataSource.getPassword());
}
// 生成 hash 值
private int assembleConnectionTypeCode(String url, String username, String password) {
return ("" + url + username + password).hashCode();
}
// setter / getter 方法
}Get connection popConnection
Next
, let’s take a look at the implementation logic of obtaining connections from the database connection pool: The method of obtaining connections from the connection pool is mainly implemented in popConnection. Let’s first look at a flowchart of it:
// 获取连接
@Override
public Connection getConnection(String username, String password) throws SQLException {
return popConnection(username, password).getProxyConnection();
}
// 从连接池中获取连接
private PooledConnection popConnection(String username, String password) throws SQLException {
// 等待的个数
boolean countedWait = false;
// PooledConnection 对象
PooledConnection conn = null;
long t = System.currentTimeMillis();
// 无效的连接个数
int localBadConnectionCount = 0;
while (conn == null) {
synchronized (state) {
// 检测是否还有空闲的连接
if (!state.idleConnections.isEmpty()) {
// 连接池中还有空闲的连接，则直接获取连接返回
conn = state.idleConnections.remove(0);
} else {
// 连接池中已经没有空闲连接了
if (state.activeConnections.size() < poolMaximumActiveConnections) {
// 活跃的连接数没有达到最大值，则创建一个新的数据库连接
conn = new PooledConnection(dataSource.getConnection(), this);
} else {
// 如果活跃的连接数已经达到允许的最大值了，则不能创建新的数据库连接
// 获取最先创建的那个活跃的连接
PooledConnection oldestActiveConnection = state.activeConnections.get(0);
long longestCheckoutTime = oldestActiveConnection.getCheckoutTime();
// 检测该连接是否超时
if (longestCheckoutTime > poolMaximumCheckoutTime) {
// 如果该连接超时，则进行相应的统计
state.claimedOverdueConnectionCount++;
state.accumulatedCheckoutTimeOfOverdueConnections += longestCheckoutTime;
state.accumulatedCheckoutTime += longestCheckoutTime;
// 将超时连接移出 activeConnections 集合
state.activeConnections.remove(oldestActiveConnection);
if (!oldestActiveConnection.getRealConnection().getAutoCommit()) {
// 如果超时未提交，则自动回滚
oldestActiveConnection.getRealConnection().rollback();
}
// 创建新的 PooledConnection 对象，但是真正的数据库连接并没有创建
conn = new PooledConnection(oldestActiveConnection.getRealConnection(), this);
conn.setCreatedTimestamp(oldestActiveConnection.getCreatedTimestamp());
conn.setLastUsedTimestamp(oldestActiveConnection.getLastUsedTimestamp());
// 设置该超时的连接为无效
oldestActiveConnection.invalidate();
} else {
// 如果无空闲连接，无法创建新的连接且无超时连接，则只能阻塞等待
// Must wait
try {
if (!countedWait) {
state.hadToWaitCount++; // 等待次数
countedWait = true;
}
long wt = System.currentTimeMillis();
// 阻塞等待
state.wait(poolTimeToWait);
state.accumulatedWaitTime += System.currentTimeMillis() - wt;
} catch (InterruptedException e) {
break;
}
}
}
}
// 已经获取到连接
if (conn != null) {
if (conn.isValid()) {
// 如果连连接有效，事务未提交则回滚
if (!conn.getRealConnection().getAutoCommit()) {
conn.getRealConnection().rollback();
}
// 设置 PooledConnection 相关属性
conn.setConnectionTypeCode(assembleConnectionTypeCode(dataSource.getUrl(), username, password));
conn.setCheckoutTimestamp(System.currentTimeMillis());
conn.setLastUsedTimestamp(System.currentTimeMillis());
// 把连接加入到活跃集合中去
state.activeConnections.add(conn);
state.requestCount++;
state.accumulatedRequestTime += System.currentTimeMillis() - t;
} else {
// 无效连接
state.badConnectionCount++;
localBadConnectionCount++;
conn = null;
}
}
}
}
return conn;
}The above is the main logic to get the connection from the connection pool.
Release link pushConnection
Now let’s look at when the close method is executed, the connection will be put into the connection pool for reuse next time. The method of putting the connection into the connection pool is the pushConnection method, which is also a main method of the PooledDataSource class. First look at its flow chart:
// 把不用的连接放入到连接池中
protected void pushConnection(PooledConnection conn) throws SQLException {
synchronized (state) {
// 首先从活跃的集合中移除掉该连接
state.activeConnections.remove(conn);
// 检测连接是否有效
if (conn.isValid()) {
// 如果空闲连接数没有达到最大值，且 PooledConnection 为该连接池的连接
if (state.idleConnections.size() < poolMaximumIdleConnections && conn.getConnectionTypeCode() == expectedConnectionTypeCode) {
// 累计 checkout 时长
state.accumulatedCheckoutTime += conn.getCheckoutTime();
// 事务回滚
if (!conn.getRealConnection().getAutoCommit()) {
conn.getRealConnection().rollback();
}
// 为返还的连接创建新的 PooledConnection 对象
PooledConnection newConn = new PooledConnection(conn.getRealConnection(), this);
// 把该连接添加的空闲链表中
state.idleConnections.add(newConn);
newConn.setCreatedTimestamp(conn.getCreatedTimestamp());
newConn.setLastUsedTimestamp(conn.getLastUsedTimestamp());
// 设置该连接为无效状态
conn.invalidate();
// 唤醒阻塞等待的线程
state.notifyAll();
} else {
// 如果空闲连接数已经达到最大值
state.accumulatedCheckoutTime += conn.getCheckoutTime();
if (!conn.getRealConnection().getAutoCommit()) {
conn.getRealConnection().rollback();
}
// 则关闭真正的数据库链接
conn.getRealConnection().close();
// 设置该连接为无效状态
conn.invalidate();
}
} else {
// 无效连接个数加1
state.badConnectionCount++;
}
}
}The above code is to put unused connections into the connection pool for next use,
In the above two methods, the isValid method is called to check whether the connection is available. In addition to checking the valid field, this method also calls the pingConnection method to try to make the database execute the test SQL statement, so as to check whether the real database connection object is still normal Available.
// 检测连接是否可用
public boolean isValid() {
return valid && realConnection != null && dataSource.pingConnection(this);
}
// 向数据库发送测试 SQL 来检测真正的数据库连接是否可用
protected boolean pingConnection(PooledConnection conn) {
// 结果
boolean result = true;
try {
// 检测真正的数据库连接是否已经关闭
result = !conn.getRealConnection().isClosed();
} catch (SQLException e) {
result = false;
}
// 如果真正的数据库连接还没关闭
if (result) {
// 是否执行测试 SQL 语句
if (poolPingEnabled) {
// 长时间（poolPingConnectionsNotUsedFor 指定的时长）未使用的连接，才需要ping操作来检测连接是否正常
if (poolPingConnectionsNotUsedFor >= 0 && conn.getTimeElapsedSinceLastUse() > poolPingConnectionsNotUsedFor) {
try {
// 发送测试 SQL 语句执行
Connection realConn = conn.getRealConnection();
Statement statement = realConn.createStatement();
ResultSet rs = statement.executeQuery(poolPingQuery);
rs.close();
statement.close();
if (!realConn.getAutoCommit()) {
realConn.rollback();
}
result = true;
} catch (Exception e) {
try {
conn.getRealConnection().close();
} catch (Exception e2) {
}
result = false;
}
}
}
}
return result;
}In addition, when modifying the corresponding fields of PooledDataSource, such as the URL of the database, user name or password, etc., all connections in the connection pool need to be closed, and then reinitialized when the connection is obtained. The method to close all connections in the connection pool is forceCloseAll.
Close connection forceCloseAll
public void forceCloseAll() {
synchronized (state) {
expectedConnectionTypeCode = assembleConnectionTypeCode(dataSource.getUrl(), dataSource.getUsername(), dataSource.getPassword());
// 处理活跃的连接
for (int i = state.activeConnections.size(); i > 0; i--) {
try {
PooledConnection conn = state.activeConnections.remove(i - 1);
// 设置连接为无效状态
conn.invalidate();
// 获取数据库真正的连接
Connection realConn = conn.getRealConnection();
// 事物回滚
if (!realConn.getAutoCommit()) {
realConn.rollback();
}
// 关闭数据库连接
realConn.close();
} catch (Exception e) {
// ignore
}
}
// 处理空闲的连接
for (int i = state.idleConnections.size(); i > 0; i--) {
try {
PooledConnection conn = state.idleConnections.remove(i - 1);
// 设置为无效状态
conn.invalidate();
Connection realConn = conn.getRealConnection();
if (!realConn.getAutoCommit()) {
realConn.rollback();
}
realConn.close();
} catch (Exception e) {
}
}
}
}to sum up
In the connection pool, the maximum number of connections and the maximum number of free connections in the connection pool are mentioned. There are judgments in obtaining connections and putting connections into the connection pool.
-
Get connection: first get it from the connection pool. If there are no free connections in the connection pool, it will judge whether the current number of active connections has reached the maximum allowable value. If not, you can also create a new connection. Then put it in the active set for use, if the current active has reached the maximum value, it will block.
- Return the connection to the connection pool. When returning the connection, make a judgment. If the number of free connections has reached the maximum allowable value, the real database connection will be directly closed, otherwise the connection will be placed in the free set for next use.
In the Mybatis data source, the main code logic is still in the connection pool class PooledDataSource. For the method popConnection to get the connection and the method pushConnection to return the connection, you need to combine the above figure to see it clearly.