Mybaits source code analysis (four) data source
Foreword : For the ORM framework, data sources are indispensable components. Although the mybaits framework integrates spring, data source objects can be specified externally and built into SqlSessionFactoryBean, but the realization principle of mybaits' own data sources is still worth exploring , This article will conduct a detailed analysis of the data source structure of the MyBatis framework, and in-depth analysis of the connection pool of MyBatis.
The content of this article is preliminarily divided into the following parts
. Overview of the
mybaits data source The construction of the mybaits data source and
the analysis of the use of UnpooledDataSource The analysis of the
PooledDataSource
simple data source case demonstration
1. Overview of Mybaits data source
1. Data source classification
MyBatis divides the data source DataSource into three types of
UnpooledDataSource: Data source that does not use connection pool
PooledDataSource: Data source that uses connection pool JndiDataSource: Data source
implemented using JNDI
2. Data source class structure
DatasourceFacotry data source is created, UnpooledDataSourceFacotry create UnpooledDataSource, PooledDataSourceFacotry create PooledDataSource, and UnpooledDataSourceFacotry inherit UnpooledDataSourceFacotry Universal Settings properties for use, PooledDataSource internal maintains UnpooledDataSource instance, create a database link for real, internal PooledDataSource connection pool cache is used to enhance And wrap the PooledConnection of the real connection object and the proxy connection object .
The class structure is as follows:
3. Creation of data source
We mainly explain UnpooledDataSource and PooledDataSource. Among them, UnpooledDataSource is a data source that does not use connection pooling. PooledDataSource wraps UnpooledDataSource and expands the connection pool function. The UnpooledDataSource is used to create a connection based on the internal use of the list container to realize the connection pool function.
The creation of the data source adopts the factory mode. The creation of UnpooledDataSource and PooledDataSource are created by different factory classes that implement DataSourceFactory, and some configuration properties are set. The following is the process of UnpooledDataSourceFactory to create a data source.
public UnpooledDataSourceFactory() { // 构造函数创建对应工厂的数据源
this.dataSource = new UnpooledDataSource();
}
public void setProperties(Properties properties) { // 设置属性
Properties driverProperties = new Properties();
MetaObject metaDataSource = SystemMetaObject.forObject(dataSource);
for (Object key : properties.keySet()) {
String propertyName = (String) key;
if (propertyName.startsWith(DRIVER_PROPERTY_PREFIX)) { // 设置驱动属性
String value = properties.getProperty(propertyName);
driverProperties.setProperty(propertyName.substring(DRIVER_PROPERTY_PREFIX_LENGTH), value);
} else if (metaDataSource.hasSetter(propertyName)) { // 其他常规数据，依据有无set方法进行设置
String value = (String) properties.get(propertyName);
Object convertedValue = convertValue(metaDataSource, propertyName, value);
metaDataSource.setValue(propertyName, convertedValue);
} else {
throw new DataSourceException("Unknown DataSource property: " + propertyName);
}
}
if (driverProperties.size() > 0) {
metaDataSource.setValue("driverProperties", driverProperties);
}
}Second, the construction of mybaits data source
The above is a brief introduction to the classification of data sources, the structure of the class diagram, and the process of creating data sources for the factory class. Knowing that the framework will create different data sources according to different factory classes, then how to create a data source factory class in mybaits What about
1. Data source construction
Below is the xml configuration file
<dataSource type="POOLED">
<property name="driver" value="${db.driver}" />
<property name="url" value="${db.url}" />
<property name="username" value="${db.username}" />
<property name="password" value="${db.password}" />
</dataSource>When the data source is created, according to the type attribute of the dataSource node, the corresponding factory class is instantiated, the data source is created, and the props attribute of the child node is added to the set method of the data source, POOLED means that PooledDataSource data will be created Source, Unpooled means that the UnpooledDataSource data source will be created.
private DataSourceFactory dataSourceElement(XNode context) throws Exception {
if (context != null) { // context是解析dataSource节点的内容
String type = context.getStringAttribute("type"); // type属性
Properties props = context.getChildrenAsProperties(); // 子属性键值对
DataSourceFactory factory = (DataSourceFactory) resolveClass(type).newInstance();
factory.setProperties(props);
return factory;
}
throw new BuilderException("Environment declaration requires a DataSourceFactory.");
}2. Use of data sources
The function of the data source is to obtain the connection object connection, so in the execution process, when do we obtain the connection object from the data source? From the previous transaction, we learned that the connection is packaged in the Transaction object. When we create the SQLSession and execute the sql statement through the SqlSession, Mybatis will call the DataSource data source to create the Connection object.
Three, UnpooledDataSource detailed explanation
The UnpooledDataSource data source is created by the UnpooledDataSourceFactory, this data source is to directly instantiate the connection object, and then for the caller to use, that is, using the getConnection method of the UnpooledDataSource data source, each call will generate a Connection object.
public UnpooledDataSource(String driver, String url, String username, String password) {
this.driver = driver;
this.url = url;
this.username = username;
this.password = password;
}
public Connection getConnection() throws SQLException {
return doGetConnection(username, password);
}
private Connection doGetConnection(String username, String password) throws SQLException {
Properties props = new Properties();
if (driverProperties != null) {
props.putAll(driverProperties); // 其他参数
}
if (username != null) { // username参数
props.setProperty("user", username);
}
if (password != null) { // password参数
props.setProperty("password", password);
}
return doGetConnection(props);
}
private Connection doGetConnection(Properties properties) throws SQLException {
initializeDriver(); // 初始化驱动
Connection connection = DriverManager.getConnection(url, properties); // 获得连接
configureConnection(connection);
return connection;
} From the above, we know that the getConnection of UnpooledDataSource is the process of directly creating the jdbc connection Connection object.
Four, PooledDataSource detailed explanation
PooledDataSource data source In the above, I introduced the data source that it has a connection pool. Now we look at the basic principle of PooledDataSource: PooledDataSource data source wraps Connection object into PooledConnection object and puts it in PoolState container for maintenance. MyBatis divides the PooledConnection in the connection pool into two states: idle and active. The PooledConnection objects of these two states are stored in the two List collections of idleConnections and activeConnections in the PoolState container, respectively.
The main function of PooledDataSource's internal packaging of UnpooledDataSource is that when you need to create a data source, you can call UnpooledDataSource to create a data source.
1. The main attributes of PoolState and PooledDataSource
public class PoolState {
protected PooledDataSource dataSource;
// 空闲连接
protected final List<PooledConnection> idleConnections = new ArrayList<PooledConnection>();
// 活跃连接
protected final List<PooledConnection> activeConnections = new ArrayList<PooledConnection>();
public class PooledDataSource implements DataSource {
private final PoolState state = new PoolState(this); // 封装空闲连接和活跃连接的集合
private final UnpooledDataSource dataSource; // unpooledDataSource用于真实获取连接用途
protected int poolMaximumActiveConnections = 10; // 最大活跃个数
protected int poolMaximumIdleConnections = 5; // 最大空闲个数
protected int poolMaximumCheckoutTime = 20000; // 最大工作超时时间
protected int poolTimeToWait = 20000; // 未获得连接的等待时间
protected String poolPingQuery = "NO PING QUERY SET"; // 心跳查询
protected boolean poolPingEnabled = false;
protected int poolPingConnectionsNotUsedFor = 0;
private int expectedConnectionTypeCode;
public PooledDataSource() {
dataSource = new UnpooledDataSource();
} 2. Core execution logic
After the attribute structure of the class is finished, the workflow of the connection pool is clear. Since it is a connection pool, from the perspective of the update of the connection pool, there is a process of opening a connection and putting it back into the connection pool when it is used up. This process corresponds to popConnection. And pushConnection method.
1) Take out the connection popConnection
The process of taking out the connection is basically if the idle is there, then it is taken from the idle, otherwise, it is judged whether the active size is greater than the maximum acitve size, if it is less, then a new link can be created, if it is greater than or equal to, then it cannot be created. , Take out the first used link to determine whether it is expired, if it expires, rollback, update to a link, if it does not expire, wait, and then continue to perform the acquisition.
private PooledConnection popConnection(String username, String password) throws SQLException {
boolean countedWait = false;
PooledConnection conn = null; // 连接包装
long t = System.currentTimeMillis();
int localBadConnectionCount = 0;
while (conn == null) {
synchronized (state) { // 锁为容器对象
// 如果空闲不为空。
if (!state.idleConnections.isEmpty()) {
conn = state.idleConnections.remove(0); // 从空闲中取
} else {
// 如果活跃尺寸少于最大活跃
if (state.activeConnections.size() < poolMaximumActiveConnections) {
创建一个新的--这个datasource 是UnpooledDataSource，即获取真实连接。
conn = new PooledConnection(dataSource.getConnection(), this);
} else {
// 否则从旧的中取
PooledConnection oldestActiveConnection = state.activeConnections.get(0);
long longestCheckoutTime = oldestActiveConnection.getCheckoutTime();
// 判断最先用的是否超时
if (longestCheckoutTime > poolMaximumCheckoutTime) {
// 是超时就移除旧的
state.activeConnections.remove(oldestActiveConnection);
if (!oldestActiveConnection.getRealConnection().getAutoCommit()) {
try { // 并且旧的要回滚
oldestActiveConnection.getRealConnection().rollback();
} catch (SQLException e) {
}
}
// 把旧的真实连接取出，包装成一个新的PooledConnection
conn = new PooledConnection(oldestActiveConnection.getRealConnection(), this);
oldestActiveConnection.invalidate();
} else {
try {
// 如果没法获得连接就等待，等待恢复后，从新执行while下面的流程判断。
if (!countedWait) {
state.hadToWaitCount++;
countedWait = true;
}
if (log.isDebugEnabled()) {
log.debug("Waiting as long as " + poolTimeToWait + " milliseconds for connection.");
}
long wt = System.currentTimeMillis();
state.wait(poolTimeToWait); // 等待
} catch (InterruptedException e) {
break;
}
}
}
}
if (conn != null) {
if (conn.isValid()) {
if (!conn.getRealConnection().getAutoCommit()) {
conn.getRealConnection().rollback(); // 再次保证全部进行了rollBack，才可以使用
}
state.activeConnections.add(conn); // 添加到activeConnections
} else {
conn = null;
}
}
}
}
return conn;
}The flow chart is shown in the figure below
2) Return connection oushConnection
From the above analysis, we know that our active collection will be created when there is no excess capacity, so how does our first level judge the number of idle containers come from? This is to analyze the process of return connection below.
The process of returning the connection is much simpler than the above, that is, if the idel container is less than the maximum limit of acitve, then it is placed in the free container, if it exceeds, it will be closed directly (note that this is a real connection call to close the connection)
protected void pushConnection(PooledConnection conn) throws SQLException {
synchronized (state) {
state.activeConnections.remove(conn); // 活跃池移除
if (conn.isValid()) {
// 如果空闲池少于最大空闲连接，就直接将活跃池中移除的添加到空闲池
if (state.idleConnections.size() < poolMaximumIdleConnections && conn.getConnectionTypeCode() == expectedConnectionTypeCode) {
state.accumulatedCheckoutTime += conn.getCheckoutTime();
if (!conn.getRealConnection().getAutoCommit()) {
conn.getRealConnection().rollback();
}
PooledConnection newConn = new PooledConnection(conn.getRealConnection(), this);
state.idleConnections.add(newConn);
newConn.setCreatedTimestamp(conn.getCreatedTimestamp());
newConn.setLastUsedTimestamp(conn.getLastUsedTimestamp());
conn.invalidate();
state.notifyAll();
// 否则直接关闭
} else {
state.accumulatedCheckoutTime += conn.getCheckoutTime();
if (!conn.getRealConnection().getAutoCommit()) {
conn.getRealConnection().rollback();
}
conn.getRealConnection().close();
conn.invalidate();
}
} else {
// ....
}
}
}3. PooledConnection analysis
What we analyzed above is the pushConnection and popConnection methods that directly manipulate the connection pool, so how do we call these two methods externally? First of all, the popConnection method is very simple, it is called by the connection pool interface method getConnection. And what about the pushConnection method? , Do we manually call this method every time we finish using the connection.
This is definitely not possible. What we hope is that every time the connection is used for close, the close is not actually performed, but the pushConnection method is called. Do not even think about this function. It must be a dynamic proxy. The specific implementation is that our PooledDataSource gets the connection ProxyConnection wrapped by PooledConnection.
The following are the main properties and constructors of PooledConnection.
class PooledConnection implements InvocationHandler {
private static final String CLOSE = "close";
private static final Class<?>[] IFACES = new Class<?>[] { Connection.class };
private int hashCode = 0;
private PooledDataSource dataSource;
private Connection realConnection;
private Connection proxyConnection;
private long checkoutTimestamp;
private long createdTimestamp;
private long lastUsedTimestamp;
private boolean valid;
public PooledConnection(Connection connection, PooledDataSource dataSource) {
this.hashCode = connection.hashCode();
this.realConnection = connection;
this.dataSource = dataSource;
this.valid = true;
this.proxyConnection = (Connection) Proxy.newProxyInstance(Connection.class.getClassLoader(), IFACES, this);
}As shown above, PooledConnection is a proxy enhancement that implements the InvocationHandler class, and a proxy connection object is created in the constructor. We only need to see how the invoke method of this class is enhanced.
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
String methodName = method.getName();
if (CLOSE.hashCode() == methodName.hashCode() && CLOSE.equals(methodName)) {
dataSource.pushConnection(this); // 核心 -- 如果Colse方法就返回池中
return null;
} else {
try {
if (!Object.class.equals(method.getDeclaringClass())) {
checkConnection();
}
// 普通方法直接执行
return method.invoke(realConnection, args);
} catch (Throwable t) {
throw ExceptionUtil.unwrapThrowable(t);
}
}
}Five, simple data source case demonstration
1. MyPoolConnection data connection packaging and proxy enhancement and instantiation
public class MyPoolConnection implements InvocationHandler {
private static final Class<?>[] IFACES = new Class<?>[] { Connection.class };
private MyPoolDataSource dataSource; // 保持数据源的链接
private Connection realConnection; // 真实连接
private Connection proxyConnection; // 代理连接
private long checkoutTimestamp; // 检测使用时间
public MyPoolConnection(Connection connection, MyPoolDataSource dataSource) {
this.realConnection = connection;
this.dataSource = dataSource;
this.proxyConnection = (Connection) Proxy.newProxyInstance(Connection.class.getClassLoader(), IFACES, this);
}
public MyPoolDataSource getDataSource() {
return dataSource;
}
public void setDataSource(MyPoolDataSource dataSource) {
this.dataSource = dataSource;
}
public Connection getRealConnection() {
return realConnection;
}
public void setRealConnection(Connection realConnection) {
this.realConnection = realConnection;
}
public Connection getProxyConnection() {
return proxyConnection;
}
public void setProxyConnection(Connection proxyConnection) {
this.proxyConnection = proxyConnection;
}
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
String methodName = method.getName();
if ("close".equals(methodName)) {
dataSource.pushConnection(this);
} else {
if (!Object.class.equals(method.getDeclaringClass())) {
// 检测链接是否有效
}
return method.invoke(realConnection, args);
}
return null;
}
public long getCheckoutTime() {
return System.currentTimeMillis() - checkoutTimestamp;
}
public void setCheckoutTimestamp(long checkoutTimestamp) {
this.checkoutTimestamp = checkoutTimestamp;
}
}2. MyPoolDataSource data source
public class MyPoolDataSource extends AbstractDataSource {
// 数据源基本属性
private String driver;
private String url;
private String username;
private String password;
// 链接池属性
protected int poolMaximumActiveConnections = 10;
protected int poolMaximumIdleConnections = 5;
protected int poolMaximumCheckoutTime = 20000;
protected int poolTimeToWait = 20000;
// 连接容器
protected final List<MyPoolConnection> idleConnections = new ArrayList<MyPoolConnection>();
protected final List<MyPoolConnection> activeConnections = new ArrayList<MyPoolConnection>();
public MyPoolDataSource(String driver, String url, String username, String password) {
super();
this.driver = driver;
this.url = url;
this.username = username;
this.password = password;
}
@Override
public Connection getConnection() throws SQLException {
return popConnection(username, password).getProxyConnection();
}
@Override
public Connection getConnection(String username, String password) throws SQLException {
return popConnection(username, password).getProxyConnection();
}
/**
* 真实创建链接
*/
public Connection doGetConnection() throws SQLException {
initDriver();
Properties props = new Properties();
if (username != null) {
props.setProperty("user", username);
}
if (password != null) {
props.setProperty("password", password);
}
Connection connection = DriverManager.getConnection(url, props);
return connection;
}
/**
* 用于判断是否已经加载驱动
*/
private static Map<String, Driver> registeredDrivers = new ConcurrentHashMap<String, Driver>();
/**
* 初始化数据源驱动
*/
private void initDriver() {
if (registeredDrivers.containsKey(this.driver)) {
try {
Class<?> driverType = Class.forName(this.driver);
Driver driverInstance = (Driver) driverType.newInstance();
registeredDrivers.put(this.driver, driverInstance);
} catch (Exception e) {
throw new ExceptionInInitializerError(e);
}
}
}
/**
* 重点方法： 返回池中
*/
public void pushConnection(MyPoolConnection conn) throws SQLException {
synchronized (this) {
activeConnections.remove(conn);
if (!conn.getRealConnection().getAutoCommit()) {
conn.getRealConnection().rollback();
}
// 空闲池有位置
if (idleConnections.size() < poolMaximumIdleConnections) {
MyPoolConnection newConn = new MyPoolConnection(conn.getRealConnection(), this);
idleConnections.add(newConn);
this.notifyAll(); // 通知有可能等待取连接的线程
} else {
// 直接真实释放
conn.getRealConnection().close();
}
}
}
/**
* 重点方法：从池中取连接
*/
public MyPoolConnection popConnection(String username, String password) throws SQLException {
MyPoolConnection conn = null;
while (conn == null) {
synchronized (this) {
// idel池有货
if (idleConnections.size() > 0) {
conn = idleConnections.remove(0);
} else {
// active池未满
if (activeConnections.size() < poolMaximumActiveConnections) {
Connection connection = this.doGetConnection();
conn = new MyPoolConnection(connection, this);
activeConnections.add(conn);
} else {
MyPoolConnection oldConnection = this.activeConnections.get(0);
long longestCheckoutTime = oldConnection.getCheckoutTime();
// 旧的超时
if (longestCheckoutTime > poolMaximumCheckoutTime) {
Connection connection = oldConnection.getRealConnection();
if (!connection.getAutoCommit()) {
connection.rollback();
}
conn = new MyPoolConnection(connection, this);
activeConnections.remove(oldConnection);
// 没取到
} else {
try {
wait(poolTimeToWait); // wait等待池等待poolTimeToWait时间，如果notifyAll就会提前取消等待
} catch (InterruptedException e) {
break;
}
}
}
}
}
}
// 取到连接了
if (conn != null) {
if (!conn.getRealConnection().getAutoCommit()) {
conn.getRealConnection().rollback();
}
conn.setCheckoutTimestamp(System.currentTimeMillis());
}
// 没取到报错
if (conn == null) {
throw new SQLException(
"PooledDataSource: Unknown severe error condition. The connection pool returned a null connection.");
}
return conn;
}
public String getDriver() {
return driver;
}
public void setDriver(String driver) {
this.driver = driver;
}
public String getUrl() {
return url;
}
public void setUrl(String url) {
this.url = url;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
public int getPoolMaximumActiveConnections() {
return poolMaximumActiveConnections;
}
public void setPoolMaximumActiveConnections(int poolMaximumActiveConnections) {
this.poolMaximumActiveConnections = poolMaximumActiveConnections;
}
public int getPoolMaximumIdleConnections() {
return poolMaximumIdleConnections;
}
public void setPoolMaximumIdleConnections(int poolMaximumIdleConnections) {
this.poolMaximumIdleConnections = poolMaximumIdleConnections;
}
public int getPoolMaximumCheckoutTime() {
return poolMaximumCheckoutTime;
}
public void setPoolMaximumCheckoutTime(int poolMaximumCheckoutTime) {
this.poolMaximumCheckoutTime = poolMaximumCheckoutTime;
}
public int getPoolTimeToWait() {
return poolTimeToWait;
}
public void setPoolTimeToWait(int poolTimeToWait) {
this.poolTimeToWait = poolTimeToWait;
}
public static Map<String, Driver> getRegisteredDrivers() {
return registeredDrivers;
}
public static void setRegisteredDrivers(Map<String, Driver> registeredDrivers) {
MyPoolDataSource.registeredDrivers = registeredDrivers;
}
public String logSate() {
return "空闲池" + idleConnections.size() + "," + "活跃池" + activeConnections.size();
}
}3. Test and use
/**
* 数据源案例的测试
*/
public class DataSourceDemo {
/**
* 测试逻辑：(空闲max参数5，活跃max参数10) 建立30个线程，执行一个sql，然后每个线程执行完，打印一次池状态，
* 并且10次再打印一次池状态。
*/
public static void main(String[] args) throws SQLException {
String driver = "com.mysql.jdbc.Driver";
String url = "jdbc:mysql://localhost:3306/java_base?characterEncoding=utf8";
String username = "root";
String password = "123456";
MyPoolDataSource pool = new MyPoolDataSource(driver, url, username, password);
excute(pool);
CyclicBarrier cb = new CyclicBarrier(10, () -> {
System.out.println("10个线程执行完：" + pool.logSate());
});
// 30个线程执行sql
for (int i = 0; i < 30; i++) {
new Thread(new Runnable() {
@Override
public void run() {
try {
excute(pool);
System.out.println("单个线程执行完：" + pool.logSate());
} catch (SQLException e) {
e.printStackTrace();
}
try {
cb.await();
} catch (Exception e1) {
e1.printStackTrace();
}
};
}, "thread" + i).start();
}
}
private static void excute(MyPoolDataSource pool) throws SQLException {
Connection connection = pool.getConnection();
PreparedStatement prepareStatement = connection.prepareStatement("select * from blog");
ResultSet executeQuery = prepareStatement.executeQuery();
while (executeQuery.next()) {
// System.out.println(executeQuery.getObject("id"));
}
connection.close();
}
}
end！