table of Contents

Sharding-Jdbc source code analysis

Sharding-Jdbc source code analysis

Apache Sharding-Sphere series catalog ( https://www.cnblogs.com/binarylei/p/12217637.html )

Before looking at Sharding-Jdbc source, it is strongly recommended that you have read the official website of the article:

Apache Sharding-Jdbc data pieces

JDBC calls as follows: APP -> ORM -> JDBC -> PROXY -> MySQL. To complete sub-library sub-table data, it may be carried out five anywhere, Sharding-Jdbc library sub-table is divided in the JDBC layers, Sharding-Proxy is sub-divided library table PROXY.

Sharding-Jdbc is a lightweight sub-library sub-table frame, the most important use is the preparation of sub-library sub-table strategy, and use the rest of the normal MySQL driver, almost do not change the code. Refer to the specific use: the Apache-Sharding example using the Jdbc

try(DataSource dataSource =  ShardingDataSourceFactory.createDataSource(
    createDataSourceMap(), shardingRuleConfig, new Properties()) {
    connection Connection = dataSource.getConnection();
    ...
}

1. Sharding-Jdbc packet structure

sharding-jdbc  
    ├── sharding-jdbc-core      重写DataSource/Connection/Statement/ResultSet四大对象
    └── sharding-jdbc-orchestration        配置中心
sharding-core
    ├── sharding-core-api       接口和配置类  
    ├── sharding-core-common    通用分片策略实现...
    ├── sharding-core-entry     SQL解析、路由、改写，核心类BaseShardingEngine
    ├── sharding-core-route     SQL路由，核心类StatementRoutingEngine
    ├── sharding-core-rewrite   SQL改写，核心类ShardingSQLRewriteEngine
    ├── sharding-core-execute   SQL执行，核心类ShardingExecuteEngine
    └── sharding-core-merge     结果合并，核心类MergeEngine
shardingsphere-sql-parser 
    ├── shardingsphere-sql-parser-spi       SQLParserEntry，用于初始化SQLParser
    ├── shardingsphere-sql-parser-engine    SQL解析，核心类SQLParseEngine
    ├── shardingsphere-sql-parser-relation
    └── shardingsphere-sql-parser-mysql     MySQL解析器，核心类MySQLParserEntry和MySQLParser
shardingsphere-underlying           基础接口和api
    ├── shardingsphere-rewrite      SQLRewriteEngine接口
    ├── shardingsphere-execute      QueryResult查询结果
    └── shardingsphere-merge        MergeEngine接口
shardingsphere-spi                  SPI加载工具类
sharding-transaction
    ├── sharding-transaction-core   接口ShardingTransactionManager，SPI加载      
    ├── sharding-transaction-2pc    实现类XAShardingTransactionManager
    └── sharding-transaction-base   实现类SeataATShardingTransactionManager

2. JDBC four objects

Everything from ShardingDataSourceFactory start, creating a ShardingDataSource slice data source. In addition ShardingDataSource (sliced data sources), as well as in MasterSlaveDataSourceFactory Sharding-Sphere (master from the data source), EncryptDataSourceFactory (desensitization data source).

public static DataSource createDataSource(
        final Map<String, DataSource> dataSourceMap,
        final ShardingRuleConfiguration shardingRuleConfig,
        final Properties props) throws SQLException {
    return new ShardingDataSource(dataSourceMap,
               new ShardingRule(shardingRuleConfig, dataSourceMap.keySet()), props);
}

Description: This paper mainly ShardingDataSource Sharding-Sphere Analysis entry point is how the four target JDBC DataSource, Connection, Statement, ResultSet for packaging.

2.1 DataSource

DataSource、Connection

Connection DataSource and are relatively simple and not too much processing logic, but dataSourceMap, shardingRule simple package.

ShardingDataSource holding data source and fragmentation rules may be acquired via ShardingConnection getConnection method.

private final ShardingRuntimeContext runtimeContext = new ShardingRuntimeContext(
                dataSourceMap, shardingRule, props, getDatabaseType());
@Override
public final ShardingConnection getConnection() {
    return new ShardingConnection(getDataSourceMap(), runtimeContext,
            TransactionTypeHolder.get());
}

ShardingDataSource function is very simple, not to say.

2.2 Connection

ShardingConnection can create Statement and PrepareStatement two operation modes:

@Override
public Statement createStatement(final int resultSetType,
        final int resultSetConcurrency, final int resultSetHoldability) {
    return new ShardingStatement(this, resultSetType,
            resultSetConcurrency, resultSetHoldability);
}

@Override
public PreparedStatement prepareStatement(final String sql, final int resultSetType,
        final int resultSetConcurrency, final int resultSetHoldability)
        throws SQLException {
    return new ShardingPreparedStatement(this, sql, resultSetType,
            resultSetConcurrency, resultSetHoldability);
}

Description: ShardingConnection mainly to create ShardingStatement and ShardingPreparedStatement two objects, the main execution logic in the Statement object. Of course ShardingConnection there are two important functions is to obtain a real database connection is a function of the transaction commits, the transaction is not in the scope of this discussion.

protected Connection createConnection(final String dataSourceName,
        final DataSource dataSource) throws SQLException {
    return isInShardingTransaction()
            ? shardingTransactionManager.getConnection(dataSourceName)
            : dataSource.getConnection();
}

Note: If there is a transaction, you need to get connected through a transaction manager, on matters beyond the scope of.

2.3 Statement

Statement relatively complex, because it is true JDBC actuators, all logic is encapsulated in the Statement.

Statement

Description: of Statement is divided into two cases ShardingStatement and ShardingPrepareStatement. In this paper, for example ShardingStatement analysis of the implementation process Sharding-Jdbc. The next section will focus on the analysis of the implementation process ShardingStatement.

2.4 ResultSet

ResultSet

Description: ShardingResultSet simply encapsulates the MergedResult.

private final MergedResult mergeResultSet;
@Override
public boolean next() throws SQLException {
    return mergeResultSet.next();
}

3. Sharding-Jdbc execution flow analysis

ShardingStatement execution timing diagram

Summary: ShardingStatement implementation process is as follows:

SimpleQueryShardingEngine (or PreparedQueryShardingEngine): complete SQL parsing, routing, rewrite, located sharding-jdbc-core projects. SimpleQueryShardingEngine direct routing functions entrusted to StatementRoutingEngine (or PreparedQueryShardingEngine), essentially encapsulates StatementRoutingEngine, SQLParseEngine, ShardingSQLRewriteEngine of.
StatementExecutor(或 PreparedStatementExecutor)：提供 SQL 执行的操作，位于 sharding-jdbc-core 工程中。本质是对 ShardingExecuteEngine 的封装。
StatementRoutingEngine：SQL 路由引擎，位于 sharding-core-route 工程中。路由引擎包装了 SQL 解析、路由、改写三点。SQL 路由分两步，先进行数据分片路由(ShardingRouter)，再进行主从路由(ShardingMasterSlaveRouter)。
SQLParseEngine：SQL 解析引擎，位于 shardingsphere-sql-parser 工程中。目前有 MySQL和 PostgreSQL 两种。
ShardingSQLRewriteEngine：SQL 改写引擎，位于 sharding-core-rewrite 工程中。
ShardingExecuteEngine：执行引擎，位于 sharding-core-execute 工程中。StatementExecutor 对
MergeEngine：结果合并引擎，位于 sharding-core-merge 工程中。

接下来一下会对 ShardingStatement 深入分析，之后会对 StatementRoutingEngine、SQLParseEngine、ShardingSQLRewriteEngine、ShardingExecuteEngine、MergeEngine 一个引擎进行分析。

4. sharding-jdbc-core 任务执行分析

ShardingStatement 内部有三个核心的类，一是 SimpleQueryShardingEngine 完成 SQL 解析、路由、改写；一是 StatementExecutor 进行 SQL 执行；最后调用 MergeEngine 对结果进行合并处理。

4.1 ShardingStatement

4.1.2 初始化

private final ShardingConnection connection;
private final StatementExecutor statementExecutor;

public ShardingStatement(final ShardingConnection connection) {
    this(connection, ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY,
            ResultSet.HOLD_CURSORS_OVER_COMMIT);
}

public ShardingStatement(final ShardingConnection connection, final int resultSetType,
        final int resultSetConcurrency, final int resultSetHoldability) {
    super(Statement.class);
    this.connection = connection;
    statementExecutor = new StatementExecutor(resultSetType, resultSetConcurrency,
            resultSetHoldability, connection);
}

说明： ShardingStatement 内部执行 SQL 委托给了 statementExecutor。关于 ResultSet.CONCUR_READ_ONLY 等参考这里。

4.1.2 执行

（1）executeQuery 执行过程

@Override
public ResultSet executeQuery(final String sql) throws SQLException {
    ResultSet result;
    try {
        clearPrevious();
        // 1. SQL 解析、路由、改写，最终生成 SQLRouteResult
        shard(sql);
        // 2. 生成执行计划 SQLRouteResult -> StatementExecuteUnit
        initStatementExecutor();
        // 3. statementExecutor.executeQuery() 执行任务
        MergeEngine mergeEngine = MergeEngineFactory.newInstance(
                connection.getRuntimeContext().getDatabaseType(),
                connection.getRuntimeContext().getRule(), sqlRouteResult,
                connection.getRuntimeContext().getMetaData().getRelationMetas(),
                statementExecutor.executeQuery());
        // 4. 结果合并
        result = getResultSet(mergeEngine);
    } finally {
        currentResultSet = null;
    }
    currentResultSet = result;
    return result;
}

（2）SQL 路由（包括 SQL 解析、路由、改写）

private SQLRouteResult sqlRouteResult;
private void shard(final String sql) {
    ShardingRuntimeContext runtimeContext = connection.getRuntimeContext();
    SimpleQueryShardingEngine shardingEngine = new SimpleQueryShardingEngine(
            runtimeContext.getRule(), runtimeContext.getProps(),
            runtimeContext.getMetaData(), runtimeContext.getParseEngine());
    sqlRouteResult = shardingEngine.shard(sql, Collections.emptyList());
}

说明： SimpleQueryShardingEngine 进行 SQL 路由（包括 SQL 解析、路由、改写），生成 SQLRouteResult。之后会有一节专门分析 SQL 路由过程。

当 ShardingStatement 完成 SQL 的路由，生成 SQLRouteResult 后，剩下的执行任务就全部交给 StatementExecutor 完成。

4.2 StatementExecutor

StatementExecutor 内部封装了 SQL 任务的执行过程，包括：SqlExecutePrepareTemplate 类生成执行计划 StatementExecuteUnit，以及 SQLExecuteTemplate 用于执行 StatementExecuteUnit。

4.2.1 类结构

StatementExecutor 类图

4.2.2 重要属性

AbstractStatementExecutor 类中重要的属性：

// SQLExecutePrepareTemplate用于生成执行计划StatementExecuteUnit
private final SQLExecutePrepareTemplate sqlExecutePrepareTemplate;
// 保存生成的执行计划StatementExecuteUnit
private final Collection<ShardingExecuteGroup<StatementExecuteUnit>> executeGroups =
            new LinkedList<>();

// SQLExecuteTemplate用于执行StatementExecuteUnit
private final SQLExecuteTemplate sqlExecuteTemplate;
// 保存查询结果
private final List<ResultSet> resultSets = new CopyOnWriteArrayList<>();

4.2.3 生成执行计划

// 执行前清理状态
private void clearPrevious() throws SQLException {
    statementExecutor.clear();
}
// 执行时初始化
private void initStatementExecutor() throws SQLException {
    statementExecutor.init(sqlRouteResult);
    replayMethodForStatements();
}

说明： StatementExecutor 是有状态的，每次执行前都要调用 statementExecutor.clear() 清理上一次执行的状态，并调用 statementExecutor.init() 重新初始化。下面我们看一下 init 主要做了些什么事。

statementExecutor.init() 初始化主要是生成执行计划 StatementExecuteUnit。

public void init(final SQLRouteResult routeResult) throws SQLException {
    setSqlStatementContext(routeResult.getSqlStatementContext());
    getExecuteGroups().addAll(obtainExecuteGroups(routeResult.getRouteUnits()));
    cacheStatements();
}

private Collection<ShardingExecuteGroup<StatementExecuteUnit>> obtainExecuteGroups(
        final Collection<RouteUnit> routeUnits) throws SQLException {
    return getSqlExecutePrepareTemplate().getExecuteUnitGroups(
            routeUnits, new SQLExecutePrepareCallback() {
                // 获取连接
                @Override
                public List<Connection> getConnections(
                        final ConnectionMode connectionMode,
                        final String dataSourceName, final int connectionSize)
                        throws SQLException {
                    return StatementExecutor.super.getConnection().getConnections(
                            connectionMode, dataSourceName, connectionSize);
                }

                // 生成执行计划RouteUnit -> StatementExecuteUnit
                @Override
                public StatementExecuteUnit createStatementExecuteUnit(
                        final Connection connection, final RouteUnit routeUnit,
                        final ConnectionMode connectionMode) throws SQLException {
                    return new StatementExecuteUnit(
                            routeUnit, connection.createStatement(
                            getResultSetType(), getResultSetConcurrency(),
                            getResultSetHoldability()), connectionMode);
                }
            });
}

说明： SqlExecutePrepareTemplate 是 sharding-core-execute 工程中提供的一个工具类，专门用于生成执行计划，将 RouteUnit 转化为 StatementExecuteUnit。同时还提供了另一个工具类 SQLExecuteTemplate 用于执行 StatementExecuteUnit，在任务执行时我们会看到这个类。

4.2.4 任务执行

public List<QueryResult> executeQuery() throws SQLException {
    final boolean isExceptionThrown = ExecutorExceptionHandler.isExceptionThrown();
    SQLExecuteCallback<QueryResult> executeCallback = 
        new SQLExecuteCallback<QueryResult>(getDatabaseType(), isExceptionThrown) {
        @Override
        protected QueryResult executeSQL(final String sql, final Statement statement,
                final ConnectionMode connectionMode) throws SQLException {
            return getQueryResult(sql, statement, connectionMode);
        }
    };
    // 执行StatementExecuteUnit
    return executeCallback(executeCallback);
}

// sqlExecuteTemplate 执行 executeGroups(即StatementExecuteUnit)
protected final <T> List<T> executeCallback(
        final SQLExecuteCallback<T> executeCallback) throws SQLException {
    // 执行所有的任务 StatementExecuteUnit
    List<T> result = sqlExecuteTemplate.executeGroup(
            (Collection) executeGroups, executeCallback);
    refreshMetaDataIfNeeded(connection.getRuntimeContext(), sqlStatementContext);
    return result;
}

说明： SqlExecuteTemplate 执行 StatementExecuteUnit 会回调 SQLExecuteCallback#executeSQL 方法，最终调用 getQueryResult 方法。

private QueryResult getQueryResult(final String sql, final Statement statement,
        final ConnectionMode connectionMode) throws SQLException {
    ResultSet resultSet = statement.executeQuery(sql);
    getResultSets().add(resultSet);
    return ConnectionMode.MEMORY_STRICTLY == connectionMode
            ? new StreamQueryResult(resultSet)
            : new MemoryQueryResult(resultSet);
}

说明： ConnectionMode 有两种模式：内存限制(MEMORY_STRICTLY)和连接限制(CONNECTION_STRICTLY)，本质是一种资源隔离，保护服务器资源不被消耗殆尽。

如果一个连接执行多个 StatementExecuteUnit 则为内存限制(MEMORY_STRICTLY)，采用流式处理，即 StreamQueryResult ，反之则为连接限制(CONNECTION_STRICTLY)，此时会将所有从 MySQL 服务器返回的数据都加载到内存中。特别是在 Sharding-Proxy 中特别有用，避免将代理服务器撑爆，见 Apache Sharding-Proxy 核心原理。

每天用心记录一点点。内容也许不重要，但习惯很重要！

sharding-jdbc-core source code analysis