Mybatis dynamic data source and its principle

I. Introduction

        The author's recent platform project requires a function. The database is dynamic and the SQL is also dynamic, so the data source needs to be injected dynamically and the database can be switched during operation. The author shares here the method and the principle of Mybatis doing this.

2. Analysis

        Next, let’s analyze the requirements, mainly focusing on two points:

        1. Dynamic data source

        Since it is dynamic, it must be placed in the configuration center. The problem is to initialize these databases. A common approach is to put them in multiple paths. Each path has one database and one configuration class. In this way, the scanned classes always belong to a certain database and are clearly decoupled. However, adding databases like this It needs to be published, and every time a library is added, a bunch of paths must be added, which is suitable under normal circumstances.

        However, most databases on the author's platform perform the same operations. Subsequent changes and releases brought about by this method are not necessary. Therefore, the data source must be placed in the collection, and the database can be added by simply changing the configuration.

        2. If the operation switching
       is one path and one library, there is no need to switch. If the data source exists in the collection, it definitely needs to be switched. Normal switching is based on AOP or interceptor. Here Mybatis provides some tools to achieve faster implementation based on aop.

3. Realization

        Try the simple one first, and make sure you have a guarantee, so that you don’t end up with too many traps in the second one.

1. Path-Library

        Configuration class. When creating the data source here, it is done by the framework of the author's company. Normal creation requires specifying the link, account, and password in yml.

@EnableDalMybatis(encryptParameters = false)
@Configuration
@MapperScan(basePackages = {"com.mapper"},
        sqlSessionFactoryRef = "sqlSessionFactory")
public class OneDBConfig {

    @Bean
    public DataSource dataSource() throws Exception {
        return factory.getOrCreateDataSource(DB_KEY);
    }

    @Bean(name = "transactionManager")
    public DataSourceTransactionManager transactionManager(DataSource dataSource) {
        return new DataSourceTransactionManager(dataSource);
    }

    @Bean(name = "sqlSessionFactory")
    public SqlSessionFactory sqlSessionFactory(DataSource dataSource) throws Exception {
        MybatisSqlSessionFactoryBean sqlSessionFactoryBean = new MybatisSqlSessionFactoryBean();
        sqlSessionFactoryBean.setDataSource(dataSource);
        PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
        sqlSessionFactoryBean
                .setMapperLocations(resolver.getResources(
                        "classpath*:/com/*.xml"));
        sqlSessionFactoryBean.setConfigLocation(resolver.getResource("classpath:/mybatis/mybatis-config.xml"));
        return sqlSessionFactoryBean.getObject();
    }

}

        There are as many configuration classes as there are libraries. There is a pitfall here. The SqlSessionFactory corresponding to each configuration class must have an alias and cannot be the same.

        SqlSessionFactory is used to create the factory class of SqlSession, which is the main object used to execute SQL statements in MyBatis. The data source is stuffed inside this object, so it doesn't matter if the data source and data source manager are not named, but they are different in the outer layer.

        Obviously xml and mapper also need to have multiple corresponding directories.

        When using it, you use mappers under different paths according to the db.

DbEnum dbEnum = DbEnum.getExecuteEnumByValue(db);
        switch (dbEnum) {
            case ONE:
                //
            case TWO:
                //
            default:
                return null;
        }

2. Dynamic multiple data sources 

        Inherits mybatis's AbstractRoutingDataSource. There is a method in this class, which is used by mybatis every time it creates a link. It finds the corresponding data source according to the key.

public class DynamicDataSource extends AbstractRoutingDataSource {
    @Override
    protected Object determineCurrentLookupKey() {
        return DataSourceThreadLocal.getDB();
    }
}

        Use ThreadLocal to store the data sources you need to use so that it will not affect other threads' data operations. 

public class DataSourceThreadLocal {
    public static final String DEFAULT_DB = "default";
    private static final ThreadLocal<String> nowDb = new ThreadLocal<>();
    public static void setDB(String dbType) {
        nowDb.set(dbType);
    }

    public static String getDB() {
        // 如果当前线程没有设置切换数据库，就使用默认数据库
        if (nowDb == null || StringUtilsExt.isBlank(nowDb.get())) {
            return DEFAULT_DB;
        }
        return (nowDb.get());
    }

    public static void clearDB() {
        nowDb.remove();
    }
}

        Mybatis provides a DynamicDataSource. The targetDataSources inside is a hashmap, which can store the db and data initialization datasource. The database in the configuration center mainly contains the db key. It is deserialized into an object in the program and json is placed in the configuration center. , map, and list are all fine.

@EnableDalMybatis(encryptParameters = false)
@Configuration
@MapperScan(
        basePackages = {
            "com.mapper"},
        sqlSessionFactoryRef = "sqlSessionFactory")
public class CommonQueryDBConfig {

    private static final LoggerService LOG = LoggerServiceFactory.getLoggerService(CommonQueryDBConfig.class);

    private static final String LOG_TITLE = "CommonQueryDBConfig";

    @Primary
    @Bean(name = "dynamicDataSource")
    public DataSource dynamicDataSource() throws Exception {
        DynamicDataSource dynamicDataSource = new DynamicDataSource();
        LOG.info(LOG_TITLE, "init data source");
        // 数据源可以存放在map里面
        Map<Object, Object> dbMap = new HashMap<>();
        String s = ConfigurationFunc.getString(QConfigConstants.CREATE_DATA_CONFIG_PARAMETER);
        List<DynamicDataSourceConfigBo> dataSourceConfigBos =
                JSONUtil.parse(s, new TypeReference<List<DynamicDataSourceConfigBo>>() {});
        LOG.info(LOG_TITLE, "dataSourceConfigBos:{}", JSONUtil.toJsonNoException(dataSourceConfigBos));
        for (DynamicDataSourceConfigBo ds : dataSourceConfigBos) {
            DataSource now = factory.createDataSource(ds.getDbCreateKey());
            dbMap.put(ds.getDbCreateKey(), now);
        }
        // 默认数据源        
        dynamicDataSource.setDefaultTargetDataSource(dsMap.get("corpcodescannerdb_dalcluster"));
        dynamicDataSource.setTargetDataSources(dsMap);
        LOG.info(LOG_TITLE, "init data source success:{}", dsMap.size());
        return dynamicDataSource;
    }

    @Bean(name = "dynamicTransactionManager")
    public DataSourceTransactionManager dynamicTransactionManager() throws Exception {
        return new DataSourceTransactionManager(dynamicDataSource());
    }

    @Bean(name = "sqlSessionFactory")
    public SqlSessionFactory sqlSessionFactory(DataSource dynamicDataSource) throws Exception {
        MybatisSqlSessionFactoryBean sqlSessionFactoryBean = new MybatisSqlSessionFactoryBean();
        sqlSessionFactoryBean.setDataSource(dynamicDataSource);
        PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
        sqlSessionFactoryBean
                .setMapperLocations(resolver.getResources(
                        "classpath*:/com.mapper/*.xml"));
        sqlSessionFactoryBean.setConfigLocation(resolver.getResource("classpath:/mybatis/mybatis-config.xml"));
        SqlSessionFactory res = sqlSessionFactoryBean.getObject();
        return res;
    }

}

        To use it, just set threadlocal. Remember to clear it after use.

DataSourceThreadLocal.setDB(db);
        LOG.info(LOG_TITLE, "change datasource:{}", db);
        // mapper做什么
        LOG.info(LOG_TITLE, "datasource res:{}", JSONUtil.toJsonNoException(res));
        DataSourceThreadLocal.clearDB();
        return res;

4. Principle

1. Design        

        Before analyzing the dynamic data source principle of mybatis, first think about how the implementation needs to be designed. The designs are all very different. Only with the design concept can you know where to look at the code of mybatis.

        If you do it yourself, you have to consider where the database information is placed. When it comes to using it, you actually need to find the link address, account number, and password of the database, then establish a network link, and perform bit stream transmission through the link. So it can be divided into the following steps:

        1. Initialize the database. Store the link address, account and password of the database in the local cache. It is convenient and safe to put it in the hashmap, because there is no change after initialization, only numbers can be retrieved, and the key-value pairs of the database name and information are also easy to get.

        2. When using it, you need to pass the database name and get the database information from the cache.

        3. Take the database information and establish a link

         It seems that the main point is when looking for database information and establishing a connection, then you can go here to find the mybatis code.

2. Mybatis implementation

        First, we need to enter his query logic

        Here you can see the link to get it

        This local cache stores information about each database and the current default database.

         The link has not been established and is ready to be opened.

        Establish a connection in the tool class

        All the database information has been obtained here. The next step is to choose which one to initialize the connection.

         This is where lookupKey is used as the key and the database information is obtained from the map. If it is not set, the default data source will be used. From this, it can be seen that there is no need to give a default value in the ThreadLocal class. The author wrote it because it is easier to understand, and colleagues who follow may not necessarily understand these principles.

        There is another pitfall here. The reason why mybatis determines whether the current link is null is because the link is reused in the same transaction. At this time, switching the database will not take effect.

        Usually every time Mybatis is queried, the openConnection method will be executed to obtain a new database connection. This ensures that each query is executed in an independent transaction, avoiding data confusion and concurrency issues.

public Connection getConnection() throws SQLException {
if (this.connection == null) {
openConnection();
}
return this.connection;
}

5. Summary

        That’s it for the methods and principles. There are still some pitfalls that I have forgotten. Students who have questions can share them in the comment area.