SpringBoot master-slave data source switch

SpringBoot master-slave data source switch

1. Principle

With the help of spring's [org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource] abstract class implementation, the routing of the data source is carried out, and the routing is selected through Aop.

2. Configure the master-slave data source

# dev server
# 多数据源时，主数据源为 master
spring.datasource.master.jdbc-url=jdbc:mysql://localhost:3306/epoint?characterEncoding=utf8&allowMultiQueries=true&useSSL=false&autoReconnect=true&failOverReadOnly=false
spring.datasource.master.username=test
spring.datasource.master.password=test

# dev server
# 多数据源时，从数据源为 slave
spring.datasource.slave.jdbc-url=jdbc:mysql://localhost:3306/epoint2?characterEncoding=utf8&allowMultiQueries=true&useSSL=false&autoReconnect=true&failOverReadOnly=false
spring.datasource.slave.username=test
spring.datasource.slave.password=test

Start error handling

spring boot ：error querying database. Cause: java.lang.IllegalArgumentException: dataSource or dataSourceClassName or jdbcUrl is required
配置多个数据源启动报错，error querying database. Cause: java.lang.IllegalArgumentException: dataSource or dataSourceClassName or jdbcUrl is required，
主要原因是在1.0 配置数据源的过程中主要是写成：spring.datasource.url 和spring.datasource.driverClassName。
而在2.0升级之后需要变更成：spring.datasource.jdbc-url和spring.datasource.driver-class-name即可解决！
更改配置：spring.datasource.master.url  ->  spring.datasource.master.jdbc-url

3. Get the data source type of the current thread

/**
 * describe:定义HandleDataSource类来获取当前线程的数据源类型
 * current user Maochao.zhu
 * current system 2020/9/15
 */
public class HandleDataSource {
    public static final ThreadLocal<String> holder = new ThreadLocal<String>();

    /**
     * 绑定当前线程数据源
     *
     * @param datasource
     */
    public static void putDataSource(String datasource) {
        holder.set(datasource);
    }

    /**
     * 获取当前线程的数据源
     *
     * @return
     */
    public static String getDataSource() {
        return holder.get();
    }

}

3.1 Thread conflict problem

Note : Because the newly added database thread processing class conflicts with the original multi-threaded processing class, it will cause the existing program to "catch" or "crash", so remove the original multi-threaded configuration of scheduled tasks.

/**
 * describe:配置多线程定时器
 * current user Maochao.zhu
 * current system 2020/1/20
 */
@Configuration
@EnableScheduling
public class ScheduleConfig implements SchedulingConfigurer {
    @Override
    public void configureTasks(ScheduledTaskRegistrar taskRegistrar) {
        Method[] methods = BatchProperties.Job.class.getMethods();
        int defaultPoolSize = 3;
        int corePoolSize = 0;
        if (methods != null && methods.length > 0) {
            for (Method method : methods) {
                Scheduled annotation = method.getAnnotation(Scheduled.class);
                if (annotation != null) {
                    corePoolSize++;
                }
            }
            if (defaultPoolSize > corePoolSize)
                corePoolSize = defaultPoolSize;
        }
        taskRegistrar.setScheduler(Executors.newScheduledThreadPool(corePoolSize));
    }
}

4. Define the implementation class of the routing data source

/**
 * describe:定义路由数据源的实现类MyAbstractRoutingDataSource
 * current user Maochao.zhu
 * current system 2020/9/15
 */
public class MyAbstractRoutingDataSource extends AbstractRoutingDataSource {
    private final Logger log = LoggerFactory.getLogger(this.getClass());
    @Override
    protected Object determineCurrentLookupKey() {
        log.info("###请求的数据源：{}",HandleDataSource.getDataSource());
        return HandleDataSource.getDataSource();//获取对应的数据源
    }
}

5. Configure data source and routing

/**
 * describe:配置数据源数据源和路由配置
 * current user Maochao.zhu
 * current system 2020/9/15
 */
@Configuration
public class DataSourceConfig {
    //主数据源
    @Bean()
    @ConfigurationProperties(prefix = "spring.datasource.master")
    public DataSource MasterDataSource() {
        return DataSourceBuilder.create().build();
    }
    //从数据源
    @Bean()
    @ConfigurationProperties(prefix = "spring.datasource.slave")
    public DataSource SlaveDataSource() {
        return DataSourceBuilder.create().build();
    }
    /**
     * 设置数据源路由，通过该类中的determineCurrentLookupKey决定使用哪个数据源
     */
    @Bean
    public AbstractRoutingDataSource routingDataSource() {
        MyAbstractRoutingDataSource proxy = new MyAbstractRoutingDataSource();
        Map<Object, Object> targetDataSources = new HashMap<>(2);//存放对于数据源的映射
        targetDataSources.put("master", MasterDataSource());
        targetDataSources.put("slave", SlaveDataSource());
        proxy.setDefaultTargetDataSource(MasterDataSource());
        proxy.setTargetDataSources(targetDataSources);
        return proxy;
    }


    @Bean(name = "SqlSessionFactory")
    @Primary
    public SqlSessionFactory MasterSqlSessionFactory(DataSource routingDataSource) throws Exception {
        SqlSessionFactoryBean bean = new SqlSessionFactoryBean();
        bean.setDataSource(routingDataSource);//DataSource使用路由数据源

        ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
        try {
            bean.setMapperLocations(resolver.getResources("classpath*:mapper/**/*.xml"));
            bean.setConfigLocation(resolver.getResource("classpath:mybatis-config.xml"));
            return bean.getObject();
        } catch (Exception e) {
            e.printStackTrace();
            throw new RuntimeException(e);
        }

    }

    @Bean(name = "TransactionManager")
    @Primary
    public DataSourceTransactionManager testTransactionManager(DataSource routingDataSource) {
        return new DataSourceTransactionManager(routingDataSource);
    }

    @Bean(name = "SqlSessionTemplate")
    @Primary
    public SqlSessionTemplate MasterSqlSessionTemplate(SqlSessionFactory sqlSessionFactory) throws Exception {
        return new SqlSessionTemplate(sqlSessionFactory);
    }
}

5.1 Startup permission shiro problem

Note : After configuring the routing settings, the original apache.shiro permissions cannot be read. After investigation, it is determined that the reason is that the configuration routing settings are used. The mybaits property originally configured in application.properties will not work, so it needs to be added. A configuration file (mybatis-config.xml), from which configuration information is read, and annotation support is added to the ShiroConfig configuration class

/**
 * 开启Shiro的注解(如@RequiresRoles,@RequiresPermissions),需借助SpringAOP扫描使用Shiro注解的类,并在必要时进行安全逻辑验证
 * 配置以下两个bean(DefaultAdvisorAutoProxyCreator和AuthorizationAttributeSourceAdvisor)即可实现此功能
 * @return
 */
@Bean
@ConditionalOnMissingBean
public DefaultAdvisorAutoProxyCreator defaultAdvisorAutoProxyCreator(){
	DefaultAdvisorAutoProxyCreator advisorAutoProxyCreator = new DefaultAdvisorAutoProxyCreator();
	advisorAutoProxyCreator.setProxyTargetClass(true);
	return advisorAutoProxyCreator;
}

/**
 * 开启shiro aop注解支持.
 * 使用代理方式;所以需要开启代码支持;
 * @param securityManager
 * @return
 * */
@Bean
public AuthorizationAttributeSourceAdvisor authorizationAttributeSourceAdvisor(SecurityManager securityManager){
	AuthorizationAttributeSourceAdvisor authorizationAttributeSourceAdvisor = new AuthorizationAttributeSourceAdvisor();
	authorizationAttributeSourceAdvisor.setSecurityManager(securityManager);
	return authorizationAttributeSourceAdvisor;
}

6. Load mybatis configuration

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration PUBLIC "-//mybatis.org//DTD Config 3.0//EN" "http://mybatis.org/dtd/mybatis-3-config.dtd">
<configuration>
    <settings>
        <setting name="cacheEnabled" value="true" />
        <setting name="lazyLoadingEnabled" value="true" />
        <setting name="multipleResultSetsEnabled" value="true" />
        <setting name="useColumnLabel" value="true" />
        <setting name="useGeneratedKeys" value="false" />
        <setting name="defaultExecutorType" value="SIMPLE" />
        <setting name="mapUnderscoreToCamelCase" value="true" />
        <setting name="logImpl" value="STDOUT_LOGGING" />
    </settings>
</configuration>

7. Create a new annotation to annotate the data source used

/**
 * describe:DataSource注解来注释Mapper接口所要使用的数据源
 * current user Maochao.zhu
 * current system 2020/9/15
 */
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface DataSource {
    String value();//设置数据源类型
}

8. Configure Aops

/**
 * describe:配置Aop切换路由选择
 * current user Maochao.zhu
 * current system 2020/9/15
 */
@Aspect
@Component
public class DataSourceAspect {
    public Logger logger = LoggerFactory.getLogger(this.getClass());
    /**
     * 在dao层方法获取datasource对象之前，在切面中指定当前线程数据源
     */
    @Pointcut("execution(* com.cn.zx.dao..*.*(..))")//切点为所有的mapper接口
    public void pointcut() {

    }

    @Before("pointcut()")
    public void before(JoinPoint point) {
        System.out.println("before");
        Object target = point.getTarget();
        String method = point.getSignature().getName();
        Class<?>[] classz = target.getClass().getInterfaces();// 获取目标类的接口， 所以@DataSource需要写在接口上
        Class<?>[] parameterTypes = ((MethodSignature) point.getSignature()).getMethod().getParameterTypes();
        try {
            Method m = classz[0].getMethod(method, parameterTypes);
            if (m != null && m.isAnnotationPresent(DataSource.class)) {
                DataSource data = m.getAnnotation(DataSource.class);
                System.out.println("####################用户选择数据库库类型：" + data.value());
                HandleDataSource.putDataSource(data.value());// 数据源放到当前线程中
            }
            logger.info("执行接口方法:{}.{}", point.getSignature().getDeclaringTypeName(), point.getSignature().getName());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

9. Set the data source type

Use the annotation DataSource("master/slave") on the corresponding mapper method to set the data source type

/**
 * 调用主数据源 master
 * @param user
 * @return
 */
@DataSource("master")
Integer insertUser(User user);

/**
 * 调用从数据源 slave
 * @param user
 * @return
 */
@DataSource("slave")
List<User> getUserList(User user);

10. Transaction invalidation problem

After adding the master-slave database, the transaction becomes invalid. The reason is still being found. The initial positioning is: Execution order of transactions and aspects @EnableTransactionManagement(order = 2) @Order(2)

solve:

SpringbootApplication启动类增加注解开启事务注解,数据库表引擎更改为innodb，如果是myisam，事务是不起作用的
@EnableTransactionManagement

There are two ways to manage transactions: The first one: programmatic transaction management, which needs to cancel the automatic commit of the database, and you need to write transaction code yourself. The second: declarative transaction management mode, spring uses the spring AOP feature to write annotations.

10.1 Features of the @Transactional annotation

1.service类标签(一般不建议在接口上)上添加@Transactional，可以将整个类纳入spring事务管理，在每个业务方法执行时都会开启一个事务，不过这些事务采用相同的管理方式。并且当在某个service实现类中某个方法调用了另一个这个实现类中的方法，则两个方法都必须声明事务，才能被当成一个事务进行管理
2.@Transactional 注解只能应用到 public 可见度的方法上。 如果应用在protected、private或者 package可见度的方法上，也不会报错，不过事务设置不会起作用。
3.默认情况下，spring会对unchecked异常进行事务回滚；如果是checked异常则不回滚。 

10.2 checked exceptions

So what is a checked exception and what is an unchecked exception. In java, exceptions derived from Error or RuntimeException (such as null pointer, 1/0) are called unchecked exceptions, and other exceptions inherited from java.lang.Exception are collectively called Checked Exceptions, such as IOException, TimeoutException, etc. are more popular: the null pointer and other exceptions in the code you write will be rolled back, the file is read and written, and there is a network problem, spring cannot roll back.

10.3 Read-only transactions

@Transactional(propagation=Propagation.NOT_SUPPORTED,readOnly=true) 
只读标志只在事务启动时应用，否则即使配置也会被忽略。 
启动事务会增加线程开销，数据库因共享读取而锁定(具体跟数据库类型和事务隔离级别有关)。通常情况下，仅是读取数据时，不必设置只读事务而增加额外的系统开销。

10.4 Transaction propagation mode

Propagation枚举了多种事务传播模式，部分列举如下：
1. REQUIRED(默认模式)：业务方法需要在一个容器里运行。如果方法运行时，已经处在一个事务中，那么加入到这个事务，否则自己新建一个新的事务。
2. NOT_SUPPORTED：声明方法不需要事务。如果方法没有关联到一个事务，容器不会为他开启事务，如果方法在一个事务中被调用，该事务会被挂起，调用结束后，原先的事务会恢复执行。
3. REQUIRESNEW：不管是否存在事务，该方法总汇为自己发起一个新的事务。如果方法已经运行在一个事务中，则原有事务挂起，新的事务被创建。
4. MANDATORY：该方法只能在一个已经存在的事务中执行，业务方法不能发起自己的事务。如果在没有事务的环境下被调用，容器抛出例外。
5. SUPPORTS：该方法在某个事务范围内被调用，则方法成为该事务的一部分。如果方法在该事务范围外被调用，该方法就在没有事务的环境下执行。
6. NEVER：该方法绝对不能在事务范围内执行。如果在就抛例外。只有该方法没有关联到任何事务，才正常执行。
7. NESTED：如果一个活动的事务存在，则运行在一个嵌套的事务中。如果没有活动事务，则按REQUIRED属性执行。它使用了一个单独的事务，这个事务拥有多个可以回滚的保存点。内部事务的回滚不会对外部事务造成影响。它只对DataSourceTransactionManager事务管理器起效。

10.5 Solve the problem that Transactional annotation does not roll back

1. 检查你方法是不是public的。
2. 你的异常类型是不是unchecked异常。空指针异常是unchecked异常,如果我想check异常也想回滚怎么办，注解上面写明异常类型即可。
@Transactional(rollbackFor={Exception.class.RuntimeException.class})
类似的还有norollbackFor，自定义不回滚的异常。如果已经在service中进行了try catch 操作,由于已经被抓获异常,事务也不会回滚
3. 数据库引擎要支持事务，如果是mysql，注意表要使用支持事务的引擎，比如innodb，如果是myisam，事务是不起作用的。
4. 是否开启了对注解的解析
	4.1 SpringMVC中开启：
		<tx:annotation-driven transaction-manager="transactionManager" proxy-target-class="true"/>
	4.2 pringboot中开启：
		启动类增加注解：@EnableTransactionManagement
		类方法中增加注解：@Transactional
5. spring是否扫描到你这个包，如下是扫描到org.test下面的包
	<context:component-scan base-package="org.test" ></context:component-scan>

10.6 Database Engine Settings

mysql database engine innodb settings

10.6.1 View supported engines

show engines;

10.6.2 Viewing the default engine

show variables like 'default_storage_engine';

10.6.3 Modify the default engine

修改mysql 默认的数据库引擎
在配置文件my.ini中的 [mysqld] 下面加入default-storage-engine=INNODB
如果启动不起来,找到 skip-innodb 项，将其改为 #skip-innodb（要不然知识修改了 InnoDB 服务起不来）
重启Mysql服务器,设置生效。

10.6.4 Modify the data table engine

If the database name is: epoint, modify the database table engine from MyISAM -> InnoDB

10.6.5 Querying the Status of All Tables

SHOW TABLE STATUS FROM epoint;

10.6.6 Query and modify the SQL of the table engine

SELECT GROUP_CONCAT(CONCAT( 'ALTER TABLE ' ,TABLE_NAME ,' ENGINE=InnoDB; ') SEPARATOR '' ) 
FROM information_schema.TABLES AS t 
WHERE TABLE_SCHEMA = 'epoint' AND TABLE_TYPE = 'BASE TABLE';

10.6.7 Get SQL Execution

ALTER TABLE branch ENGINE=InnoDB; 

10.6.8 Querying the Status of All Tables

SHOW TABLE STATUS FROM epoint;

The engine that modifies all database tables ends with InnoDB

11. Master-slave database synchronization problem

Master-slave database data synchronization problem positioning: 1. Program synchronization. 2. To synchronize data in database operations, because read-write separation is required, database permissions must be set. The master database can read and write, and the slave database can only read, so this method does not work, and the master-slave data synchronization can only be set from the database side.

1. View the installation path of mysql

Check the installation path of mysql through the mysql command:

select @@basedir as basePath from dual;
SELECT @@basedir;

Get path: C:\Program Files\MySQL\MySQL Server 5.7\

2. Configure the main database

Find [mysqld] in the my.ini file and add the following configuration (how many databases need to be synchronized can be written in, master-slave synchronization will find the corresponding cluster library to synchronize data according to the library name)

server-id=1#主库和从库需要不一致
log-bin=mysql-bin
binlog-do-db=mstest#同步的数据库
binlog-ignore-db=mysql#不需要同步的数据库

2.1 Restart the MySql service and query the status of the main database

mysql> SHOW VARIABLES LIKE 'server_id';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| server_id     | 1     |
+---------------+-------+
1 row in set

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000007 |     5138 | epoint       |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set

2.2 Create an account for the slave library and assign all permissions

grant all on *.* to 'user'@'%' identified by 'user';

3. Configure the slave database

Find [mysqld] in the my.ini file and add the following configuration (how many databases need to be synchronized can be written in, master-slave synchronization will find the corresponding cluster library to synchronize data according to the library name)

server-id=2#和主库不一致
replicate-do-db=test#需要同步的库1
replicate-ignore-db=mysql#不需要同步的库

3.1 Start the copy function from the library

STOP SLAVE; #停止从复制功能的命令
change master to master_host='127.0.0.1',master_port=3306,master_user='slave',master_password='123',master_log_file='mysql-bin.000004',master_log_pos=717;

Description: Correspondingly changed to your own configuration, master_host: the ip of the main library, master_port: the port of the main library, master_user: the account name created by the main library for the cluster library, master_password: the account password is about master_log_file and Position ('mysql-bin .000005' 98) is obtained by "show master status" in the master library configuration;

START SLAVE; #启动从复制功能
RESET SLAVE; #重置从复制功能的配置，会清除 master.info 和 relay-log.info 两个文件
show slave status;   (没有分号)，查看

The Slave_IO_Running and Slave_SQL_Running properties are turned on, indicating that they are turned on

Slave_IO_Running: Yes
Slave_SQL_Running: Yes

Configure MySQL master-slave database end

12. Reference blog information

> 主从Mysql数据库同步：https://blog.csdn.net/fengrenyuandefz/article/details/89420201
> 一台电脑装两个MySQL：https://blog.csdn.net/weixin_41953055/article/details/79820221
> server_uuid重复：https://blog.csdn.net/sunbocong/article/details/81634296
> mysql主从同步 binlog-do-db replicate-do-db： https://blog.csdn.net/z69183787/article/details/70183284
> mysql主从数据库同步：https://blog.csdn.net/fengrenyuandefz/article/details/89420201

{{o.name}}

{{m.name}}