Database series-read and write separation based on MySQL master-slave synchronization

Premise :

Build a master and multiple slave database cluster Refer to the previous article    https://blog.csdn.net/Coder_Boy_/article/details/110950347

Github code address of this article: https://github.com/cheriduk/spring-boot-integration-template

 

The core of this article: read and write separation at the code level

The code environment is springboot+mybatis+druib connection pool. If you want to separate read and write, you need to configure multiple data sources. When performing a write operation, select the data source for writing, and select the data source for reading during a read operation. There are two key points:

1. How to switch data sources
2. How to choose the right data source according to different methods

1) How to switch data source

Usually springboot uses its default configuration. You only need to define the connection properties in the configuration file, but now we need to configure it ourselves. Spring supports multiple data sources, and multiple datasources are placed in one HashMapTargetDataSource. , Get the key through dertermineCurrentLookupKey to determine which data source to use. Therefore, our goal is very clear. Create multiple datasources and put them in the TargetDataSource, and at the same time override the dertermineCurrentLookupKey method to decide which key to use.

2) How to choose a data source

Transactions are generally annotated in the Service layer, so when you start the service method call, you must determine the data source. Is there any general method to perform operations before starting to execute a method? I believe you have already thought of that is the aspect. There are two ways to cut:

• Annotation, define a read-only annotation, the method marked by the data uses the read library
• Method name, write cut point according to method name, for example, getXXX uses read library, setXXX uses write library

3), code writing

Project directory structure:

a. Write a configuration file and configure two data source information

Only required information, other default settings

application.properties

mysql.datasource.config-location=classpath:/mybatis-config.xml
mysql.datasource.mapper-locations=classpath:/mapper/*.xml
mysql.datasource.num=2

mysql.datasource.read1.driverClass=com.mysql.jdbc.Driver
mysql.datasource.read1.password=123456
mysql.datasource.read1.url=jdbc:mysql://127.0.0.1:3307/test?serverTimezone=Asia/Shanghai
mysql.datasource.read1.username=root

mysql.datasource.read2.driverClass=com.mysql.jdbc.Driver
mysql.datasource.read2.password=123456
mysql.datasource.read2.url=jdbc:mysql://127.0.0.1:3308/test?serverTimezone=Asia/Shanghai
mysql.datasource.read2.username=root


mysql.datasource.write.driverClass=com.mysql.jdbc.Driver
mysql.datasource.write.password=123456
mysql.datasource.write.url=jdbc:mysql://127.0.0.1:3306/test?serverTimezone=Asia/Shanghai
mysql.datasource.write.username=root

b, write DbContextHolder class

This class is used to set the database category. There is a ThreadLocal to store whether each thread uses the read library or the write library. code show as below:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

/**
 * Description 这里切换读/写模式
 * 原理是利用ThreadLocal保存当前线程是否处于读模式（通过开始READ_ONLY注解在开始操作前设置模式为读模式，
 * 操作结束后清除该数据，避免内存泄漏，同时也为了后续在该线程进行写操作时任然为读模式
 *
 * @author 杜康
 * @date 2020-08-31
 */
public class DbContextHolder {

    private static Logger log = LoggerFactory.getLogger(DbContextHolder.class);
    public static final String WRITE = "write";
    public static final String READ = "read";

    private static ThreadLocal<String> contextHolder= new ThreadLocal<>();

    public static void setDbType(String dbType) {
        if (dbType == null) {
            log.error("dbType为空");
            throw new NullPointerException();
        }
        log.info("设置dbType为：{}",dbType);
        contextHolder.set(dbType);
    }

    public static String getDbType() {
        return contextHolder.get() == null ? WRITE : contextHolder.get();
    }

    public static void clearDbType() {
        contextHolder.remove();
    }
}

c, rewrite the determineCurrentLookupKey method

Spring will use this method to decide which database to use when it starts database operations, so we call the getDbType()method of the DbContextHolder class above to get the current operation category, and at the same time, load balancing of the reading library can be performed. The code is as follows

package com.gary.dbrw.datachange;

import com.gary.dbrw.util.NumberUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;

public class MyAbstractRoutingDataSource extends AbstractRoutingDataSource {

    @Value("${mysql.datasource.num}")
    private int num;

    private final Logger log = LoggerFactory.getLogger(this.getClass());

    @Override
    protected Object determineCurrentLookupKey() {
        String typeKey = DbContextHolder.getDbType();
        if (typeKey.equals(DbContextHolder.WRITE)) {
            log.info("使用了写库w");
            return typeKey;
        }
        //使用随机数决定使用哪个读库（可写负载均衡算法）
        int sum = NumberUtil.getRandom(1, num);
        log.info("使用了读库r{}", sum);
        return DbContextHolder.READ + sum;
    }
}

d, write configuration class

Due to the separation of reading and writing, the default configuration of springboot can no longer be used, and we need to configure it manually. First generate the data source, use @ConfigurProperties to automatically generate the data source:

package com.gary.dbrw.config;

import com.alibaba.druid.pool.DruidDataSource;
import com.gary.dbrw.common.Properties;
import com.gary.dbrw.datachange.DbContextHolder;
import com.gary.dbrw.datachange.MyAbstractRoutingDataSource;
import org.apache.ibatis.session.SqlSessionFactory;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.annotation.MapperScan;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.core.io.support.ResourcePatternResolver;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;

import javax.sql.DataSource;
import java.sql.SQLException;
import java.util.HashMap;
import java.util.Map;

@MapperScan(basePackages = "com.gary.dbrw.mapper", sqlSessionFactoryRef = "sqlSessionFactory")
@Configuration
public class DataSourceConfig {

//    @Value("${mysql.datasource.type-aliases-package}")
//    private String typeAliasesPackage;
//
//    @Value("${mysql.datasource.mapper-locations}")
//    private String mapperLocation;
//
//    @Value("${mysql.datasource.config-location}")
//    private String configLocation;

    /**
     * 写数据源
     *
     * @Primary 标志这个 Bean 如果在多个同类 Bean 候选时，该 Bean 优先被考虑。
     * 多数据源配置的时候注意，必须要有一个主数据源，用 @Primary 标志该 Bean
     */
    @Primary
    @Bean
    @ConfigurationProperties(prefix = "mysql.datasource.write")
    public DataSource writeDataSource() {
        return new DruidDataSource();
    }

    /**
     * 读数据源
     */
    @Bean
    @ConfigurationProperties(prefix = "mysql.datasource.read1")
    public DataSource read1() {
        return new DruidDataSource();
    }


    /**
     * 多数据源需要自己设置sqlSessionFactory
     */
    @Bean
    public SqlSessionFactory sqlSessionFactory() throws Exception {
        SqlSessionFactoryBean bean = new SqlSessionFactoryBean();
        bean.setDataSource(routingDataSource());
        ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
        // 实体类对应的位置
        bean.setTypeAliasesPackage(Properties.typeAliasesPackage);
        // mybatis的XML的配置
        bean.setMapperLocations(resolver.getResources(Properties.mapperLocation));
        bean.setConfigLocation(resolver.getResource(Properties.configLocation));
        return bean.getObject();
    }

    /**
     * 设置事务，事务需要知道当前使用的是哪个数据源才能进行事务处理
     */
    @Bean
    public DataSourceTransactionManager dataSourceTransactionManager() {
        return new DataSourceTransactionManager(routingDataSource());
    }

    /**
     * 设置数据源路由，通过该类中的determineCurrentLookupKey决定使用哪个数据源
     */
    @Bean
    public AbstractRoutingDataSource routingDataSource() {
        MyAbstractRoutingDataSource proxy = new MyAbstractRoutingDataSource();
        Map<Object, Object> targetDataSources = new HashMap<>(2);
        targetDataSources.put(DbContextHolder.WRITE, writeDataSource());
        targetDataSources.put(DbContextHolder.READ + "1", read1());
        proxy.setDefaultTargetDataSource(writeDataSource());
        proxy.setTargetDataSources(targetDataSources);
        return proxy;
    }

}

 

Note how many read data sources must be set up for how many read libraries there are, and the Bean name is read + serial number.

Finally, set the data source, using the MyAbstractRoutingDataSource class we wrote before

  /**
     * 设置数据源路由，通过该类中的determineCurrentLookupKey决定使用哪个数据源
     */
    @Bean
    public AbstractRoutingDataSource routingDataSource() {
        MyAbstractRoutingDataSource proxy = new MyAbstractRoutingDataSource();
        Map<Object, Object> targetDataSources = new HashMap<>(2);
        targetDataSources.put(DbContextHolder.WRITE, writeDataSource());
        targetDataSources.put(DbContextHolder.READ + "1", read1());
        proxy.setDefaultTargetDataSource(writeDataSource());
        proxy.setTargetDataSources(targetDataSources);
        return proxy;
    }

Then you need to set up sqlSessionFactory, configure MyBatis's sqlSessionFactoryRef reference

 /**
     * 多数据源需要自己设置sqlSessionFactory
     */
    @Bean
    public SqlSessionFactory sqlSessionFactory() throws Exception {
        SqlSessionFactoryBean bean = new SqlSessionFactoryBean();
        bean.setDataSource(routingDataSource());
        ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
        // 实体类对应的位置
        bean.setTypeAliasesPackage(Properties.typeAliasesPackage);
        // mybatis的XML的配置
        bean.setMapperLocations(resolver.getResources(Properties.mapperLocation));
        bean.setConfigLocation(resolver.getResource(Properties.configLocation));
        return bean.getObject();
    }

@MapperScan(basePackages = "com.gary.dbrw.mapper", sqlSessionFactoryRef = "sqlSessionFactory")
@Configuration
public class DataSourceConfig {
   
   

Finally, you have to configure the transaction, otherwise the transaction will not take effect

 /**
     * 设置事务，事务需要知道当前使用的是哪个数据源才能进行事务处理
     */
    @Bean
    public DataSourceTransactionManager dataSourceTransactionManager() {
        return new DataSourceTransactionManager(routingDataSource());
    }

 

4), select the data source

The multiple data sources are configured, but how to choose the data source at the code level?

Use the popular annotations to add aspects.

First define a read-only annotation. This annotation method uses the read library, and others use the write library. If the project is transformed into read-write separation in the middle, you can use this method. No need to modify the business code, just add an annotation to the read-only service method That's it.

@Target({ElementType.METHOD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface ReadOnly {
}

Then write an aspect to switch which data source is used for the data, rewrite getOrder to ensure that the priority of this aspect is higher than the priority of the transaction aspect, (so that transactions can be used) add @EnableTransactionManagement(order = 10)it to the startup class , for the code as follows:

@Aspect
@Component
public class ReadOnlyInterceptor implements Ordered {
    private static final Logger log = LoggerFactory.getLogger(ReadOnlyInterceptor.class);

    @Around("@annotation(readOnly)")
    public Object setRead(ProceedingJoinPoint joinPoint, ReadOnly readOnly) throws Throwable {
        try {
            DbContextHolder.setDbType(DbContextHolder.READ);
            return joinPoint.proceed();
        } finally {
            //清楚DbType一方面为了避免内存泄漏，更重要的是避免对后续在本线程上执行的操作产生影响
            DbContextHolder.clearDbType();
            log.info("清除threadLocal");
        }
    }

    @Override
    public int getOrder() {
        return 0;
    }
}

Configure the test class:

@Service
public class StudentServiceImpl implements StudentService {
    @Resource
    StudentMapper studentMapper;

    @ReadOnly
    @Override
    public List<Student> selectAllList() {
        return studentMapper.selectAll();
    }


    @Override
    public int addOneStudent(Student student) {
        return studentMapper.insertSelective(student);
    }
}

@RestController
@RequestMapping("user")
public class StudentController {

    @Resource
    StudentService studentService;

    @GetMapping("/testRW")
    public String DbRead(Integer dbType) {
        System.out.println("dbType=:" + dbType);
        List<Student> students = studentService.selectAllList();
        return "ReadDB=>" + students;
    }

    @PostMapping("/testRW")
    public String DbWrite(Student student) {
        int count = studentService.addOneStudent(student);
        return "Wr DB=>" + count;
    }
}

4. Test verification

Write the code to try the result, the following is a screenshot of the operation:

 

---

Database change view:

---