Springboot integrates ShardingSphere to realize sub-database sub-table, vertical split, horizontal split, and common table processing

There are two main solutions for separate tables: vertical split and horizontal split

vertical split

Operate a table in the database, save a part of the field data in this table into a new table, and then save another part of the field data in this table into another table. in a table

split horizontally

Split a database into two databases with the same structure to store data

storage method

Method 1: Implemented by id modulo, disadvantage: not easy to expand

Method 2: Implemented by sharding. Disadvantage: Because the id is continuous, the hot data may be concentrated in one database

Other ideas: divide a large range by time, and divide a small range by modulo

Possible problems with sub-libraries and sub-tables:

Transaction consistency problem: single-table operation transaction execution failure can be rolled back, and may be inserted into multiple databases after the table is divided

Cross-node association query: when querying a fixed range of data also in multiple databases

Primary key avoidance problem: database table primary key conflict

Therefore, in development: do not do horizontal splitting immediately when data increases, first consider cache processing, separation of read and write, use of indexes, etc.

ShardingSphere

Open source distributed database middleware solutions, mainly Sharding-JDBC and Sharding-Proxy

Client sub-database sub-table: put it in the java application in the form of a jar package, and find the corresponding data through Sharding-JDBC, mainly for data fragmentation, read-write separation

Server-side sub-database and sub-table: use Sharding-Proxy to disguise as a database, and sub-database and sub-table are implemented by Sharding-Proxy

Sharding-JDBC implements horizontal table division

Create a database and split tables horizontally

Create two tables course_1 and course_2 in the database

Convention rules: if the added course id is an even number, add the data to course_1, if the odd number add it to course_2

Create a springboot project and add dependencies

<dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter</artifactId>
     </dependency>
     <dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter-test</artifactId>
     </dependency>
     <dependency>
     <groupId>com.alibaba</groupId>
     <artifactId>druid-spring-boot-starter</artifactId>
     <version>1.1.20</version>
     </dependency>
     <dependency>
     <groupId>mysql</groupId>
     <artifactId>mysql-connector-java</artifactId>
     </dependency>
     <dependency>
     <groupId>org.apache.shardingsphere</groupId>
     <artifactId>sharding-jdbc-spring-boot-starter</artifactId>
     <version>4.0.0-RC1</version>
     </dependency>
     <dependency>
     <groupId>com.baomidou</groupId>
     <artifactId>mybatis-plus-boot-starter</artifactId>
     <version>3.0.5</version>
     </dependency>
     <dependency>
     <groupId>org.projectlombok</groupId>
     <artifactId>lombok</artifactId>
     </dependency>

entity class

@Data
public class Course {
    private Long cid;
    private String cname;
    private Long userId;
    private String cstatus;
}

mapper layer

@Repository
public interface CourseMapper extends BaseMapper<Course> {
}

Startup class annotation

@MapperScan("com.lzq.mapper")

Configure Sharding-JDBC sharding strategy

Configure in config file

# shardingjdbc 分片策略
# 配置数据源，给数据源起名称
spring.shardingsphere.datasource.names=m1

# 一个实体类对应两张表，覆盖
spring.main.allow-bean-definition-overriding=true

#配置数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.m1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m1.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m1.url=jdbc:mysql://localhost:3306/sharding?serverTimezone=GMT%2B8
spring.shardingsphere.datasource.m1.username=root
spring.shardingsphere.datasource.m1.password=root
#指定 course 表分布情况，配置表在哪个数据库里面，表名称都是什么 m1.course_1 ,m1.course_2
spring.shardingsphere.sharding.tables.course.actual-data-nodes=m1.course_$->{1..2}
# 指定 course 表里面主键 cid 生成策略 SNOWFLAKE 雪花算法
spring.shardingsphere.sharding.tables.course.key-generator.column=cid
spring.shardingsphere.sharding.tables.course.key-generator.type=SNOWFLAKE
# 指定分片策略 约定 cid 值偶数添加到 course_1 表，如果 cid 是奇数添加到 course_2表
spring.shardingsphere.sharding.tables.course.table-strategy.inline.sharding-column=cid
spring.shardingsphere.sharding.tables.course.table-strategy.inline.algorithm-expression=course_$->{cid % 2 + 1}
# 打开 sql 输出日志
spring.shardingsphere.props.sql.show=true

test

@SpringBootTest
class ShardingJdbcApplicationTests {
    @Autowired
    private CourseMapper courseMapper;
    @Test
    void add() {
        Course course = new Course();
        course.setCname("java");
        course.setUserId(10l);
        course.setCstatus("normal");
        courseMapper.insert(course);
    }
    @Test
    public void find(){
        QueryWrapper<Course> wrapper = new QueryWrapper<>();
        wrapper.eq("cid",714887289653690369l);
        Course course = courseMapper.selectOne(wrapper);
        System.out.println(course);
    }
}

Different versions of springboot may report errors. Since one entity class corresponds to two databases, add it in the configuration file

spring.main.allow-bean-definition-overriding=true

Horizontal sub-library

Create two databases, and create two tables in each database, course_1 and course_2

Agreed sharding rules: userid is an even number to join the first database, and cid is an even number to add to the course_1 table

Modify the previous code: configure the sharding rules and add them to the configuration file

# 配置数据源，给数据源起名称,
# 水平分库，配置两个数据源
spring.shardingsphere.datasource.names=m1,m2 
 
# 一个实体类对应两张表，覆盖
spring.main.allow-bean-definition-overriding=true 
 
#配置第一个数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.m1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m1.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m1.url=jdbc:mysql://localhost:3306/edu_db_1?serverTimezone=GMT%2B8 
spring.shardingsphere.datasource.m1.username=root 
spring.shardingsphere.datasource.m1.password=root 
 
#配置第二个数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.m2.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m2.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m2.url=jdbc:mysql://localhost:3306/edu_db_2?serverTimezone=GMT%2B8 
spring.shardingsphere.datasource.m2.username=root 
spring.shardingsphere.datasource.m2.password=root 
 
#指定数据库分布情况，数据库里面表分布情况
# m1 m2 course_1 course_2
spring.shardingsphere.sharding.tables.course.actual-data-nodes=m$->{1..2}.course_$->{1..2} 
 
# 指定 course 表里面主键 cid 生成策略 SNOWFLAKE
spring.shardingsphere.sharding.tables.course.key-generator.column=cid 
spring.shardingsphere.sharding.tables.course.key-generator.type=SNOWFLAKE 
 
# 指定表分片策略 约定 cid 值偶数添加到 course_1 表，如果 cid 是奇数添加到course_2 表
spring.shardingsphere.sharding.tables.course.table-strategy.inline.sharding.column=cid 
spring.shardingsphere.sharding.tables.course.table-strategy.inline.algorithm.expression=course_$->{cid % 2 + 1} 
 
# 指定数据库分片策略 约定 user_id 是偶数添加 m1，是奇数添加 m2
#spring.shardingsphere.sharding.default-database-strategy.inline.sharding.column=user_id
#spring.shardingsphere.sharding.default-database-strategy.inline.algorithm.expression=m$->{user_id % 2 + 1}
#写法二 对具体表有固定规则
spring.shardingsphere.sharding.tables.course.database-strategy.inline.sharding-column=user_id 
spring.shardingsphere.sharding.tables.course.database-strategy.inline.algorithm-expression=m$->{user_id % 2 + 1}

Test as above

Vertical library :

It is dedicated to the special database. When querying user information, go to the user database to check the user table

Create database user_db, create user information table t_user

Create entity class and mapper

@Data
@TableName(value = "t_user") //指定对应表
public class User {
    private Long userId;
    private String username;
    private String ustatus;
}

@Repository
public interface UserMapper extends BaseMapper<User> {
}

config file add

# 配置数据源，给数据源起名称
spring.shardingsphere.datasource.names=m1,m2,m0
.....
#配置第三个数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.m0.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m0.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m0.url=jdbc:mysql://localhost:3306/user_db?serverTimezone=GMT%2B8 
spring.shardingsphere.datasource.m0.username=root 
spring.shardingsphere.datasource.m0.password=root 
 
# 配置 user_db 数据库里面 t_user 专库专表
spring.shardingsphere.sharding.tables.t_user.actual-data-nodes=m$->{0}.t_user 
 
# 指定 course 表里面主键 cid 生成策略 SNOWFLAKE
spring.shardingsphere.sharding.tables.t_user.key-generator.column=user_id 
spring.shardingsphere.sharding.tables.t_user.key-generator.type=SNOWFLAKE 

public table

Store fixed data, which rarely changes, and needs to be associated when querying, such as the user's state table, dictionary, etc.

Create tables with the same structure in multiple data needs

add in config file

# 配置公共表
spring.shardingsphere.sharding.broadcast-tables=t_udict 
spring.shardingsphere.sharding.tables.t_udict.key-generator.column=dictid 
spring.shardingsphere.sharding.tables.t_udict.key-generator.type=SNOWFLAKE

When there is a modification operation, it will modify the t_udict table in multiple databases