A powerful tool for sub-database and sub-table——shardingJdbc

Enterprise 2023-05-04 21:25:35 views: null

1. Sub-database and sub-table method

1.1 Vertical split

1.1.1 Vertical table split (split fields - but the amount of data in each table is constant)

Store some fields of one table in a new table, and store another part of fields in another new table
insert image description here

1.1.2 Vertical sub-database (special database and table)

Split a single database according to business, special database table, reduce the pressure on a single database
insert image description here

1.2 Horizontal splitting

1.2.1 Horizontal table

Copy several duplicate data tables with the same structure on the basis of the original data table, and then read them according to specific rules to further reduce the pressure on a single table
insert image description here

1.2.2 Horizontal sub-library

Copy several copy databases with the same structure on the basis of the original database, and then read them according to specific rules to further alleviate the pressure of single database
insert image description here

2. The application and problems of sub-database and sub-table

2.1 Application

(1) Vertical sharding and vertical table sharding should be considered in database design
(2) As the amount of data continues to increase, do not immediately consider horizontal sharding. First, consider cache processing, read-write separation, and indexing. If These methods cannot fundamentally solve the problem, and then consider doing horizontal sub-database and sub-table

2.2 Problems

(1) Cross-node connection query (when connecting and querying, you need to check it out separately in different databases and then splicing and summarizing the data)
insert image description here

(2) Multiple data source management issues

3. sharding-Jdbc (lightweight Java framework, enhanced version of JDBC)

It does not do sub-database sub-table, but performs related operations on the data table after sub-database sub-table. Main functions: data sharding, read-write separation

3.1 Horizontal table

3.1.1 Environment Construction

(1) Create a database course_db and two tables, namely course_1 and course_2, both of which have the same structure
insert image description here

(2) Introduce dependencies

<!--  引入druid      -->
<dependency>
  <groupId>com.alibaba</groupId>
  <artifactId>druid-spring-boot-starter</artifactId>
  <version>1.2.11</version>
</dependency>
<!--  引入mybatis-plus      -->
<dependency>
  <groupId>com.baomidou</groupId>
  <artifactId>mybatis-plus-boot-starter</artifactId>
  <version>3.4.3.4</version>
</dependency>
<!--  mysql驱动      -->
<dependency>
  <groupId>mysql</groupId>
  <artifactId>mysql-connector-java</artifactId>
  <version>5.1.49</version>
</dependency>
<!--  sharding-JDBC      -->
<dependency>
  <groupId>org.apache.shardingsphere</groupId>
  <artifactId>sharding-jdbc-spring-boot-starter</artifactId>
  <version>4.0.0-RC1</version>
</dependency>

(3) Configuration file ( official document )

# shardingjdbc 分片策略

# 配置数据源，给数据源起名称
spring.shardingsphere.datasource.names=db1

# 一个实体类对应多张表，覆盖
spring.main.allow-bean-definition-overriding=true

# 配置数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.db1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.db1.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.db1.url=jdbc:mysql://127.0.0.1:3306/course_db?useUnicode=true&&characterEncoding=utf-8&&useSSL=false
spring.shardingsphere.datasource.db1.username=root
spring.shardingsphere.datasource.db1.password=root

# 指定course表分布情况，配置表在哪个数据库里面，表名称都是什么 m1.course_1, m1.course_2
spring.shardingsphere.sharding.tables.course.actual-data-nodes=db1.course_$->{
    
    1..2}

# 指定course表的主键生成策略
spring.shardingsphere.sharding.tables.course.key-generator.column=cid
spring.shardingsphere.sharding.tables.course.key-generator.type=SNOWFLAKE

# 数据表分片策略 约定cid为偶数存course_1表, 为奇数存course_2表
spring.shardingsphere.sharding.tables.course.table-strategy.inline.sharding-column=cid
spring.shardingsphere.sharding.tables.course.table-strategy.inline.algorithm-expression=course_$->{
    
    cid % 2+1}

# 开启sql输出日志
spring.shardingsphere.props.sql.show=true

(4) Insert data test

    @Test
    public void addCourse() {
    
    
    for (int i = 1; i < 10; i++) {
    
    
        Course course = new Course();
        course.setCname("Java "+i);
        course.setUserId(100L);
        course.setCstatus("Normal");
        courseMapper.insert(course);
    }
}

insert image description here
insert image description here
The primary key cid is all even numbers in the course_1 table, and the odd numbers are in the course_2 table
(5) Query test

    @Test
    public void selectCourse() {
    
    
        QueryWrapper<Course> wrapper = new QueryWrapper<>();
        wrapper.eq("cid",770051025917706241L);
        System.out.println(courseMapper.selectOne(wrapper));
    }

insert image description here
770051025917706241 is an odd number, so the query is still in the course_2 table

3.2 Horizontal sub-library

3.2.1 Environment Construction

(1) Create two databases with the same structure
insert image description here
Database rules:
Add to edu_db_1 database when userid is even, add to edu_db_2 database when userid is odd
Table rules:
Add to course_1 table when cid is even, add to course_2 table when it is odd
insert image description here
(2 ) configuration file

# shardingjdbc 分片策略（水平分库）
# 配置数据源，给数据源起名称（对应两个数据库）
spring.shardingsphere.datasource.names=db1,db2

# 一个实体类对应多张表，覆盖
spring.main.allow-bean-definition-overriding=true

# 配置第一个数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.db1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.db1.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.db1.url=jdbc:mysql://127.0.0.1:3306/edu_db_1?useUnicode=true&&characterEncoding=utf-8&&useSSL=false
spring.shardingsphere.datasource.db1.username=root
spring.shardingsphere.datasource.db1.password=root

# 配置第二个数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.db2.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.db2.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.db2.url=jdbc:mysql://127.0.0.1:3306/edu_db_2?useUnicode=true&&characterEncoding=utf-8&&useSSL=false
spring.shardingsphere.datasource.db2.username=root
spring.shardingsphere.datasource.db2.password=root

# 指定数据库分布情况，数据表分布情况
# db1 db2      course_1  course_2
spring.shardingsphere.sharding.tables.course.actual-data-nodes=db$->{
    
    1..2}.course_$->{
    
    1..2}
# 指定course表的主键生成策略
spring.shardingsphere.sharding.tables.course.key-generator.column=cid
spring.shardingsphere.sharding.tables.course.key-generator.type=SNOWFLAKE
# 指定数据库分片策略(默认的，也就是对全部表的user_id字段都生效)
#spring.shardingsphere.sharding.default-database-strategy.inline.sharding-column=user_id
#spring.shardingsphere.sharding.default-database-strategy.inline.algorithm-expression=edu_db_$->{user_id%2+1}
# 指定数据库分片策略(指定表的某些字段生效)
spring.shardingsphere.sharding.tables.course.database-strategy.inline.sharding-column=user_id
spring.shardingsphere.sharding.tables.course.database-strategy.inline.algorithm-expression=db$->{
    
    user_id % 2+1}
# 数据表分片策略 约定cid为偶数存course_1表, 为奇数存course_2表
spring.shardingsphere.sharding.tables.course.table-strategy.inline.sharding-column=cid
spring.shardingsphere.sharding.tables.course.table-strategy.inline.algorithm-expression=course_$->{
    
    cid % 2+1}
# 开启sql输出日志
spring.shardingsphere.props.sql.show=true

(3) test

    @Test
    public void insertCourse() {
    
    
    for (int i = 1; i <= 10; i++) {
    
    
        Course course = new Course();
        //数据库(偶数在1 奇数在2)
        course.setUserId(11L);
        course.setCname("水平分库-" + i);
        course.setCstatus("Normal");
        courseMapper.insert(course);
    }
}

If the user_id is an even number, it will be stored in the No. 1 library, if the user_id is an odd number, it will be stored in the No. 2 library; if the cid
is an even number, it will be stored in the No. 1 table, and if the cid is an odd number, it will be stored in the No. 2 table
insert image description here
insert image description here

3.3 Vertical sub-database (special database and table)

3.3.1 Environment Construction

(1) Create a database
Add another database user_db on the basis of the above horizontal sub-database, and create a table t_user
insert image description here
(2) Configuration file

# shardingjdbc 分片策略（垂直分库————专库专表）
# 配置数据源，给数据源起名称（对应两个数据库）
spring.shardingsphere.datasource.names=db1,db2,db3

# 一个实体类对应多张表，覆盖
spring.main.allow-bean-definition-overriding=true

# 配置第一个数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.db1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.db1.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.db1.url=jdbc:mysql://127.0.0.1:3306/edu_db_1?useUnicode=true&&characterEncoding=utf-8&&useSSL=false
spring.shardingsphere.datasource.db1.username=root
spring.shardingsphere.datasource.db1.password=root

# 配置第二个数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.db2.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.db2.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.db2.url=jdbc:mysql://127.0.0.1:3306/edu_db_2?useUnicode=true&&characterEncoding=utf-8&&useSSL=false
spring.shardingsphere.datasource.db2.username=root
spring.shardingsphere.datasource.db2.password=root

# 配置第三个数据源具体内容，包含连接池，驱动，地址，用户名和密码
spring.shardingsphere.datasource.db3.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.db3.driver-class-name=com.mysql.jdbc.Driver
spring.shardingsphere.datasource.db3.url=jdbc:mysql://127.0.0.1:3306/user_db?useUnicode=true&&characterEncoding=utf-8&&useSSL=false
spring.shardingsphere.datasource.db3.username=root
spring.shardingsphere.datasource.db3.password=root

#--------------------------------------user_db数据库里面t_user 专库专表----------------------------------------------
# 配置user_db数据库里面t_user专库专表
spring.shardingsphere.sharding.tables.t_user.actual-data-nodes=db3.t_user

# 指定course表的主键生成策略
spring.shardingsphere.sharding.tables.t_user.key-generator.column=user_id
spring.shardingsphere.sharding.tables.t_user.key-generator.type=SNOWFLAKE

# 数据表分片策略 user表
spring.shardingsphere.sharding.tables.t_user.table-strategy.inline.sharding-column=user_id
spring.shardingsphere.sharding.tables.t_user.table-strategy.inline.algorithm-expression=t_user

#----------------------------------------edu_db_数据库里面course表 水平分库--------------------------------------------
# 配置edu_db_数据库里面course表的情况
spring.shardingsphere.sharding.tables.course.actual-data-nodes=db$->{
    
    1..2}.course_$->{
    
    1..2}

# 指定course表的主键生成策略
spring.shardingsphere.sharding.tables.course.key-generator.column=cid
spring.shardingsphere.sharding.tables.course.key-generator.type=SNOWFLAKE

# 数据表分片策略 约定cid为偶数存course_1表, 为奇数存course_2表
spring.shardingsphere.sharding.tables.course.table-strategy.inline.sharding-column=cid
spring.shardingsphere.sharding.tables.course.table-strategy.inline.algorithm-expression=course_$->{
    
    cid % 2+1}

# 指定数据库分片策略(默认的，也就是对全部表的user_id字段都生效)
#spring.shardingsphere.sharding.default-database-strategy.inline.sharding-column=user_id
#spring.shardingsphere.sharding.default-database-strategy.inline.algorithm-expression=edu_db_$->{user_id%2+1}

# 指定数据库分片策略(指定表的某些字段生效)
spring.shardingsphere.sharding.tables.course.database-strategy.inline.sharding-column=user_id
spring.shardingsphere.sharding.tables.course.database-strategy.inline.algorithm-expression=db$->{
    
    user_id % 2+1}

# 开启sql输出日志
spring.shardingsphere.props.sql.show=true

(3) test

@Test
    public void addUser(){
    
    
    User user = new User();
    user.setUsername("lucy");
    user.setUstatus("Normal");
    userMapper.insert(user);
}

@Test
    public void findUser(){
    
    
    List<User> users = userMapper.selectList(null);
    System.out.println(users);
}

Note: The name of the entity class table corresponding to the data table must be the same as the name of the database table (it must be able to correspond, otherwise an exception of datasource is null may be thrown) - it is best to specify the name of the database table

4. Sharding-JDBC operates public tables

4.1 Public tables

(1) Tables that store fixed data, the table data rarely changes, and are often associated when querying
(2) There are tables with exactly the same structure in each database
to implement common operations, that is, common tables in each database The table is added, modified, and deleted at the same time
. Add configuration:

# 配置公共表
spring.shardingsphere.sharding.broadcast-tables=t_common
spring.shardingsphere.sharding.tables.t_common.key-generator.column=common_id
spring.shardingsphere.sharding.tables.t_common.key-generator.type=SNOWFLAKE

Five, read and write separation

5.1 Concept

In order to ensure the stability of the database, most of them have the function of dual-machine hot backup. The principle is: the master database (master) handles transactional addition, deletion, and modification operations; the slave database (slave) handles select operations.
insert image description here

Master-slave replication: when the master server has a write operation, the slave server automatically obtains
the principle: the slave server monitors the binlog log of the master server in real time, and when the user writes to the master server, the binlog log is changed, and the binlog log is pulled from the server to read and Parse execution. So as to achieve synchronization with the main database.

Read-write separation: write operations are on the master server, and read operations are on the slave server

5.2 mysql configuration read and write separation

(1) Create two mysql database services.
Directly copy the previous mysql installation root directory,
insert image description here
modify the configuration (port number, installation directory, data storage directory), and
insert image description here
install the windows service from the database after the modification. Mysqls1 is the service name
cmd administrator authority:
registration service

mysqld install mysqls1 --defaults-file=“D:\code\mysql-5.7.23-winx64-s1\my.ini”

delete service

sc delete service name
insert image description here
to connect to the slave database,
insert image description here
(2) configure mysql master-slave database
and modify the master database configuration file my.ini

[mysqld]
#开启日志
log-bin = mysql-bin
#设置服务id,主从不能一致
server-id = 1
#选择ROW模式
binlog_format=ROW
#设置需要同步的数据库
binlog-do-db=user_db
#屏蔽系统库同步
binlog-ignore-db=mysql
binlog-ignore-db=information_schema
binlog-ignore-db=performance_schema

Modify the slave database configuration file my.ini

[mysqld]
#从服务器配置
#开启日志
log-bin=mysql-bin
#设置服务id，主从不能一致
server-id=2
#选择ROW模式
binlog_format=ROW
#设置需要同步的数据库
replicate_wild_do_table=user_db.%
#屏蔽系统库同步
replicate_wild_ignore_table=mysql.%
replicate_wild_ignore_table=information-schema.%
replicate_wild_ignore_table=performance_schema.%

Guess you like

Origin blog.csdn.net/weixin_42194695/article/details/127934931
Recommended
Linus is the most active in "eating dog food"!
Ranking
Share good programmer web front-end array and sorting, de-duplication and random roll call
Compilation error caused by cv_bridge and python version problems error: return-statement with no value, in function returning'void*' [-fpe
魔众帮助中心系统 v3.1.0 首页切换器，界面优化
Die beim Millimeterwellenradar-Integrationstest aufgetretene Grube (Multiprozessbindung an einen UDP-Port verursacht Probleme)
How to suppress the "requires transitive directive for an automatic module" warning properly?
LeetCode-1743. Restore the Array From Adjacent Pairs-Analysis and Code (Java)
Summer 2019 Summer soft essay 7 workers
Python中Assert断言的使用语法和例子
LeetCode one question per day (2021-2-3 sliding window median)
Fairchild, the ancestor of semiconductors, the legend of the first trillion-dollar start-up
Daily
More
2024-05-20(5)
2024-05-19(0)
2024-05-18(31)
2024-05-17(6)
2024-05-16(23)
2024-05-15(5)
2024-05-14(9)
2024-05-13(8)
2024-05-12(28)
2024-05-11(32)

Code World

© 2018 codetd.com