Various function points and core source code of the third lesson of shardingsphere

1. Function points

1. Audit function

The shard audit function is for auditing the executed SQL statements in the database sharding scenario. Fragmentation audit can not only intercept illegal SQL statements configured by the system, but also perform statistical operations on SQL statements.
Currently, there is only one sharding audit algorithm built into ShardingSphere, DML_SHARDING_CONDITIONS. Its function is to require the shard key to be included when querying the logical table.

# 分片审计规则： SQL查询必须带上分片键
spring.shardingsphere.rules.sharding.tables.course.audit-strategy.auditor-names[0]=course_auditor
spring.shardingsphere.rules.sharding.tables.course.audit-strategy.allow-hint-disable=true

spring.shardingsphere.rules.sharding.auditors.course_auditor.type=DML_SHARDING_CONDITIONS

When the query does not include the shard id, an error will be reported.

When the shard id is included, there is no problem

2. Encryption

ShardingSphere has a variety of built-in encryption algorithms that can be used to quickly encrypt key data.
Actually here is to store the plaintext in the password field, store the ciphertext in the password_cipher field, and specify the corresponding encryptor

# 打印SQL
spring.shardingsphere.props.sql-show = true
spring.main.allow-bean-definition-overriding = true

# ----------------数据源配置
# 指定对应的库
spring.shardingsphere.datasource.names=m0,m1

spring.shardingsphere.datasource.m0.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m0.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m0.url=jdbc:mysql://localhost:3306/coursedb?serverTimezone=UTC
spring.shardingsphere.datasource.m0.username=root
spring.shardingsphere.datasource.m0.password=root

spring.shardingsphere.datasource.m1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m1.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m1.url=jdbc:mysql://localhost:3306/coursedb2?serverTimezone=UTC
spring.shardingsphere.datasource.m1.username=root
spring.shardingsphere.datasource.m1.password=root
#------------------------分布式序列算法配置
# 生成字符串类型分布式主键。
spring.shardingsphere.rules.sharding.key-generators.user_keygen.type=NANOID
#spring.shardingsphere.rules.sharding.key-generators.user_keygen.type=UUID
# 指定分布式主键生成策略
spring.shardingsphere.rules.sharding.tables.user.key-generate-strategy.column=userid
spring.shardingsphere.rules.sharding.tables.user.key-generate-strategy.key-generator-name=user_keygen
#-----------------------配置实际分片节点
spring.shardingsphere.rules.sharding.tables.user.actual-data-nodes=m$->{
    
    0..1}.user_$->{
    
    1..2}
# HASH_MOD分库
spring.shardingsphere.rules.sharding.tables.user.database-strategy.standard.sharding-column=userid
spring.shardingsphere.rules.sharding.tables.user.database-strategy.standard.sharding-algorithm-name=user_db_alg

spring.shardingsphere.rules.sharding.sharding-algorithms.user_db_alg.type=HASH_MOD
spring.shardingsphere.rules.sharding.sharding-algorithms.user_db_alg.props.sharding-count=2
# HASH_MOD分表
spring.shardingsphere.rules.sharding.tables.user.table-strategy.standard.sharding-column=userid
spring.shardingsphere.rules.sharding.tables.user.table-strategy.standard.sharding-algorithm-name=user_tbl_alg

spring.shardingsphere.rules.sharding.sharding-algorithms.user_tbl_alg.type=INLINE
# 字符串类型要先hashcode转为long，再取模。但是Grovvy的 "xxx".hashcode%2 不知道为什么会产生 -1,0,1三种结果
#spring.shardingsphere.rules.sharding.sharding-algorithms.user_tbl_alg.props.algorithm-expression=user_$->{Math.abs(userid.hashCode()%2) +1}
# 用户信息分到四个表
spring.shardingsphere.rules.sharding.sharding-algorithms.user_tbl_alg.props.algorithm-expression=user_$->{
    
    Math.abs(userid.hashCode()%4).intdiv(2) +1}
# 数据加密:对password字段进行加密
# 存储明文的字段
spring.shardingsphere.rules.encrypt.tables.user.columns.password.plainColumn = password
# 存储密文的字段
spring.shardingsphere.rules.encrypt.tables.user.columns.password.cipherColumn = password_cipher
# 加密器
spring.shardingsphere.rules.encrypt.tables.user.columns.password.encryptorName = user_password_encry
# AES加密器
#spring.shardingsphere.rules.encrypt.encryptors.user_password_encry.type=AES
#spring.shardingsphere.rules.encrypt.encryptors.user_password_encry.props.aes-key-value=123456
# MD5加密器
#spring.shardingsphere.rules.encrypt.encryptors.user_password_encry.type=MD5
# SM3加密器
spring.shardingsphere.rules.encrypt.encryptors.user_password_encry.type=SM3
spring.shardingsphere.rules.encrypt.encryptors.user_password_encry.props.sm3-salt=12345678

# sm4加密器
#spring.shardingsphere.rules.encrypt.encryptors.user_password_encry.type=SM4

Next, when querying, you can actively pass in the password_cipher query field, and query according to the ciphertext. At the same time, the query for the password field will also be converted into a ciphertext query. Query case

@Test
public void queryUser() {
    
    
QueryWrapper<User> queryWrapper = new QueryWrapper<>();
// queryWrapper.eq("userid","1644954727911317506");
queryWrapper.eq("password","123qweasd");
//
queryWrapper.eq("password_cipher","c7d79d9c0898f7c4e252cd1ec19ed0c5c91aae0b0bc8bafc5f38322
418215d38");
List<User> users = userMapper.selectList(queryWrapper);
for(User user : users){
    
    
System.out.println(user);
}

3. Read and write separation

In ShardingSphere, it is also very simple to achieve read-write separation. You only need to create a
sharding rule of type readwrite-splitting. Let's use the previously created user table to quickly configure an example of read-write separation.

# 打印SQL
spring.shardingsphere.props.sql-show = true
spring.main.allow-bean-definition-overriding = true

# ----------------数据源配置
# 指定对应的库
spring.shardingsphere.datasource.names=m0,m1

spring.shardingsphere.datasource.m0.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m0.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m0.url=jdbc:mysql://localhost:3306/coursedb?serverTimezone=UTC
spring.shardingsphere.datasource.m0.username=root
spring.shardingsphere.datasource.m0.password=root

spring.shardingsphere.datasource.m1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m1.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m1.url=jdbc:mysql://localhost:3306/coursedb2?serverTimezone=UTC
spring.shardingsphere.datasource.m1.username=root
spring.shardingsphere.datasource.m1.password=root
#------------------------分布式序列算法配置
# 生成字符串类型分布式主键。
spring.shardingsphere.rules.sharding.key-generators.user_keygen.type=NANOID
#spring.shardingsphere.rules.sharding.key-generators.user_keygen.type=UUID
# 指定分布式主键生成策略
spring.shardingsphere.rules.sharding.tables.user.key-generate-strategy.column=userid
spring.shardingsphere.rules.sharding.tables.user.key-generate-strategy.key-generator-name=user_keygen
#-----------------------配置读写分离
# 要配置成读写分离的虚拟库
spring.shardingsphere.rules.sharding.tables.user.actual-data-nodes=userdb.user
# 配置读写分离虚拟库 主库一个，从库多个
spring.shardingsphere.rules.readwrite-splitting.data-sources.userdb.static-strategy.write-data-source-name=m0
spring.shardingsphere.rules.readwrite-splitting.data-sources.userdb.static-strategy.read-data-source-names[0]=m1
# 指定负载均衡器
spring.shardingsphere.rules.readwrite-splitting.data-sources.userdb.load-balancer-name=user_lb
# 配置负载均衡器
# 按操作轮训
spring.shardingsphere.rules.readwrite-splitting.load-balancers.user_lb.type=ROUND_ROBIN
# 按事务轮训
#spring.shardingsphere.rules.readwrite-splitting.load-balancers.user_lb.type=TRANSACTION_ROUND_ROBIN
# 按操作随机
#spring.shardingsphere.rules.readwrite-splitting.load-balancers.user_lb.type=RANDOM
# 按事务随机
#spring.shardingsphere.rules.readwrite-splitting.load-balancers.user_lb.type=TRANSACTION_RANDOM
# 读请求全部强制路由到主库
#spring.shardingsphere.rules.readwrite-splitting.load-balancers.user_lb.type=FIXED_PRIMARY

The m0 library is inserted, and the m1 library is read

4. Broadcast table

The broadcast table refers to the table that exists in all fragmented data sources, and the table structure and its data are completely consistent in each database. It is suitable for scenarios where the amount of data is not large and needs to be associated with tables of massive data, such as dictionary tables.
The actual broadcast table understanding is to insert the same table into all libraries

# 打印SQL
spring.shardingsphere.props.sql-show = true
spring.main.allow-bean-definition-overriding = true

# ----------------数据源配置
# 指定对应的库
spring.shardingsphere.datasource.names=m0,m1

spring.shardingsphere.datasource.m0.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m0.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m0.url=jdbc:mysql://localhost:3306/coursedb?serverTimezone=UTC
spring.shardingsphere.datasource.m0.username=root
spring.shardingsphere.datasource.m0.password=root

spring.shardingsphere.datasource.m1.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m1.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m1.url=jdbc:mysql://localhost:3306/coursedb2?serverTimezone=UTC
spring.shardingsphere.datasource.m1.username=root
spring.shardingsphere.datasource.m1.password=root
#------------------------分布式序列算法配置
# 生成字符串类型分布式主键。
spring.shardingsphere.rules.sharding.key-generators.dict_keygen.type=SNOWFLAKE
# 指定分布式主键生成策略
spring.shardingsphere.rules.sharding.tables.dict.key-generate-strategy.column=dictId
spring.shardingsphere.rules.sharding.tables.dict.key-generate-strategy.key-generator-name=dict_keygen
#-----------------------配置读写分离
# 要配置成读写分离的虚拟库
#spring.shardingsphere.rules.sharding.tables.dict.actual-data-nodes=m$->{0..1}.dict_$->{1..2}
spring.shardingsphere.rules.sharding.tables.dict.actual-data-nodes=m$->{
    
    0..1}.dict
# 指定广播表。广播表会忽略分表的逻辑，只往多个库的同一个表中插入数据。
spring.shardingsphere.rules.sharding.broadcast-tables=dict

5. Binding table

The binding table refers to a set of sharding tables with consistent sharding rules. When using a bound table for multi-table association query, you must use the shard key for association, otherwise Cartesian product association or cross-database association will occur, which will affect query efficiency. For example, we create another user information table to demonstrate this situation together with the user table

# 打印SQL
spring.shardingsphere.props.sql-show = true
spring.main.allow-bean-definition-overriding = true

# ----------------数据源配置
# 指定对应的库
spring.shardingsphere.datasource.names=m0

spring.shardingsphere.datasource.m0.type=com.alibaba.druid.pool.DruidDataSource
spring.shardingsphere.datasource.m0.driver-class-name=com.mysql.cj.jdbc.Driver
spring.shardingsphere.datasource.m0.url=jdbc:mysql://localhost:3306/coursedb?serverTimezone=UTC
spring.shardingsphere.datasource.m0.username=root
spring.shardingsphere.datasource.m0.password=root
#------------------------分布式序列算法配置
# 生成字符串类型分布式主键。
spring.shardingsphere.rules.sharding.key-generators.usercourse_keygen.type=SNOWFLAKE
# 指定分布式主键生成策略
spring.shardingsphere.rules.sharding.tables.user_course_info.key-generate-strategy.column=infoid
spring.shardingsphere.rules.sharding.tables.user_course_info.key-generate-strategy.key-generator-name=usercourse_keygen
# ----------------------配置真实表分布
spring.shardingsphere.rules.sharding.tables.user.actual-data-nodes=m0.user_$->{
    
    1..2}
spring.shardingsphere.rules.sharding.tables.user_course_info.actual-data-nodes=m0.user_course_info_$->{
    
    1..2}
# ----------------------配置分片
spring.shardingsphere.rules.sharding.tables.user.table-strategy.standard.sharding-column=userid
spring.shardingsphere.rules.sharding.tables.user.table-strategy.standard.sharding-algorithm-name=user_tbl_alg

spring.shardingsphere.rules.sharding.tables.user_course_info.table-strategy.standard.sharding-column=userid
spring.shardingsphere.rules.sharding.tables.user_course_info.table-strategy.standard.sharding-algorithm-name=usercourse_tbl_alg
# ----------------------配置分表策略
spring.shardingsphere.rules.sharding.sharding-algorithms.user_tbl_alg.type=INLINE
spring.shardingsphere.rules.sharding.sharding-algorithms.user_tbl_alg.props.algorithm-expression=user_$->{
    
    Math.abs(userid.hashCode()%4).intdiv(2) +1}

spring.shardingsphere.rules.sharding.sharding-algorithms.usercourse_tbl_alg.type=INLINE
spring.shardingsphere.rules.sharding.sharding-algorithms.usercourse_tbl_alg.props.algorithm-expression=user_course_info_$->{
    
    Math.abs(userid.hashCode()%4).intdiv(2) +1}
# 指定绑定表
spring.shardingsphere.rules.sharding.binding-tables[0]=user,user_course_info

You can see that after configuring the binding table, only two SQL queries are queried, but if there is no configuration, there will be a Cartesian product query

2. Interpretation of the working principle of ShardingJDBC core

0. Configuration control

Before entering the core of ShardingSphere, ShardingSphere has done a lot of configuration information control. Not only does it parse the configuration information of the application, but ShardingSphere also supports putting the configuration information in a third-party registration center, so that the horizontal expansion of the application layer can be realized.

1. SQL Parser: SQL parsing engine

The parsing process is divided into lexical parsing and grammar parsing. The lexical analyzer is used to disassemble SQL into indivisible atomic symbols called tokens. And according to dictionaries provided by different database dialects, it is classified into keywords, expressions, literals and operators. Then use a syntax parser to convert SQL into an abstract syntax tree
SQL parsing is the core of the entire sub-database and sub-table product, and its performance and compatibility are the most important metrics.

2. SQL Router-SQL routing engine

Match the sharding strategy of the database and table according to the parsing context, and generate a routing path. For SQL with shard keys, it can be divided into single-shard routing (the operator of the shard key is the equal sign), multi-shard routing (the operator of the shard key is IN) and range routing (the operator of the shard key is
The operator for shard keys is BETWEEN). SQL that does not carry a shard key uses broadcast routing.
Fragmentation strategies can usually be built into the database or configured by the user. The built-in scheme of the database is relatively simple, and the built-in sharding strategies can be roughly divided into mantissa modulo, hash, range, label, time, etc. The sharding strategy configured by the user is more flexible, and the composite sharding strategy can be customized according to the needs of the user.

3. SQL Rewriter: SQL optimization engine

First, in terms of data dialects. Apache ShardingSphere provides the ability to translate SQL dialects, whether it can realize automatic conversion between database dialects. For example, users can use the MySQL client to connect to ShardingSphere and send SQL based on the MySQL dialect. ShardingSphere can automatically identify the user protocol and storage node type and automatically complete the SQL dialect conversion to access heterogeneous storage nodes such as PostgreSQL.

4. SQL Executor: SQL execution engine

ShardingSphere uses a set of automated execution engines, which are responsible for sending the real SQL after routing and rewriting to the underlying data source for execution safely and efficiently. It does not simply send SQL directly to the data source for execution through JDBC; nor does it directly put the execution request into the thread pool for concurrent execution. It pays more attention to balancing the creation of data source connections and the consumption of memory usage, as well as maximizing the reasonable use of concurrency and other issues. The goal of the execution engine is to automatically balance resource control and execution efficiency.

Here is mainly to understand the memory limit mode and connection limit mode. easy to understand

• The memory-limited mode only needs to maintain a JDBC connection, and a single thread can complete all data access of a certain real library.
• The connection restriction mode needs to maintain multiple JDBC connections, which requires multiple threads to complete all data access of a certain real library concurrently.

5. Result Merger: Merge the results

• Streaming merge means that every time the data obtained from the result set can be obtained one by one, the correct single piece of data can be returned, which is most consistent with the original way of returning the result set of the database. Traversing, sorting, and streaming grouping are all types of streaming merges. Usually the memory limit mode can use streaming merge, which is more suitable for OLTP scenarios
• Memory merging requires traversing and storing all the data in the result set in memory, and after performing unified grouping, sorting, and aggregation calculations, it is packaged into a data result set that is accessed one by one and returned. . Usually the connection limit mode can use memory merge, which is more suitable for OLAP scenarios.