Sharding-JDBC Data Sharding Practice of ShardingSphere

Offer arrives, dig friends to pick up! I am participating in the 2022 Spring Recruitment Check-In Event, click to view the event details .

Introduction

ShardingSphere is an ecosystem of open source distributed database middleware solutions. It consists of three independent products: Sharding-JDBC, Sharding-Proxy and Sharding-Sidecar (planned). They all provide standardized data sharding, distributed transactions, and database governance functions, which can be applied to various application scenarios such as Java isomorphism, heterogeneous languages, and cloud native.

ShardingSphere is positioned as a relational database middleware, which aims to fully and reasonably utilize the computing and storage capabilities of relational databases in distributed scenarios, rather than implementing a new relational database. It is coexisting with NoSQL and NewSQL rather than mutually exclusive. NoSQL and NewSQL are the frontiers of new technology exploration, looking to the future and embracing changes are highly recommended. Conversely, you can also look at the problem with another way of thinking, look to the future, pay attention to the unchanged things, and then grasp the essence of things. Relational databases still occupy a huge market today, and are the cornerstone of each company's core business. It will be difficult to shake it in the future. At this stage, we are more concerned about the increment based on the original foundation, rather than the subversion.

ShardingSphere has graduated from the Apache incubator on April 16, 2020 and has become an Apache top-level project.

Quick start

create table

It should be noted here that the field should not have keyword subclasses, and it should be as long as possible.

create database `cloud-order`;

use `cloud-order`;

create database if not exists `cloud-order` default charset utf8mb4 collate utf8mb4_general_ci;

create table t_order_1 (
  `oid` bigint(20) unsigned not null auto_increment comment '主键id',
  `order_code` varchar(64) not null default '' comment '订单号',
  primary key (`oid`),
  key `idx_order_code` (`order_code`)
) comment '订单表';

create table t_order_2 (
  `oid` bigint(20) unsigned not null auto_increment comment '主键id',
  `order_code` varchar(64) not null default '' comment '订单号',
  primary key (`oid`),
  key `idx_order_code` (`order_code`)
)comment '订单表';

复制代码

add dependencies

Let me tell you the basics of the project:

• spring boot 2.6.5
• spring cloud 2021.0.1
• mybatis-spring-boot-starter 2.2.2
• sharding-jdbc-spring-boot-starter 4.1.1

At present, they are all relatively new versions, so there will definitely be a lot of pitfalls. The specific configuration is as follows (because my configuration is a modification of the previous project, it is not concise enough, let’s make do with it):

plugins {
    id 'java'
    id 'org.springframework.boot' version '2.6.5'
    id 'io.spring.dependency-management' version '1.0.11.RELEASE'
}

group 'io.zhengsh'
version = '0.0.1-SNAPSHOT'
sourceCompatibility = 1.8

repositories {
    mavenLocal()
    maven { url 'https://maven.aliyun.com/nexus/content/groups/public/' }
    mavenCentral()
}

ext {
    set('springCloudVersion', "2021.0.1")
}

configurations {
    compile.exclude module: 'spring-boot-starter-tomcat'
}


dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.boot:spring-boot-starter-undertow'
    implementation 'org.springframework.boot:spring-boot-starter-actuator'

    implementation 'org.springframework.cloud:spring-cloud-starter-netflix-eureka-client'
    implementation 'org.springframework.cloud:spring-cloud-starter-openfeign'
    implementation 'org.springframework.cloud:spring-cloud-starter-circuitbreaker-resilience4j'

    // mysql
    implementation 'mysql:mysql-connector-java'
    // mybatis
    implementation 'org.mybatis.spring.boot:mybatis-spring-boot-starter:2.2.2'
    // shardingsphere
    implementation 'org.apache.shardingsphere:sharding-jdbc-spring-boot-starter:4.1.1'

    testImplementation 'org.springframework.boot:spring-boot-starter-test'

    compileOnly 'org.projectlombok:lombok:1.18.22'
    annotationProcessor 'org.projectlombok:lombok:1.18.22'

    testCompileOnly 'org.projectlombok:lombok:1.18.22'
    testAnnotationProcessor 'org.projectlombok:lombok:1.18.22'
}


dependencyManagement {
    imports {
        mavenBom "org.springframework.cloud:spring-cloud-dependencies:${springCloudVersion}"
    }
}


tasks.named('test') {
    useJUnitPlatform()
}

复制代码

Add YML configuration

Add YML configuration, mainly to have the following configuration

• Setting the data source is to configure which libraries
• Set up data tables, tables participating in sharding
• To set the strategy of sub-tables, I use the snowflake algorithm to generate id, and specify the data location by modulo.

spring:
  shardingsphere:
    # 设置数据源
    datasource:
      names: o1
      o1:
        type: com.zaxxer.hikari.HikariDataSource
        driver-class-name: com.mysql.cj.jdbc.Driver
        jdbc-url: jdbc:mysql://127.0.0.1:3306/cloud-order?useUnicode=true&zeroDateTimeBehavior=convertToNull&useSSL=false
        username: root
        password: root123
    props:
      # 打印 sql 
      sql.show: true
    sharding:
      default-data-source-name: o1
      tables:
        t_order:
          # 指定需要分的表
          # 表达式， 实际数据节点： 根据上一个节点找到此值， {0..1}为groovy语言，$会替换成{0..1}的一个值，数据库表是: t_order_1 , t_order_2
          # 这个配置是告诉sharding有多少个表
          actual-data-nodes: o1.t_order_$->{1..2}
          key-generator:
            # 指定主键
            column: oid
						# 生成方式: 雪花算法
            type: snowflake
            props:
              worker:
                id: 1
          # 分表策略
          table-strategy:
            inline:
							# 配置sharding的计算列
              sharding-column: oid
              # 配置sharding的表达式,对应的id必须和sharding-column的值对应，否则报错
              algorithm-expression: t_order_$->{oid % 2 + 1}
复制代码

Java code

1. Entity class

public class OrderEntity {

    private Long oid;

    private String orderCode;

    // 此处省略 get、set 方法
}
复制代码

2. Mapper file

@Mapper
public interface OrderEntityMapper {


    @Insert("insert into o1.t_order (`order_code`) values(#{orderCode})")
    @Options(useGeneratedKeys = true, keyProperty = "oid", keyColumn = "oid")
    int insertSelective(OrderEntity record);

    @Select("select * from o1.t_order where oid = #{oid}")
    OrderEntity selectOne(@Param("oid") Long oid);
}
复制代码

3. Unit testing

@Test
public void insertOrder() {
  for (int i = 0; i < 10; i++) {
    OrderEntity order = new OrderEntity();
    order.setOrderCode("OR000" + i);
    int row = orderEntityMapper.insertSelective(order);
    Assert.assertTrue(row > 0);
  }
}

@Test
public void selectOne() {
  OrderEntity order1 = orderEntityMapper.selectOne(
715721929377452033L);
  Assert.assertNotNull(order1);
}
复制代码

Results of the

1. Insert data

2. Query results

common problem

health check error

The core is the yml configuration. One of the problems I encountered before was that the health detection was wrong and could not be configured, so the configuration in the Java config method was used to avoid this startup error.

Add the Java config configuration as follows:

@Configuration
public class DataSourceConfig {

    @Bean
    public HealthIndicator dbHealthIndicator(DataSource dataSource) {
        DataSourceHealthIndicator indicator = new DataSourceHealthIndicator(dataSource);
        indicator.setQuery("select 1");
        return indicator;
    }
}
复制代码

null pointer exception

When MyBatis queried, an error was reported and a null pointer was reported. When querying other tables, no error is reported, but this table reports an error, as shown below:

### Cause: java.lang.NullPointerException
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
	... 75 common frames omitted
Caused by: java.lang.NullPointerException: null
	at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:18)
	at org.apache.shardingsphere.sql.parser.mysql.visitor.impl.MySQLDMLVisitor.createProjection(MySQLDMLVisitor.java:446)
// 省略部分异常  
复制代码

At first, I thought it was the reason for the sub-table, but later I checked the information on the Internet, and it was caused by the keywords in the table. When viewing the exception information, ignore the top prompt:

line 1:176 mismatched input 'order' expecting {..... 忽略一些关键字}
复制代码

invoke method mod() on null object exception

Later I encountered this problem again Cannot invoke method mod() on null object] with root causeexception , the complete error message is as follows:

### SQL: insert into o1.t_order (`order_code`, `code`) values(?, ?)
### Cause: java.lang.NullPointerException: Cannot invoke method mod() on null object
	at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:96)
	at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:441)
	at com.sun.proxy.$Proxy112.insert(Unknown Source)
	at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:272)
// 省略部分异常
复制代码

Cause of the problem: The fields in the shard key and the shard strategy are inconsistent