When mysql single table often requires large amount of data sub-library sub-table, Sharding-JDBC Dangdang CAPE source database middleware sub-library sub-table. Sharding-JDBC positioned as a lightweight java framework, the use of direct client database, providing services to jar package form, no proxy proxy layer, no additional deployment, no other dependencies, DBA also need to change the way the original operation and maintenance. This paper describes the integration framework with spring + mybatis use.
1. Preparations
Because it is sub-library sub-table, so it is necessary to create the same table in different databases. Were established t_user0, t_user1, t_user2 three tables in sharding_0, sharding_1 two databases, need to use the SQL statement is as follows:
SET FOREIGN_KEY_CHECKS=0;
DROP TABLE IF EXISTS `t_user_0`;
CREATE TABLE `t_user_0` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`name` varchar(255) NOT NULL,
`age` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
DROP TABLE IF EXISTS `t_user_1`;
CREATE TABLE `t_user_1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`name` varchar(255) NOT NULL,
`age` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
DROP TABLE IF EXISTS `t_user_2`;
CREATE TABLE `t_user_2` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user_id` int(11) NOT NULL,
`name` varchar(255) NOT NULL,
`age` int(11) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;The overall configuration of the project as shown below:
sharding-jdbc dependent need to use is as follows:
<dependency>
<groupId>io.shardingjdbc</groupId>
<artifactId>sharding-jdbc-core</artifactId>
<version>${latest.release.version}</version>
</dependency>2. Detailed Code
Code structure as shown above, Mapper, Service Layer and general SSM projects, without making changes, mainly to increase the algorithm following two files, one is a sub-library strategy, it is a sub-table strategy. Then spring configuration file has been modified slightly, spring-database.xml set is the database connection information. spring-sharding.xml is provided a specific sub-library sub-table information.
spring-database.xml configuration is as follows:
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<list>
<value>classpath:config/resource/jdbc_dev.properties</value>
</list>
</property>
</bean>
<bean name="sharding_0" class="com.alibaba.druid.pool.DruidDataSource" init-method="init" destroy-method="close">
<property name="url" value="${jdbc_url0}" />
<property name="username" value="${jdbc_username0}" />
<property name="password" value="${jdbc_password0}" />
<!-- <property name="driverClass" value="${jdbc_driver0}" /> -->
<!-- 初始化连接大小 -->
<property name="initialSize" value="0" />
<!-- 连接池最大使用连接数量 -->
<property name="maxActive" value="20" />
<!-- 连接池最小空闲 -->
<property name="minIdle" value="0" />
<!-- 获取连接最大等待时间 -->
<property name="maxWait" value="60000" />
<property name="validationQuery" value="${validationQuery}" />
<property name="testOnBorrow" value="false" />
<property name="testOnReturn" value="false" />
<property name="testWhileIdle" value="true" />
<!-- 配置间隔多久才进行一次检测，检测需要关闭的空闲连接，单位是毫秒 -->
<property name="timeBetweenEvictionRunsMillis" value="60000" />
<!-- 配置一个连接在池中最小生存的时间，单位是毫秒 -->
<property name="minEvictableIdleTimeMillis" value="25200000" />
<!-- 打开removeAbandoned功能 -->
<property name="removeAbandoned" value="true" />
<!-- 1800秒，也就是30分钟 -->
<property name="removeAbandonedTimeout" value="1800" />
<!-- 关闭abanded连接时输出错误日志 -->
<property name="logAbandoned" value="true" />
<property name="filters" value="stat" />
</bean>
<bean name="sharding_1" class="com.alibaba.druid.pool.DruidDataSource" init-method="init" destroy-method="close">
<property name="url" value="${jdbc_url1}" />
<property name="username" value="${jdbc_username1}" />
<property name="password" value="${jdbc_password1}" />
<!-- <property name="driverClass" value="${jdbc_driver1}" /> -->
<!-- 初始化连接大小 -->
<property name="initialSize" value="0" />
<!-- 连接池最大使用连接数量 -->
<property name="maxActive" value="20" />
<!-- 连接池最小空闲 -->
<property name="minIdle" value="0" />
<!-- 获取连接最大等待时间 -->
<property name="maxWait" value="60000" />
<property name="validationQuery" value="${validationQuery}" />
<property name="testOnBorrow" value="false" />
<property name="testOnReturn" value="false" />
<property name="testWhileIdle" value="true" />
<!-- 配置间隔多久才进行一次检测，检测需要关闭的空闲连接，单位是毫秒 -->
<property name="timeBetweenEvictionRunsMillis" value="60000" />
<!-- 配置一个连接在池中最小生存的时间，单位是毫秒 -->
<property name="minEvictableIdleTimeMillis" value="25200000" />
<!-- 打开removeAbandoned功能 -->
<property name="removeAbandoned" value="true" />
<!-- 1800秒，也就是30分钟 -->
<property name="removeAbandonedTimeout" value="1800" />
<!-- 关闭abanded连接时输出错误日志 -->
<property name="logAbandoned" value="true" />
<property name="filters" value="stat" />
</bean>spring-sharding.xml configuration is shown below, can be seen from the figure mainly policy strategy and sub-library sub-table specifies required, and the package, a specific sub-library sub-table need to write policy.
<!-- 配置好dataSourceRulue,即对数据源进行管理 -->
<bean id="dataSourceRule" class="com.dangdang.ddframe.rdb.sharding.api.rule.DataSourceRule">
<constructor-arg>
<map>
<entry key="sharding_0" value-ref="sharding_0"/>
<entry key="sharding_1" value-ref="sharding_1"/>
</map>
</constructor-arg>
</bean>
<!-- 对t_user表的配置，进行分库配置，逻辑表名为t_user，每个库有实际的三张表 -->
<bean id="userTableRule" class="com.dangdang.ddframe.rdb.sharding.api.rule.TableRule">
<constructor-arg value="t_user" index="0"/>
<constructor-arg index="1">
<list>
<value>t_user_0</value>
<value>t_user_1</value>
<value>t_user_2</value>
</list>
</constructor-arg>
<constructor-arg index="2" ref="dataSourceRule"/>
<constructor-arg index="3" ref="userDatabaseShardingStrategy"/>
<constructor-arg index="4" ref="userTableShardingStrategy"/>
</bean>
<!-- t_user分库策略 -->
<bean id="userDatabaseShardingStrategy" class="com.dangdang.ddframe.rdb.sharding.api.strategy.database.DatabaseShardingStrategy">
<constructor-arg index="0" value="user_id"/>
<constructor-arg index="1">
<bean class="com.study.dangdang.sharding.jdbc.algorithm.UserSingleKeyDatabaseShardingAlgorithm" />
</constructor-arg>
</bean>
<!-- t_user 分表策略 -->
<bean id="userTableShardingStrategy" class="com.dangdang.ddframe.rdb.sharding.api.strategy.table.TableShardingStrategy">
<constructor-arg index="0" value="user_id"/>
<constructor-arg index="1">
<bean class="com.study.dangdang.sharding.jdbc.algorithm.UserSingleKeyTableShardingAlgorithm" />
</constructor-arg>
</bean>
<!-- 构成分库分表的规则 传入数据源集合和每个表的分库分表的具体规则 -->
<bean id="shardingRule" class="com.dangdang.ddframe.rdb.sharding.api.rule.ShardingRule">
<constructor-arg index="0" ref="dataSourceRule"/>
<constructor-arg index="1">
<list>
<ref bean="userTableRule"/>
</list>
</constructor-arg>
</bean>
<!-- 对datasource进行封装 -->
<bean id="shardingDataSource" class="com.dangdang.ddframe.rdb.sharding.api.ShardingDataSource">
<constructor-arg ref="shardingRule"/>
</bean>Next, look at the specific sub-sub-table files and library files:
Part table file UserSingleKeyTableShardingAlgorithm code as follows:
public class UserSingleKeyTableShardingAlgorithm implements SingleKeyTableShardingAlgorithm<Integer>{
/**
* sql 中 = 操作时，table的映射
*/
public String doEqualSharding(Collection<String> tableNames, ShardingValue<Integer> shardingValue) {
for (String each : tableNames) {
if (each.endsWith(shardingValue.getValue() % 3 + "")) {
return each;
}
}
throw new IllegalArgumentException();
}
/**
* sql 中 in 操作时，table的映射
*/
public Collection<String> doInSharding(Collection<String> tableNames, ShardingValue<Integer> shardingValue) {
Collection<String> result = new LinkedHashSet<String>(tableNames.size());
for (Integer value : shardingValue.getValues()) {
for (String tableName : tableNames) {
if (tableName.endsWith(value % 3 + "")) {
result.add(tableName);
}
}
}
return result;
}
/**
* sql 中 between 操作时，table的映射
*/
public Collection<String> doBetweenSharding(Collection<String> tableNames,
ShardingValue<Integer> shardingValue) {
Collection<String> result = new LinkedHashSet<String>(tableNames.size());
Range<Integer> range = (Range<Integer>) shardingValue.getValueRange();
for (Integer i = range.lowerEndpoint(); i <= range.upperEndpoint(); i++) {
for (String each : tableNames) {
if (each.endsWith(i % 3 + "")) {
result.add(each);
}
}
}
return result;
}
}Sub-library file UserSingleKeyDatabaseShardingAlgorithm code as follows:
public class UserSingleKeyDatabaseShardingAlgorithm implements SingleKeyDatabaseShardingAlgorithm<Integer>{
/**
* sql 中关键字 匹配符为 =的时候，表的路由函数
*/
public String doEqualSharding(Collection<String> availableTargetNames, ShardingValue<Integer> shardingValue) {
for (String each : availableTargetNames) {
if (each.endsWith(shardingValue.getValue() % 2 + "")) {
return each;
}
}
throw new IllegalArgumentException();
}
/**
* sql 中关键字 匹配符为 in 的时候，表的路由函数
*/
public Collection<String> doInSharding(Collection<String> availableTargetNames, ShardingValue<Integer> shardingValue) {
Collection<String> result = new LinkedHashSet<String>(availableTargetNames.size());
for (Integer value : shardingValue.getValues()) {
for (String tableName : availableTargetNames) {
if (tableName.endsWith(value % 2 + "")) {
result.add(tableName);
}
}
}
return result;
}
/**
* sql 中关键字 匹配符为 between的时候，表的路由函数
*/
public Collection<String> doBetweenSharding(Collection<String> availableTargetNames,
ShardingValue<Integer> shardingValue) {
Collection<String> result = new LinkedHashSet<String>(availableTargetNames.size());
Range<Integer> range = (Range<Integer>) shardingValue.getValueRange();
for (Integer i = range.lowerEndpoint(); i <= range.upperEndpoint(); i++) {
for (String each : availableTargetNames) {
if (each.endsWith(i % 2 + "")) {
result.add(each);
}
}
}
return result;
}
}We can see the sql statement for different keywords have different strategies, the main achievement of doEqualSharding, doInSharding, doBetweenSharding three methods, you can see the source code that need to implement different interfaces for different sub-library sub-table rules.
3. Test
Test code is as follows:
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "classpath*:config/spring/spring-database.xml",
"classpath*:config/spring/spring-sharding.xml" })
public class ShardingJdbcMybatisTest {
@Resource
public UserService userService;
@Test
public void testUserInsert() {
User u = new User();
for(int i=0;i<100;i++){
u.setUserId(26+i);
u.setAge(2+i);
u.setName("war3"+i);
Assert.assertEquals(userService.insert(u), true);
}
}
}查看数据库可以看到100个用例均匀分布到了数据库的表中。
参考文献
http://blog.csdn.net/clypm/article/details/54378523
http://shardingjdbc.io/docs/00-overview/
https://my.oschina.net/editorial-story/blog/888650
http://www.infoq.com/cn/news/2016/01/sharding-jdbc-dangdang
