Application scenario: The database middleware version is ShardingSphere-4.1.1. (Single database) There are 5 tables in the database, namely a, b, c, d, e. Now only a, b, and c are divided into tables, and the other two tables d and e are not divided into tables. My question is: can the sql query that only contains two tables d and e not use the parser of shardingsphere, and want to avoid the parsing of irrelevant tables directly. The current situation is that the parser is gone.
That is, to solve this problem in the case of a single database application, directly avoid the parsing of irrelevant tables. Because the actual use of many sql is with subquery, and these data tables participating in the query are also many partial tables.
Related issues: https://gitee.com/Sharding-Sphere/sharding-sphere/issues/I1GVZ1
ShardingSphere middleware sql parsing interceptor configuration or rewriting to minimize the impact of database middleware, this function has not yet been implemented, here is a note to record, after the solution will continue to share here.
After spending a few days, I also consulted with their development, and it is temporarily impossible to realize this requirement through the control of ShardingSphere.
solution:
Annotations are adopted in the project to control whether to use the data source of ShardingSphere (hosted data source), that is, to introduce multiple data sources to realize multi-data source management. Although ShardingSphere is the top project of Apache, it is not compatible with SQL. Using "multiple data sources + ShardingSphere" should be able to skip many shortcomings of ShardingSphere.
Note: Mysql database master/slave master-slave replication is not described here, the key configuration parameters of Mysql 8.x
master library:
[mysqld]
server-id=1
log-bin=master-mysql-bin
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
lower_case_table_names=1
default_authentication_plugin=mysql_native_passwordslave library:
[mysqld]
server-id=2
log-bin=slave-mysql-bin
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
lower_case_table_names=1
default_authentication_plugin=mysql_native_password
The project uses ShardingSphere's data table splitting and read-write separation functions, which
@DS("sharding")can be switched to the ShardingSphere data source by adding it to the service method that requires table splitting or read-write separation . The open source multi-data source component: dynamic-datasource-spring-boot-starter is used. After integration, the@DSdata source can be switched using annotations, which is very convenient. The pom depends on the following:<!--Multiple data source switching--> <dependency> <groupId>com.baomidou</groupId> <artifactId>dynamic-datasource-spring-boot-starter</artifactId> <version>3.4.1</version> </dependency>
application-datasource.yml
## dynamic-datasource
spring:
datasource:
dynamic:
primary: master #Set the default data source or data source group, the default value is master
strict: false #Strictly match the data source, the default is false. If true does not match the specified data source, an exception will be thrown, and false will use the default data source
datasource:
master:
url: jdbc:mysql://127.0.0.1:33016/shardingds1?useSSL=false&serverTimezone=GMT%2B8&characterEncoding=utf-8
username: root
password: 123456
driver-class-name: com.mysql.cj.jdbc.Driver
slave:
url: jdbc:mysql://127.0.0.1:33017/shardingds2?useSSL=false&serverTimezone=GMT%2B8&characterEncoding=utf-8
username: root
password: 123456
driver-class-name: com.mysql.cj.jdbc.Driver
## jpa print sql in console
jpa:
show-sql: true
properties:
hibernate:
format_sql: true
## mybatis print sql in console
my shoe:
configuration:
log-impl: org.apache.ibatis.logging.stdout.StdOutImpl
Key code configuration:
/**
* Sub-table data source name
*/
private static final String SHARDING_DATASOURCE_NAME = "sharding";
/**
* Dynamic data source configuration items
*/
@Autowired
private DynamicDataSourceProperties properties;
@Lazy
@Resource(name = "shardingDataSource")
AbstractDataSourceAdapter shardingDataSource;
@Bean
public DynamicDataSourceProvider dynamicDataSourceProvider() {
Map<String, DataSourceProperty> datasourceMap = properties.getDatasource();
return new AbstractDataSourceProvider() {
@Override
public Map<String, DataSource> loadDataSources() {
Map<String, DataSource> map = createDataSourceMap(datasourceMap);
// Here, the data source managed by shardingjdbc is handed over to the dynamic-datasource dynamic data source for management
map.put(SHARDING_DATASOURCE_NAME, shardingDataSource);
return map;
}
};
}
@Primary
@Bean
public DataSource dataSource() {
DynamicRoutingDataSource dataSource = new DynamicRoutingDataSource();
dataSource.setPrimary(SHARDING_DATASOURCE_NAME);
dataSource.setStrict(properties.getStrict());
dataSource.setStrategy(properties.getStrategy());
dataSource.setP6spy(properties.getP6spy());
dataSource.setSeata(properties.getSeata());
return dataSource;
}
@DS("data source name key") annotation is used:
@Autowired
private CommonJPAService jpaService;
@Autowired
private CommonJDBCTemplateService jdbcTemplateService;
@Override
@DS("master")
public List getResultByComplexSQL(String sql, int curPage, int rows) {
return this.jpaService.findPageResultBySQL(sql, curPage, rows);
}
@Override
@DS("slave")
public List getSlaveResultByComplexSQL(String sql, int curPage, int rows) {
return this.jpaService.findPageResultBySQL(sql, curPage, rows);
}
@Override
@DS("sharding")
public List getResultBySimpleSQL(String sql, int curPage, int rows) {
return this.jpaService.findPageResultBySQL(sql, curPage, rows);
}
"Can not find owner from table" error log, for debug trace reference:
java.lang.IllegalStateException: Can not find owner from table.
at org.apache.shardingsphere.sql.parser.binder.segment.select.projection.engine.ProjectionsContextEngine.find(ProjectionsContextEngine.java:197)
at org.apache.shardingsphere.sql.parser.binder.segment.select.projection.engine.ProjectionsContextEngine.findShorthandProjection(ProjectionsContextEngine.java:139)
at org.apache.shardingsphere.sql.parser.binder.segment.select.projection.engine.ProjectionsContextEngine.containsItemWithOwnerInShorthandProjections(ProjectionsContextEngine.java:135)
at org.apache.shardingsphere.sql.parser.binder.segment.select.projection.engine.ProjectionsContextEngine.containsItemInShorthandProjection(ProjectionsContextEngine.java:121)
at org.apache.shardingsphere.sql.parser.binder.segment.select.projection.engine.ProjectionsContextEngine.containsProjection(ProjectionsContextEngine.java:105)
at org.apache.shardingsphere.sql.parser.binder.segment.select.projection.engine.ProjectionsContextEngine.getDerivedOrderColumns(ProjectionsContextEngine.java:96)
at org.apache.shardingsphere.sql.parser.binder.segment.select.projection.engine.ProjectionsContextEngine.getDerivedOrderByColumns(ProjectionsContextEngine.java:88)
at org.apache.shardingsphere.sql.parser.binder.segment.select.projection.engine.ProjectionsContextEngine.createProjectionsContext(ProjectionsContextEngine.java:71)
at org.apache.shardingsphere.sql.parser.binder.statement.dml.SelectStatementContext.<init>(SelectStatementContext.java:99)
at org.apache.shardingsphere.sql.parser.binder.SQLStatementContextFactory.getDMLStatementContext(SQLStatementContextFactory.java:103)
at org.apache.shardingsphere.sql.parser.binder.SQLStatementContextFactory.newInstance(SQLStatementContextFactory.java:87)
at org.apache.shardingsphere.underlying.route.DataNodeRouter.createRouteContext(DataNodeRouter.java:99)
at org.apache.shardingsphere.underlying.route.DataNodeRouter.executeRoute(DataNodeRouter.java:89)
at org.apache.shardingsphere.underlying.route.DataNodeRouter.route(DataNodeRouter.java:76)
at org.apache.shardingsphere.underlying.pluggble.prepare.SimpleQueryPrepareEngine.route(SimpleQueryPrepareEngine.java:54)
at org.apache.shardingsphere.underlying.pluggble.prepare.BasePrepareEngine.executeRoute(BasePrepareEngine.java:96)
at org.apache.shardingsphere.underlying.pluggble.prepare.BasePrepareEngine.prepare(BasePrepareEngine.java:83)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingStatement.prepare(ShardingStatement.java:224)
at org.apache.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingStatement.executeQuery(ShardingStatement.java:87)
at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:441)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:396)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:456)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:464)
at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:497)