Shardingjdbc stepped on a pit caused by cranky

Project inside a sub-table uses sharding-jdbc

At that time too tangled with mycat or use sharding-jdbc, but eventually it took sharding-jdbc, for the following reasons:

1. mycat比较重, 相对于sharding-jdbc只需导入jar包就行, mycat还需要部署维护一个中间件服务.由于我们只有一个表需要分表, 直接用轻量级的sharding-jdbc即可.
2. mycat作为一个中间代理服务, 难免有性能损耗
3. 其他组用mycat的时候出现过生产BUG

However sharding-jdbc also continue to be bumpy, we changed from version 4.x versions 2.x, 3.x and from version 4.x down version, each version stepped on a pit (some official, some are due to our project dependent),
eventually was forced to change the times in the past at the source (in fact, the comment line of code).

Today to talk about a pit them - Paging points table

Problem Description

background

CREATE TABLE `order_00` (
  `id` bigint(18) NOT NULL AUTO_INCREMENT COMMENT '逻辑主键',
  `orderId` varchar(32) NOT NULL COMMENT '订单ID',
  `CREATE_TM` datetime DEFAULT NULL COMMENT '订单创建时间',
  PRIMARY KEY (`ID`) USING BTREE,
  UNIQUE KEY `IDX_ORDER_POSTID` (`orderId`) USING BTREE,
  KEY `IDX_ORDER_CREATE_TM` (`CREATE_TM`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT COMMENT='订单表';

CREATE TABLE `order_01` (
  `id` bigint(18) NOT NULL AUTO_INCREMENT COMMENT '逻辑主键',
  `orderId` varchar(32) NOT NULL COMMENT '订单ID',
  `CREATE_TM` datetime DEFAULT NULL COMMENT '订单创建时间',
  PRIMARY KEY (`ID`) USING BTREE,
  UNIQUE KEY `IDX_ORDER_POSTID` (`orderId`) USING BTREE,
  KEY `IDX_ORDER_CREATE_TM` (`CREATE_TM`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT COMMENT='订单表';

CREATE TABLE `order_02` (
  `id` bigint(18) NOT NULL AUTO_INCREMENT COMMENT '逻辑主键',
  `orderId` varchar(32) NOT NULL COMMENT '订单ID',
  `CREATE_TM` datetime DEFAULT NULL COMMENT '订单创建时间',
  PRIMARY KEY (`ID`) USING BTREE,
  UNIQUE KEY `IDX_ORDER_POSTID` (`orderId`) USING BTREE,
  KEY `IDX_ORDER_CREATE_TM` (`CREATE_TM`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPACT COMMENT='订单表';

Suppose there are three or more sub-table, with the logical sub-table orderId modulus, i.e. orderId = 0 is written order_00, orderId = 1 writes order_01, orderId = 2 is written order_02.

NOTE: Why not here and the time-table with orderId do hash, was also controversial.
In theory order form more suitable time to do the score sheet, so the smaller the older the frequency to a data access time, the old score sheet will gradually become cold table, no longer be accessed.
At that time, head of the argument is that, because of this table read and write frequency is higher (and often the scene need to read the main library), with orderId points table can be load balanced and write reading load.
Although it is a bit far-fetched, but it may have merit, I first realized so

You need to be paged queries based orderId or CREATE_TM business, namely sql query about the wording of mybatis as follows:

    <select id="queryPage" parameterType="xxx" resultMap="BaseResultMap">
        select
        <include refid="Base_Column_List"/>
        from ORDER
        <if test="orderId !=null and orderId !='' ">
                AND orderId=#{orderId , jdbcType=VARCHAR}
        </if>
        <if test="createTmStartStr!=null and createTmStartStr!='' ">
                AND create_tm >= concat(#{createTmStartStr, jdbcType=VARCHAR},' 00:00:00')
        </if>
        <if test="createTmEndStr!=null and createTmEndStr!='' ">
                AND create_tm <= concat(#{createTmEndStr, jdbcType=VARCHAR},' 23:59:59')
        </if>
        limit #{page.begin}, #{page.pageSize}
    </select>

Used sharding-jdbc knows, sharding-jdbc a total of 5 partitioning strategy, as shown in FIG. Not used may refer to the official website

In addition Hint fragmentation strategy, other partitioning strategy requires where conditions sql needs to contain fragmentation column (is orderId at our table), it is clear that our business scenario can not guarantee where conditions sql in will be included orderId, so we can only use HintShardingStrategy, will pass the query page to partitioning strategy algorithm, and then determine which table queries, probably the code below

public class OrderHintShardingAlgorithm implements HintShardingAlgorithm {
     public static final String ORDER_TABLE = "ORDER";
    @Override
    public Collection<String> doSharding(Collection<String> availableTargetNames, ShardingValue shardingValue) {
        ListShardingValue<String> listShardingValue = (ListShardingValue<String>) shardingValue;
        List<String> list = Lists.newArrayList(listShardingValue.getValues());
        List<String> actualTable = Lists.newArrayList();
        // 页面上的查询条件会以json的方式传到shardingValue变量中
        String json = list.get(0);
        OrderQueryCondition req = JSON.parseObject(json, OrderQueryCondition.class);
        String orderId = req.getOrderId();
        // 查询条件没有orderId, 要查所有的分表
        if(StringUtils.isEmpty(orderId)){
            // 所有的分表
            for(int i = 0 ; i< 3; i++){
                actualTable.add(ORDER_TABLE + "_0" + i);
            }
        }else{
            // 如果指定了orderId, 只查orderId所在的分表即可
            long tableSuffix = ShardingUtils.getHashInteger(orderId);
            actualTable.add(ORDER_TABLE + "_0" + tableSuffix);
        }
        // actualTable中包含sharding-jdbc实际会查询的表
        return actualTable;
    }
}

This way, if we are to look up words according to orderId, sql sharding-jdbc final execution is (assuming that per Article 10):

select * from ORDER_XX where orderId = ? limit 0 ,10

If the query is not orderId, sql then the final execution is three (assuming that each page 10):

select * from ORDER_00 where create_tm >= ?  and create_tm <= ? limit 0 ,10 ;
select * from ORDER_01 where create_tm >= ?  and create_tm <= ? limit 0 ,10 ;
select * from ORDER_02 where create_tm >= ?  and create_tm <= ? limit 0 ,10 ;

Note that in the case where a plurality of part tables, each table of data taken out of the top 10 (30 total), and then sorted from the former 10, such logic is wrong. Sharding-jdbc give an example , If the following figure:

FIG example, want to get two tables together in accordance with Article 2 and 3 of the score data sorting, 95 and 90 should be. Since only get SQL execution section 2 and 3 from each of the data tables, i.e. table acquired from t_score_0 90 and 80; t_score_0 acquired from the table 85 and 75. Therefore the result of merging, merging can only be acquired from among the 90,80,85 and 75, then merge the results no matter how achieved, can not get the right result.

then what should we do?

sharding-jdbc approach to rewrite our sql, first check out all the data, do merge sort

For example the first two queries

原sql是:
select * from ORDER_00 where create_tm >= ?  and create_tm <= ? limit 10 ,10 ;
select * from ORDER_01 where create_tm >= ?  and create_tm <= ? limit 10 ,10 ;
select * from ORDER_02 where create_tm >= ?  and create_tm <= ? limit 10 ,10 ;
会被改写成:
select * from ORDER_00 where create_tm >= ?  and create_tm <= ? limit 0 ,20 ;
select * from ORDER_01 where create_tm >= ?  and create_tm <= ? limit 0 ,20 ;
select * from ORDER_02 where create_tm >= ?  and create_tm <= ? limit 0 ,20 ;

When the query page 3

原sql是:
select * from ORDER_00 where create_tm >= ?  and create_tm <= ? limit 20 ,10 ;
select * from ORDER_01 where create_tm >= ?  and create_tm <= ? limit 20 ,10 ;
select * from ORDER_02 where create_tm >= ?  and create_tm <= ? limit 20 ,10 ;
会被改写成:
select * from ORDER_00 where create_tm >= ?  and create_tm <= ? limit 0 ,30 ;
select * from ORDER_01 where create_tm >= ?  and create_tm <= ? limit 0 ,30 ;
select * from ORDER_02 where create_tm >= ?  and create_tm <= ? limit 0 ,30 ;

Of course, I am sure you would think that this processing performance will be poor, in fact, and actually be, but sharing-jdbc is on this basis is optimized, is the above-mentioned "merger",
the specific merging process can poke here to view Description official website. long length, I do not paste out here

Isolated probably is the first logical data for all pages, then skip the previous page by streaming, need only take the final page, the ultimate purpose of the tab

Stepped pit

Since sharding-jdbc have been optimized well, so we stepped pit in the end what is it?

Listen to me slowly come

() There is a logic in io.shardingsphere.shardingjdbc.jdbc.core.statement.ShardingPreparedStatement # getResultSet,
the number of sub-table query if only one, then it will not do the logical merger of (check However, even if only a sub-table, sql the limit will be rewritten clause), as shown:

Returning to our business scenario, if the query contains orderId, then, because you can target specific table, so the final points table query need only one.

So the question, because sharding-jdbc our sql to rewrite the limit clause,
behind only the investigation but because a score sheet and do not merge (that is, without skipping the previous page), so in the end regardless of the query the first few pages, the implementation of sql are (assuming that the page size is 10000):

select * from ORDER_XX where orderId = ? limit 0 ,10000
select * from ORDER_XX where orderId = ? limit 0 ,20000
select * from ORDER_XX where orderId = ? limit 0 ,30000
select * from ORDER_XX where orderId = ? limit 0 ,40000
......

This leads to a problem, no matter what I preach the page number, sharding-jdbc I will return to the same data. Obviously this is wrong.

Of course, the cautious friends may find that, because orderId is a unique index, it is certainly only one data, it will never be the presence of the second page of the query.

Normally indeed the case, but in our code inside, there is an old logic: export query results (that is, export data for all pages) When will asynchronously in the background page by page
export, export until all query page or reached the maximum number of times (assuming that the query is 10,000).

Therefore, in accordance with the time orderId exported, because every page returns the same data, so can not judge when is the "lead over all the pages," so the correct result should be only one data, but in the sharding-jdbc perform a million times, to export the same data ten thousand, you say this is not a pit it?

Knowing the problem, then solve is simple but this article does not want to talk about how to solve this problem, but to talk about the problems caused by thinking:

在mysql分表环境下, 如何高效地做分页查询?

Reflections on mysql pagination

limit Optimization

Before discussing the paging performance under sub-table environment, we first look at a single table environment should implement paging.

As we all know, in mysql which implement paging you can just use the limit clause, namely

select * from order  limit (pageNo-1) * pageSize, pageSize

Since achieving mysql inside, offset limit, the scan size is to skip over the offset of data, then take the size of data.
When larger pageNo time, offset will be greater, the greater mysql scanned data, so performance would drop dramatically.

Therefore, paging the first problem to be solved is that when pageNo too large, how to optimize performance.

The first option is an index covering This article describes the program.
Conclusion is to sql rewritten as:

select * from order where id >= (select id from order  limit (pageNo-1) * pageSize, 1) limit pageSize

Using the principle of index covering, is positioned directly before the first minimum id data of this page, and then take the data required.

This can indeed improve performance, but I think it still did not completely solve the problem, because when pageNo too much time, mysql still need to scan many rows to find the smallest id. And those lines are scanned does not make sense.

scroll cursor query

Cursor query is a term elasticSearch inside, but here I do not mean a real scroll query, but learn from thinking inside the ES to implement paging queries mysql.

Is the so-called scroll scroll, page by page, check about the idea as follows:

  1. 查询第1页
     select * from order limit 0, pageSize;

  2. 记录第1页的最大id: maxId
  3. 查询第2页
     select * from order where id > maxId limit pageSize
  4. 把maxId更新为第2页的最大id 
  ... 以此类推

This algorithm can be seen for mysql is no pressure, because each only need to scan pageSize of data will be able to achieve their goals. With respect to the above scheme index covering, can greatly improve query performance.

Of course, it also has its limitations:

1. 性能的提高带来的代价是代码逻辑的复杂度提高. 这个分页逻辑实现起来比较复杂.

2. 这个算法对业务数据是有要求的, 例如id必须是单调递增的,而且查询的结果需要是用Id排序的.
如果查询的结果需要按其他字段(例如createTime)排序, 那就要求createTime也是单调的, 并把算法中的id替换成createTime.
有某些排序的场景下, 这种算法会不适用.

3. 这个算法是需要业务上做妥协的, 你必须说服你的产品经理放弃"跳转到特定页"的功能, 只能通过点击"下一页"来进行翻页.
(这才是scroll的含义, 在手机或平板上,只能通过滚动来翻页,而无法直接跳转到特定页)

Paging query in the sub-table environment

As discussed above, a single table in the environment, efficient paging desired, is relatively simple.

What if in the points table environment, there will be paged to achieve what difference will it make?

As mentioned above, sharding-jdbc has argued, the sub-table query paging environment, if you do not put

select * from ORDER_00 where create_tm >= ?  and create_tm <= ? limit (pageNo-1) * pageSize ,pageSize ;
select * from ORDER_01 where create_tm >= ?  and create_tm <= ? limit (pageNo-1) * pageSize ,pageSize;
select * from ORDER_02 where create_tm >= ?  and create_tm <= ? limit (pageNo-1) * pageSize ,pageSize ;

Rewrite

select * from ORDER_00 where create_tm >= ?  and create_tm <= ? limit 0 , (pageNo-1) * pageSize + pageSize ;
select * from ORDER_01 where create_tm >= ?  and create_tm <= ? limit 0 , (pageNo-1) * pageSize + pageSize;
select * from ORDER_02 where create_tm >= ?  and create_tm <= ? limit 0 , (pageNo-1) * pageSize + pageSize ;

So check out the final data, it is likely not the correct data. Therefore, under the sub-table environment, the above mentioned "Index covering Law" and "Law cursor query" is definitely not applied, because all have to find out data node, and then merge that data is correct.

Therefore, to achieve sub-table paging functionality in the environment, to substantially limit clause of the rewritten.

First look at sharing-jdbc solutions, limit 0 rewritten, (pageNo-1) * pageSize + pageSize and the original limit (pageNo-1) * pageSize , pageSize contrast, query the database side pressure are about the same, because almost all want to be
scanned (pageNo-1) * pageSize line to get to the data. the difference is rewritten sql, the client's memory consumption and network consume larger.

sharding-jdbc clever use of streaming mode and priority queues combination of
the pressure off the client memory consumption, but consumption is still affecting network can not be eliminated.

So really there is no better solution up?

That is definitely yes,
in the industry's problems - "cross-database pages" of four scenarios in this article, the author mentions a "second query Law", very cleverly solved this problem pagination queries.
We can just for reference.

But think carefully about it, there are still some limitations:

1. 当分表数为N时, 查一页数据要执行N*2条sql.(这个无解, 只要分表了就必须这样)

2. 当offset很大的时候, 第一次查询中扫描offset行数据依然会非常的慢, 如果只分表不分库的话, 那么一次查询会在一个库中产生N条慢sql

3. 算法实现起来代码逻辑应该不简单, 如果为了一个分页功能写这么复杂的逻辑, 是不是划不来,
而且后期也不好维护

If the algorithm eggs I saw the original author of the bones here, I would not want to play a little ~ ~

In fact, I want to say is that since the paging query in the sub-table environment is not a perfect solution, or if it costs too much to achieve, then it is not considered to be: under the sub-table environment should not do paging query?

Off-line computing service lossy +

When it comes to the above, in fact, under the sub-table on the environment is not suitable to do the paging function queries.
But the demands on the business does not mean that the cut on the cut, paging functionality is a must in many cases, however, there is usually paged query in order to protect database, remove the pagination, but greater pressure on the database.

So points table and paging can only choose one?

No, all I want to, I want to divide the table, pagination I have to!

But paging feature sub-table environment which do not, but in addition a summary table which do pagination query function.

Probably the scheme is:
1. 正常的业务读写分表 2. 根据具体的业务需求,例如实时计算/离线计算技术(spark, hadoop,hive, kafka等)生成各分表的一张汇总表 3. 分页查询的接口直接查询汇总表
Also note that this program is certainly detrimental to the business, the specific performance:
`` `

Whether calculated or off-line real-time calculation, we can not guarantee real-time query result is definitely a delay
Since it is impossible to include all Summary table of data points, the summary comprising only part of the data must be, for example only, according to the specific service on the scene within one month

总的来说, 就是报表系统的数据由数据仓库系统来生成, 但只能生成用户非要不可的数据,其他的都砍掉.

写这篇总结在找资料的时候, 看到一句话:

In fact, the fundamental purpose of sub-table is to write a load-sharing, rather than being distributed load reading

其实是有一定道理的, 如果读负载过高, 我们可以增加缓存, 增加数据节点等很多方法, 而写负载过高的话, 分表基本就是势在必行了.


从这个理论来说, 分页是一个读操作, 根本就没有必要去读取分表, 从其他地方读取(我们这里是数据仓库)即可

#### 不分表(分区 tidb mongoDb ES)

其实大多数mysql的表都没有必要分表的

在mysql5.5之前, 表数量大概在在500W之后就要进行优化, 在mysql5.5之后, 表数量在1KW到2KW左右才需要做优化.
在这个性能拐点之前, 可以认为mysql是完全有能力扛得住的.当然, 具体还要看qps以及读写冲突等的频率的.

到了性能拐点之后呢?  那就要考虑对mysql的表进行拆分了. 表拆分的手段可以是分表分库, 或者就简单的分区.

基本来说, 分区和分表带来的性能提升是一样的, 
由于分区实际上就可以认为是mysql底层来帮我们实现分表的逻辑了, 所以相对来说分表会比分区带来更高的编码复杂度(分区就根本不用考虑多表分页查询的问题了).
从这个角度来说, 一般的业务直接分区就可以了.

当然, 选择分区还是分表还是需要做一点权衡的:

Data in the table only part of frequently accessed hot data, other infrequently accessed, it is suitable partition table
When the partition table is relatively easy to maintain, can be checked for a single partition, optimization, bulk delete large amounts of data, partition table will be faster than the average table
Partition table may be distributed on different physical devices, so that a plurality of hard disks can be efficiently utilized
If the query does not contain a partition key, then took part in the partition table is not necessarily high efficiency table
If the partition table is definitely hot data, each data are likely to be accessed, it is not suitable partition
If a large amount of data, because the only points 1024 mysql partition, if the data partition of 1024 are more than 10 million it is certainly not suitable for a partition


综上所述, 如果分区表就足够满足我们的话, 那其实就没有必要进行分表了增加编程的复杂度了.


另外, 如果不想将数据表进行拆分, 而表的数据量又的确很大的话, nosql也是一个替代方案. 特别是那些不需要强事务的表操作,
就很适合放在nosql, 从而可以避免编程的复杂度, 同时性能上也没有过多的损耗.

nosql的方案也有很多:

mongoDb
hbase
tidb
elasticSearch
```

Course also be used in combination mysql + nosql embodiment, for example, a conventional read and write operations mysql, paging query ES like walking.

First wrote this today, and have the opportunity to write about mysql nosql