background
The relational database itself is relatively easy to become a system bottleneck, and the storage capacity, number of connections, and processing capacity of a single machine are limited. When the data volume of a single table reaches 1000W or 100G, due to the large number of query dimensions, even if the database is added and the index is optimized, the performance will still drop severely when many operations are performed. At this time, it is necessary to consider segmentation. The purpose of segmentation is to reduce the burden on the database and shorten the query time.
According to its segmentation type, it can be divided into two ways: vertical (longitudinal) segmentation and horizontal (horizontal) segmentation.
1. Vertical division
Vertical sub-database: According to business coupling, different tables with low correlation are stored in different databases. The approach is similar to the splitting of a large system into multiple small systems, which are divided independently by business classification. Similar to the "microservice governance" approach, each microservice uses a separate database.
Vertical table splitting: It is based on the "columns" in the database. If a table has more fields, you can create an extended table, and split the fields that are not frequently used or with larger field lengths into the extended table. In the case of a lot of fields, through "large table split small table", it is easier to develop and maintain, and it can also avoid additional performance overhead.
When an application is difficult to fine-grained vertical segmentation, or the number of rows of data after segmentation is huge, there is a single library read/write, storage performance bottleneck, then horizontal segmentation is required.
2. Horizontal segmentation
In-database sub-table: only solves the problem of excessive data volume in a single table, but does not distribute the tables to the libraries of different machines, so it is not very helpful to reduce the pressure on the MySQL database. Everyone is still competing for the same physical The CPU, memory, and network IO of the machine are best solved by sub-database and table.
Sub-database and sub-table: According to the internal logical relationship of the data, the same table is distributed to multiple databases or multiple tables under different conditions. Each table contains only a part of the data, so that the amount of data in a single table is reduced. Distributed effect.
The advantages of horizontal segmentation:
there is no performance bottleneck caused by excessive data volume and high concurrency in a single database, system stability and load capacity
are improved, and the data volume of the table is small, and the single SQL execution efficiency is high, which naturally reduces the burden on the CPU
application side Small transformation, no need to split business modules
. Disadvantages of horizontal segmentation:
cross- shard transaction consistency is difficult to ensure
cross-database join related query performance is poor,
data expansion is difficult and the amount of maintenance is great
Focus on the JSPGenFire implementation method of horizontal segmentation, and demonstrate the sub-tables in the library and the sub-tables in the library.
Introduction to JSPGenFire
JSPGenFire is positioned as a lightweight Java database operation framework, providing services in the form of jar packages at the JDBC layer of Java without additional deployment and dependencies. Data segmentation is to store data in multiple databases, so that the amount of data in a single database is reduced, and the performance problems of a single database are alleviated by expanding the number of hosts, so as to achieve the purpose of improving database operation performance.
Key concept
1. Logical table
A general term for tables with the same logic and data structure of a horizontally split database (table). Example: The order data is split into 10 tables according to the mantissa of the primary key, which are jspgen_login_0 to jspgen_login_9, and their logical table is named jspgen_login.
2. Physical table
A physical table that actually exists in a fragmented database. That is, jspgen_login_0 to jspgen_login_9 in the previous example.
3. Data node
The smallest unit of data fragmentation. It consists of data source name and data table, for example: ds_0. jspgen_login_0.
Sharding strategy
After horizontal splitting, the same table will appear in multiple databases/tables, and the content of each database/table is different. Several typical data fragmentation rules are:
1. According to the numerical range
Divide according to time interval or ID interval. For example: Distribute the data of different months or even days to different libraries by date; divide the records with id 1~9999 into the first library, divide the records with 10000~20000 into the second library, and so on.
2. Take the modulus based on the value
Generally, a modular splitting method is adopted. For example, the login table is divided into 2 libraries according to the id field, the remainder of 0 is placed in the first library, the remainder of 1 is placed in the second library, and so on .
Application in the project
1. Data structure
-- ----------------------------
-- Table structure for jspgen_login
-- ----------------------------
DROP TABLE IF EXISTS `jspgen_login`;
CREATE TABLE `jspgen_login` (
`id` varchar(32) NOT NULL,
`uid` varchar(32) default NULL,
`name` varchar(50) default NULL,
`score` int(20) default '0',
`time` bigint(13) default NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-- ----------------------------
-- Table structure for jspgen_login_0
-- ----------------------------
DROP TABLE IF EXISTS `jspgen_login_0`;
CREATE TABLE `jspgen_login_0` (
`id` varchar(32) NOT NULL,
`uid` varchar(32) default NULL,
`name` varchar(50) default NULL,
`score` int(20) default '0',
`time` bigint(13) default NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-- ----------------------------
-- Table structure for jspgen_login_1
-- ----------------------------
DROP TABLE IF EXISTS `jspgen_login_1`;
CREATE TABLE `jspgen_login_1` (
`id` varchar(32) NOT NULL,
`uid` varchar(32) default NULL,
`name` varchar(50) default NULL,
`score` int(20) default '0',
`time` bigint(13) default NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-- ----------------------------
-- Table structure for jspgen_user
-- ----------------------------
DROP TABLE IF EXISTS `jspgen_user`;
CREATE TABLE `jspgen_user` (
`id` varchar(32) NOT NULL,
`name` varchar(50) default NULL,
`time` bigint(13) default NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;2. Configuration file
3. Strategy implementation
package fire.sub.provider;
import java.util.Map;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import fire.sub.SubProvider;
/**
* 数据库分库实现
* @author JSPGen
* @copyright (c) JSPGen.com
* @created 2020年03月
* @email [email protected]
* @address www.jspgen.com
*/
public class DbProvider implements SubProvider {
// 日志工具
private static Logger logger = LoggerFactory.getLogger(DbProvider.class);
/**
* 获取名称
* @return String
*/
public String getSuffix(Map<String, Object> paramMap){
String name = "";
try {
name = name + (Integer.parseInt((String) paramMap.get("id"))%2);
logger.info("db_name:" + name);
} catch (Exception e) {
e.printStackTrace();
}
return name;
}
}package fire.sub.provider;
import java.util.Map;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import fire.sub.SubProvider;
/**
* 数据库分表实现
* @author JSPGen
* @copyright (c) JSPGen.com
* @created 2020年03月
* @email [email protected]
* @address www.jspgen.com
*/
public class TableProvider implements SubProvider {
// 日志工具
private static Logger logger = LoggerFactory.getLogger(TableProvider.class);
/**
* 获取名称
*
* @return String
*/
public String getSuffix(Map<String, Object> paramMap){
String name = "_";
try {
/*
// 一天一张表
Long time = (Long) paramMap.get("time");
if(time == null) time = Dates.getTimeMillis();
String year = Dates.getDateTime(time, "yyyy");
String mon = Dates.getDateTime(time, "MM");
String day = Dates.getDateTime(time, "dd");
name = name + year + mon + day;
*/
name = name + (Integer.parseInt((String) paramMap.get("id"))%2);
logger.info("table_name:" + name);
} catch (Exception e) {
e.printStackTrace();
}
return name;
}
}4. Data test
package jspgen.action;
import java.util.HashMap;
import java.util.Map;
import fire.FireAccess;
import fire.FireBuild;
import grapes.Dates;
import grapes.Grapes;
/**
* Action类：分库分表测试
* @author JSPGen
* @copyright (c) JSPGen.com
* @created 2020年03月
* @email [email protected]
* @address www.jspgen.com
*/
public class DemoSubAction extends Action {
/**
* 默认方法
*/
@Override
public String execute() {
return text("分库分表测试");
}
// 分库测试
public String user() {
long start = Dates.getTimeMillis(); // 开始时间
FireAccess fa = FireBuild.getInstance().getAccess("JSPGen");
String sql = "insert into `"+FireAccess.getTable("user")+"` (`id`,`name`,`time`) values (:id, :name, :time)";
Map<String, Object> paramMap = null;
for (int i =1 ; i< 10 ; i++){
paramMap = new HashMap<String, Object>();
paramMap.put("id", i+"");
paramMap.put("name", 100+i);
paramMap.put("time", Dates.getTimeMillis());
fa.createSQL(sql).setParameter(paramMap).executeUpdate();
}
fa.close();
long end = Dates.getTimeMillis(); // 结束时间
long count = end-start;
return text("总共用了：" + Dates.getUnitTime(count, true) + " ("+count+"毫秒)");
}
// 分库查询测试
public String userfind() {
FireAccess fa = FireBuild.getInstance().getAccess("JSPGen");
String sql = "select * from `"+FireAccess.getTable("user")+"` where `id`=:id";
Map<String, Object> paramMap = new HashMap<String, Object>();
//paramMap.put("id", getParameter("id"));
paramMap.put("id", Grapes.rand(1,9)+"");
fa.createSQL(sql).setParameter(paramMap);
Map<String, Object> map = fa.unlist();
fa.close();
return text(map.toString());
}
// 分表测试
public String login() {
long start = Dates.getTimeMillis(); // 开始时间
FireAccess fa = FireBuild.getInstance().getAccess("JSPGen");
String sql = "insert into `"+FireAccess.getTable("login")+"` (`id`,`name`,`time`) values (:id, :name, :time)";
Map<String, Object> paramMap = null;
for (int i =1 ; i< 10 ; i++){
paramMap = new HashMap<String, Object>();
paramMap.put("id", i+"");
paramMap.put("name", 100+i);
paramMap.put("time", Dates.getTimeMillis());
fa.createSQL(sql).setParameter(paramMap).executeUpdate();
}
fa.close();
long end = Dates.getTimeMillis(); // 结束时间
long count = end-start;
return text("总共用了：" + Dates.getUnitTime(count, true) + " ("+count+"毫秒)");
}
// 分表查询测试
public String loginfind() {
FireAccess fa = FireBuild.getInstance().getAccess("JSPGen");
String sql = "select * from `"+FireAccess.getTable("login")+"` where `id`=:id";
Map<String, Object> paramMap = new HashMap<String, Object>();
//paramMap.put("id", getParameter("id"));
paramMap.put("id", Grapes.rand(1,9)+"");
fa.createSQL(sql).setParameter(paramMap);
Map<String, Object> map = fa.unlist();
fa.close();
return text(map.toString());
}
}5. Test log
6. Data recording
A. When the database is not divided into the table 
B, after the database is divided into the table 
C, after the database is divided into the table
Write at the end
Not all tables need to be segmented, mainly depends on the growth rate of data. After the segmentation, the complexity of the business will be increased to some extent. In addition to the storage and query of the data carried by the database, it is also one of its important tasks to assist the business to better fulfill its requirements.
It is not recommended to use the sub-database and sub-table as a last resort to avoid "over design" and "premature optimization". Before sub-database sub-table, do not divide for sub-division, first try to do what you can, such as: upgrade hardware, upgrade the network, read and write separation, index optimization, etc. When the amount of data reaches the bottleneck of a single table, consider sub-database sub-table.