This article is suitable for readers: novices who need to transform from a single database and a single table to multiple databases and multiple tables.
This article mainly describes the factors that need to be considered in the process of sub-library and sub-meter transformation, and the corresponding solutions, as well as those pits that have been stepped on.
I. Introduction
Since we want to sub-databases and sub-tables, there must always be a motivation to do things. So, before doing it, we must first understand the following two questions.
1 What is the sub-database and sub-table?
In fact, it is literal and easy to understand:
- Sub-database: The process of splitting a single database into multiple databases, and the data is scattered among multiple databases.
- Split table: The process of splitting a single table into multiple tables, and the data is scattered in multiple tables.
2 Why do we need to sub-database and sub-table?
Keywords: Improve performance, increase usability.
From a performance point of view
As the amount of data in a single database becomes larger and the query QPS of the database becomes higher and higher, correspondingly, the time required for reading and writing to the database becomes more and more. The read and write performance of the database may become a bottleneck for business development. Correspondingly, it is necessary to optimize the database performance. In this article, we only discuss optimization at the database level, and do not discuss application-level optimization methods such as caching.
If the query QPS of the database is too high, you need to consider splitting the database and sharing the connection pressure of a single database by splitting the database. For example, if the query QPS is 3500, assuming that a single database can support 1000 connections, then you can consider splitting it into 4 databases to spread the query connection pressure.
If the amount of data in a single table is too large, when the amount of data exceeds a certain level, whether it is for data query or data update, after the traditional optimization methods at the pure database level such as index optimization, there may still be performance problems. This is the qualitative change caused by the quantitative change. At this time, we need to change our thinking to solve the problem, such as solving the problem from the source of data production and data processing. Since the amount of data is large, then we will divide and conquer and break it into zero. This results in a split table, which splits the data into multiple tables according to certain rules to solve the access performance problem that cannot be solved in a single table environment.
From the point of view of usability
If an accident occurs to a single database, it is very likely that all data will be lost. Especially in the cloud era, many databases are running on virtual machines. If the virtual machine/host machine has an accident, it may cause irreparable losses. Therefore, in addition to the traditional Master-Slave, Master-Master and other deployment levels to solve the reliability problem, we can also consider solving this problem from the data split level.
Here we take the database downtime as an example:
- In the case of a single database deployment, if the database is down, the impact of the failure is 100%, and recovery may take a long time.
- If we split into two libraries and deploy them on different machines, and one of the libraries is down, the impact of the failure will be 50%, and 50% of the data can continue to be served.
- If we split into 4 libraries and deploy them on different machines, and one of the libraries is down, the impact of the failure will be 25%, and 75% of the data can continue to be served, and the recovery time will be very short. .
Of course, we can't unrestrictedly dismantle the library. This is also a way to improve performance and availability at the expense of storage resources. After all, resources are always limited.
How to divide the database and table
1 Sub-library? Sub-table? Or is it both sub-database and sub-table?
Judging from the information learned in the first part, the sub-database sub-table scheme can be divided into the following three types:
2 How to choose our own segmentation plan?
If you need to divide the table, how many tables are appropriate?
Since all technologies serve the business, we first review the business background from the data aspect.
For example, our business system is designed to solve members' consultation demands and serve members through our XSpace customer service platform system. Currently, we mainly use synchronized offline work order data as our data source to build our own data.
Assuming that each offline ticket will generate a corresponding member's consulting question (we referred to as: question sheet), if:
- Online channels: 3w chat sessions are generated every day, assuming that 50% of the sessions will generate an offline work order, then 3w * 50% = 1.5w work orders can be generated every day;
- Hotline channel: 2.5w calls are generated every day, assuming that 80% of the calls will generate a work order, then 2.5w * 80% = 2w calls per day can be generated;
- Offline channels: Assume that offline channels directly generate 3w pens per day;
A total of 1.5w + 2w + 3w = 6.5w pens/day
Taking into account the new business scenarios that may continue to be covered in the future, it is necessary to reserve some expansion space in advance. Here we assume that 8w question sheets are generated every day.
In addition to the question sheet, there are two other commonly used business sheets: user operation log sheet and user-submitted form data sheet.
Among them, each question order will generate multiple user operation logs. According to historical statistics, it can be seen that on average, each question order will generate about 8 operation logs. We reserve a part of the space, assuming that each question order generates about 10 user operation logs.
If the design service life of the system is 5 years, then the amount of questionnaire data is approximately = 5 years, 365 days/year 8w/day = 146 million, then the estimated number of tables is as follows:
- The question sheet needs: 146 million/500w = 29.2 sheets, we will divide it according to 32 sheets;
- The operation log needs: 32 10 = 320 tables, we will divide it according to 32 16 = 512 tables.
If you need to divide the library, how many libraries are appropriate?
In addition to the usual business peak reading and writing QPS, when sub-databases should be considered, for example, the peaks that may be reached during the Double 11 promotion period must be estimated in advance.
According to our actual business scenario, the data query source of the question sheet mainly comes from the homepage of Ali customer service, Xiaomi. Therefore, it can be evaluated based on historical QPS, RT and other data. Assuming that we only need 3500 database connections, if a single database can bear up to 1000 database connections, then we can split it into 4 databases.
3 How to segment the data?
According to industry practice, the split is usually performed in two ways, horizontal splitting and vertical splitting. Of course, some complex business scenarios may also choose a combination of the two.
(1) Horizontal division
This is a horizontal segmentation based on business dimensions, such as common segmentation based on membership dimensions, and different member-related data is scattered in different database tables according to certain rules. Since our business scenario decision is to read and write data from the perspective of members, we choose to split the database in a horizontal manner.
(2) Vertical division
Vertical segmentation can be simply understood as splitting different fields of a table into different tables.
For example: Suppose there is a small e-commerce business, and the product information, buyer and seller information, and payment information related to an order are all placed in a large table. You can consider using vertical segmentation to separate product information, buyer information, seller information, and payment information into separate tables, and associate them with basic order information through the order number.
There is also a situation, if a table has 10 fields, of which only 3 fields need frequent modification, then you can consider splitting these 3 fields into sub-tables. Avoid modifying these 3 data, affecting the query row lock of the remaining 7 fields.
New problems after the three-point library and sub-table
1 After sub-database and sub-table, how to make the data evenly scattered in each sub-database and sub-table?
For example, when a hot event occurs, how to avoid the centralized access of hot data to a specific library/table, causing the problem of uneven reading and writing pressure of each sub-database and sub-table.
In fact, after careful consideration, you can find that this problem is actually very similar to the problem of load balancing, so we can learn from the solution of load balancing to solve it. Our common balancing algorithms are as follows:
Our choice: tailoring based on the consistent Hash algorithm. Compared with the consistent Hash algorithm, our tailored algorithm is
mainly different in the following points:
(1) The difference in the number of nodes in the Hash ring
Consistent Hash has 2^32-1 nodes. Considering that we are segmented according to buyerId, and the base of buyerId is very large, the whole has a certain degree of uniformity, so we reduce the number of Hash rings to 4096;
(2) The difference of DB index algorithm
Consistent Hash calculates the position of DB in the Hash ring through a formula similar to hash (DB's IP)% 2^32. If the number of DBs is small, it is necessary to add virtual nodes to solve the Hash ring skew problem, and the location of the DB may change with the change of IP, especially in the cloud environment.
The data is evenly distributed to the Hash ring. After previous judgments, we can use Math.abs(buyerId.hashCode())% 4096 to calculate the location of the Hash ring. Then the remaining problem is to make the DB evenly distributed to this Hash. Just put it on the ring. Since we all use Ali’s TDDL middleware, we only need to locate the DB based on the logical sub-database index number. Therefore, we can divide the sub-database DB evenly on this Hash ring. If it is a hash ring, there are 4096 links. If the 4 libraries are split, the 4 libraries are located on the 1, 1025, 2049, and 3073 nodes respectively. The index positioning of the sub-library can be calculated by the formula (Math.abs(buyerId.hashCode())% 4096) / (4096 / DB_COUNT).
The Java pseudo-code implementation of sub-library index is as follows:
/**
* 分库数量
*/
public static final int DB_COUNT = 4;
/**
* 获取数据库分库索引号
*
* @param buyerId 会员ID
* @return
*/
public static int indexDbByBuyerId(Long buyerId) {
return (Math.abs(buyerId.hashCode()) % 4096) / (4096 / DB_COUNT);
}2 How to solve the problem of uniqueness of the primary key ID after the database is divided under the environment of sub-database sub-table?
In a single database environment, the ID of the main table of our question sheet adopts the method of MySQL self-increment. However, if you continue to use the self-increment method of the database after sub-database, it is easy to have the problem of duplicate primary key ID at each door.
For this situation, there are many solutions, such as using UUID, but the UUID is too long, the query performance is too poor, the space occupied is also large, and the type of the primary key has also changed, which is not conducive to smooth application migration.
In fact, we can also continue to split the ID, such as segmenting the ID. Different database tables use different ID segments, but it will also cause new problems. How long should this ID segment be? If the ID segment is allocated, it may occupy the ID segment of the second library, causing the problem of non-unique ID.
However, if we let all the ID segments used by the sub-libraries be separated according to the arithmetic sequence, and each time the ID segment is used up, it is increased according to a fixed step ratio, then this problem can be solved.
For example, as shown below, assuming that the ID interval for each allocation is 1000, that is, the step length is 1000, then the start and end index of the ID segment for each allocation can be calculated according to the following formula:
- The starting index of the ID segment allocated for the Xth library and the Yth time is:
X * 步长 + (Y-1) * (库数量 * 步长)- The end index of the ID segment allocated for the Xth library and the Yth time is:
X * 步长 + (Y-1) * (库数量 * 步长) + (1000 -1)If it is divided into 4 libraries, then the final allocated ID segment will look like this:
Our question list library uses this method of segmenting the ID first, and then increasing it by a fixed step. This is also the official solution provided by TDDL.
In addition, in actual scenarios, usually in order to analyze and troubleshoot problems, some additional information is often added to the ID. For example, our own problem ticket ID contains information such as date, version, and sub-database index.
Question ticket ID generation Java pseudo code reference:
import lombok.Setter;
import org.apache.commons.lang3.time.DateFormatUtils;
/**
* 问题单ID构建器
* <p>
* ID格式(18位)：6位日期 + 2位版本号 + 2位库索引号 + 8位序列号
* 示例：180903010300001111
* 说明这个问题单是2018年9月3号生成的，采用的01版本的ID生成规则，数据存放在03库，最后8位00001111是生成的序列号ID。* 采用这种ID格式还有个好处就是每天都有1亿(8位)的序列号可用。* </p>
*/
@Setter
public class ProblemOrdIdBuilder {
public static final int DB_COUNT = 4;
private static final String DATE_FORMATTER = "yyMMdd";
private String version = "01";
private long buyerId;
private long timeInMills;
private long seqNum;
public Long build() {
int dbIndex = indexDbByBuyerId(buyerId);
StringBuilder pid = new StringBuilder(18)
.append(DateFormatUtils.format(timeInMills, DATE_FORMATTER))
.append(version)
.append(String.format("%02d", dbIndex))
.append(String.format("%08d", seqNum % 10000000));
return Long.valueOf(pid.toString());
}
/**
* 获取数据库分库索引号
*
* @param buyerId 会员ID
* @return
*/
public int indexDbByBuyerId(Long buyerId) {
return (Math.abs(buyerId.hashCode()) % 4096) / (4096 / DB_COUNT);
}
}3 How to solve the transaction problem in a sub-database and sub-table environment?
In a distributed environment, a transaction may span multiple sub-databases, so processing is relatively complicated. There are currently two common solutions:
(1) Use distributed transactions
- Advantages: The application server/database manages the affairs, which is simple to implement
- Disadvantages: The performance cost is high, especially when it involves a large number of sub-libraries. Moreover, it also relies on the distributed transaction implementation solutions provided by some specific application servers/databases.
(2) Jointly controlled by application + database
- Principle: Make major events smaller, split multiple major transactions into smaller transactions that can be handled by a single sub-database, and the application program controls these minor transactions.
- Advantages: good performance, without a distributed transaction coordination processing layer
- Disadvantages: The flexible design of transaction control needs to be done from the application itself. To deal with business applications, the cost of transformation should be high.
For the above two distributed transaction solutions, how should we choose?
First of all, there is no one-size-fits-all solution, only one that suits you. Let's take a look at the usage scenarios of transactions in our business first.
Whether it is a member who comes to consult a problem, or a customer service junior who solves a problem for a member, or synchronizes relevant data from a third-party system. There are mainly 2 core actions:
- Query related progress data in the member dimension, including member problem data, and the corresponding problem handling operation log/progress data;
- Submit relevant vouchers/feedback new information and other data from the perspective of the member, or the second-generation customer service member submits these data. The submitted data may also determine whether the problem is solved (closed).
Since the questionnaire data and operation log are queried separately, the distributed relational query scenario is not involved, and this can be ignored.
Then there is only the user submitting the data scenario, and the problem sheet and the operation log data may be written at the same time.
Now that the usage scenario is determined, you can choose a transaction solution. Although the implementation of distributed transactions is simple, it is simple because middleware helps us solve its own complexity. The higher the complexity, the certain performance loss will inevitably be brought. Moreover, most of the current applications are developed based on SpringBoot, and the embedded tomcat container is used by default. Unlike the heavyweight application servers such as WebSphere Application Server and Oracle's WebLogic provided by IBM, they all provide a built-in distributed transaction manager. . Therefore, if we want to access, we must introduce an additional distributed transaction manager ourselves, and this access cost is even higher. Therefore, this scheme will not be considered for the time being. Then, you can only find a way to divide the big transaction into small transactions that can be solved by a single database.
Therefore, the question now becomes, how to write the question sheet data of the same member and the operation log data related to this question sheet into the same sub-database. In fact, the solution is relatively simple. Since the member ID is used for segmentation, the same sub-database routing rules can be used.
Finally, let me look at the final TDDL sub-database sub-table rule configuration:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd">
<beans>
<bean id="vtabroot" class="com.taobao.tddl.interact.rule.VirtualTableRoot" init-method="init">
<property name="dbType" value="MYSQL" />
<property name="defaultDbIndex" value="PROBLEM_0000_GROUP" />
<property name="tableRules">
<map>
<entry key="problem_ord" value-ref="problem_ord" />
<entry key="problem_operate_log" value-ref="problem_operate_log" />
</map>
</property>
</bean>
<!-- 问题(诉求)单表 -->
<bean id="problem_ord" class="com.taobao.tddl.interact.rule.TableRule">
<property name="dbNamePattern" value="PROBLEM_{0000}_GROUP" />
<property name="tbNamePattern" value="problem_ord_{0000}" />
<property name="dbRuleArray" value="((Math.abs(#buyer_id,1,4#.hashCode()) % 4096).intdiv(1024))" />
<property name="tbRuleArray">
<list>
<value>
<![CDATA[
def hashCode = Math.abs(#buyer_id,1,32#.hashCode());
int dbIndex = ((hashCode % 4096).intdiv(1024)) as int;
int tableCountPerDb = 32 / 4;
int tableIndexStart = dbIndex * tableCountPerDb;
int tableIndexOffset = (hashCode % tableCountPerDb) as int;
int tableIndex = tableIndexStart + tableIndexOffset;
return tableIndex;
]]>
</value>
</list>
</property>
<property name="allowFullTableScan" value="false" />
</bean>
<!-- 问题操作日志表 -->
<bean id="problem_operate_log" class="com.taobao.tddl.interact.rule.TableRule">
<property name="dbNamePattern" value="PROBLEM_{0000}_GROUP" />
<property name="tbNamePattern" value="problem_operate_log_{0000}" />
<!-- 【#buyer_id,1,4#.hashCode()】 -->
<!-- buyer_id 代表分片字段；1代表分库步长；4代表一共4个分库，当执行全表扫描时会用到 -->
<property name="dbRuleArray" value="((Math.abs(#buyer_id,1,4#.hashCode()) % 4096).intdiv(1024))" />
<property name="tbRuleArray">
<list>
<value>
<![CDATA[
def hashCode = Math.abs(#buyer_id,1,512#.hashCode());
int dbIndex = ((hashCode % 4096).intdiv(1024)) as int;
int tableCountPerDb = 512 / 4;
int tableIndexStart = dbIndex * tableCountPerDb;
int tableIndexOffset = (hashCode % tableCountPerDb) as int;
int tableIndex = tableIndexStart + tableIndexOffset;
return tableIndex;
]]>
</value>
</list>
</property>
<property name="allowFullTableScan" value="false" />
</bean>
</beans>
4 How can the historical data be smoothly migrated after the database is divided into tables?
Database replication solution, Alibaba Cloud also opened the previous database replication and migration solution "Data Transmission Service" [1] used internally by Alibaba. For details, please consult Alibaba Cloud customer service or Alibaba Cloud database experts.
The release process of sub-library switching can choose between shutdown and non-stop release:
(1) If you choose to stop publishing
- First, choose a night with high black wind and no one everywhere. The biting cold wind can make you sober and there is no one around, so you can do things and steal data. We chose to switch when there is no one at 4 in the morning; if possible, it is best to temporarily close the business access entrance.
- Then, add a full data copy task on DTS to copy the data of a single database to the new sub-database (this process is very fast, and tens of millions of data should be done in about 10 minutes);
- After that, switch the TDDL configuration (single library -> sub library) and restart the application to check whether it takes effect.
- Finally, open the business access portal to provide services.
(2) If you choose to publish without stopping, the process will be slightly more complicated
- First of all, you also need to choose a dark and windy night to set off your handsomeness.
- Then, use DTS to copy the data before a certain point in time, such as the historical data before today.
- After that, switch from single library to sub-library (it is better to publish the application in advance and prepare the configuration), so that it only takes a few minutes to restart to take effect. Before switching to the sub-database, contact the DBA to stop the old single-database reading and writing during the switching period.
- Finally, after the sub-library switch is completed, the data generated after this morning in the old single library is incrementally copied through DTS.
- Finally, continue to observe for a period of time, if there is no problem, the old single library can be offline.
5 Precautions when TDDL configures sub-database and sub-table routing
Since Ali's TDDL middleware uses groovy scripts to calculate sub-database and sub-table routing, and groovy's / operator or /= operator may produce a double type result, not an integer like Java does, so you need to use x. The intdiv(y) function does an integer division operation.
// 在 Java 中
System.out.println(5 / 3); // 结果 = 1
// 在 Groovy 中
println (5 / 3); // 结果 = 1.6666666667
println (5.intdiv(3)); // 结果 = 1（Groovy整除正确用法）For details, please refer to the official Groovy description "The case of the division operator":
Case illustration in the quartet database sub-table text
This article is the original content of Alibaba Cloud and may not be reproduced without permission.