分库分表
什么是分库分表
– 通过某种特定条件,将存放在一个数据库(主机)中的数
据,分散存放到多个数据库(主机)中。
– 已达到分散单台设备负载的效果,即分库分表
– 数据的切分根据其切分规则的类型,分为2中切分模式
– 垂直分割(纵向) 和 水平分割(横向)
垂直分割
• 纵向切分
– 把单一的表,拆分成多个表,并分散到不同的数据库
(主机)上。
– 一个数据库由多个表构成,每个表对应不同的业务,
可以按照业务对表进行分类,将其分布到不同的数据
库(主机)上,实现专库专用,让不同的库(主机)分担不
同的业务。
• 横向切分
– 按照表中某个字段的某种规则,把向表中写入的记录
分散到多个库(主机)中。
– 简单来说,就是按照数据行切分,将表中的某些行存
储到指定的数据库 (主机) 中。
mycat
基于Java的分布式数据库系统中间层,为高并发下的
分布式提供解决方案
– 支持JDBC形式连接
– 支持MySQL、Oracle、Sqlserver、Mongodb等
– 提供数据读写分离服务
– 可以实现数据库服务器的高可用
– 提供数据分片服务
– 基于阿里巴巴Cobar进行研发的开源软件
– 适合数据大量写入数据的存储需求
分片算法
• mycat服务提供10种分片算法。
– 1枚举法 sharding-by-intfile
– 2固定分片hash算法 rule1
– 3范围约定 auto-sharding-long
– 4求模法 mod-log
– 5日期列分区法 sharding-by-date
– 6通配取模 sharding-by-pattern
– 7ASCII码求模通配 sharding-by-prefixpattern
– 8编程指定 sharding-by-substring
– 9字符串拆分hash解析 sharding-by-stringhash
– 10一致性hash sharding-by-murmur
工作过程
当Mycat收到一个SQL时
– 会先解析这个SQL查找涉及到的表,然后看此表的定义
– 如果有分片规则,则获取到SQL里分片字段的值,并匹
配分片函数,得到该QL对应的分片列表
– 然后将SQL发往这些分片去执行,最后收集和处理所有
分片返回的结果数据,并输出到客户端
以select * from Orders where prov=?语句为例,查到prov=wuhan,按
照分片函数,wuhan返回dn1,于是SQL就发给了MySQL1,去取DB1
上的查询结果,并返回给用户。
如果上述SQL改为elect * from Orders where prov in (‘wuhan’,‘beijing’),
那么,SQL就会发给ySQL1与MySQL2去执行,然后结果集合并后输出
给用户。但通常业务中我们的SQL会有Order By 以及Limit翻页语法,
此时就涉及到结果集在Mycat端的二次处理。
54主机 --数据库服务器
55主机 --数据库服务器
56主机 分片服务器
room90 客户端54主机
[root@host54 ~]# yum -y install perl-JSON
[root@host54 ~]# rpm -Uvh mysql-community-*.rpm
[root@host54 ~]# systemctl start mysqld
[root@host54 ~]# vim /etc/my.cnf
[mysqld]
lower_case_table_names = 1
validate_password_policy=0
validate_password_length=6
[root@host54 ~]# mysql -uroot -p'equv3R%l+F5_'
mysql> alter user root@"localhost" identified by "123456";
mysql> grant all on *.* to
-> root@"%"
-> identified by "123456";
[root@host54 ~]# systemctl restart mysqld
55主机
[root@host55 ~]# yum -y install perl-JSON
[root@host55 ~]# rpm -Uvh mysql-community-*.rpm
[root@host55 ~]# systemctl start mysqld
[root@host55 ~]# vim /etc/my.cnf
[mysqld]
lower_case_table_names = 1
validate_password_policy=0
validate_password_length=6
[root@host55 ~]# mysql -uroot -p'equv3R%l+F5_'
mysql> alter user root@"localhost" identified by "123456";
mysql> grant all on *.* to
-> root@"%"
-> identified by "123456";
[root@host54 ~]# systemctl restart mysqld
56主机
配置数据分片数据56步骤
1.装包
2.修改配置文件
2.1 定义连接mycat服务的用户和密码及虚拟数据库名词: server.xml
test test testdb 读写权限
user user tsetDB 只读权限
2.2 对哪些表做数据分片及使用的分片规则
schema.xml
逻辑表名 使用的分片规则
存储到哪个数据库服务器 dn1 dn2
指定dn1 存储数据的库名 db1
指定dn2 存储数据的库名 db2
指定dn1 对应的数据库服务器ip地址
指定dn2 对应的数据库服务器ip地址
用rpm包的时候，其会自动把相应的包添加到相应的默认文件下
[root@host56 ~]# tar -xf Mycat-server-1.4-beta-20150604171601-linux.tar.gz #免安装,解压后即可使用
[root@host56 ~]# cd mycat/
[root@host56 mycat]# ls
bin catlet conf lib logs version.txt
[root@host56 ~]# rpm -qa | grep -i jdk #安装系统自带的即可
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64
copy-jdk-configs-2.2-3.el7.noarch
[root@host56 ~]# mv mycat/ /usr/local/
[root@host56 ~]# ls /usr/local/
bin etc games include lib lib64 libexec mycat sbin share src
[root@host56 local]# cd mycat/
[root@host56 mycat]# ls
bin catlet conf lib logs version.txt目录结构说明
– bin mycat命令 如 启动 停止 等
– catlet 扩展功能
– conf 配置文件
– lib mycat使用的jar包 mycat是java开发的
– log mycat启动日志和运行日志
– wrapper.log mycat 服务启动日志 ,启动有问题可以看这个日志的内容
– mycat.log 记录 sql脚本执行后的具体报错内容
重要配置文件说明
– server.xml设置连接mycat服务的账号 、密码等
– schema.xml 配置mycat使用的真实数据库和表
– rule.xml 定义mycat分片规则
[root@host56 mycat]# ls bin/
mycat startup_nowrap.sh wrapper-linux-x86-32
rehash.sh wrapper-linux-ppc-64 wrapper-linux-x86-64
[root@host56 mycat]# ls conf/
autopartition-long.txt partition-range-mod.txt sequence_db_conf.properties
cacheservice.properties router.xml sequence_time_conf.properties
ehcache.xml rule.xml server.xml
log4j.xml schema.xml wrapper.conf
partition-hash-int.txt sequence_conf.properties
[root@host56 mycat]# ls
bin catlet conf lib logs version.txt
[root@host56 mycat]# cd conf
[root@host56 conf]# ls
autopartition-long.txt partition-range-mod.txt sequence_db_conf.properties
cacheservice.properties router.xml sequence_time_conf.properties
ehcache.xml rule.xml server.xml
log4j.xml schema.xml wrapper.conf
partition-hash-int.txt sequence_conf.properties
[root@host56 conf]# vim rule.xml
[root@host56 conf]# vim server.xml
[root@host56 conf]# vim schema.xml
[root@host56 conf]# cp schema.xml schema.xml.bak
[root@host56 conf]# ls
autopartition-long.txt partition-range-mod.txt sequence_conf.properties
cacheservice.properties router.xml sequence_db_conf.properties
ehcache.xml rule.xml sequence_time_conf.properties
log4j.xml schema.xml server.xml
partition-hash-int.txt schema.xml.bak wrapper.conf
[root@host56 conf]# ls
autopartition-long.txt partition-range-mod.txt sequence_conf.properties
cacheservice.properties router.xml sequence_db_conf.properties
ehcache.xml rule.xml sequence_time_conf.properties
log4j.xml schema.xml server.xml
partition-hash-int.txt schema.xml.bak wrapper.conf
[root@host56 conf]# cd ..
[root@host56 mycat]# ls
bin catlet conf lib logs version.txt.
[root@host56 mycat]# java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b12)
OpenJDK 64-Bit Server VM (build 25.131-b12, mixed mode)
[root@host56 mycat]# rpm -qa | grep -i jdk
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64
copy-jdk-configs-2.2-3.el7.noarch
[root@host56 mycat]# which java
/usr/bin/java
[root@host56 conf]# vim server.xml
34 <user name="test"> #连接mycat服务时使用的用户名 test
35 <property name="password">test</property> #使用test用户连接mycat用户时使用的密码
36 <property name="schemas">TESTDB</property>#连接上mycat服务后,可以看到的库名多个时,使用逗号分 隔 (是逻辑上的库名)
37 </user>
39 <user name="user">
40 <property name="password">user</property>
41 <property name="schemas">TESTDB</property>
42 <property name="readOnly">true</property>
#定义只读权限,使用定义的user用户连接mycat服务后只有读记录的权限
43 </user>
[root@host56 conf]# cp schema.xml schema.xml.bak
[root@host56 conf]# vim schema.xml
5<schema name="TESTDB" checkSQLschema="false" sqlMaxLimit="100">
6 <!-- auto sharding by id (long) -->
7 <table name="travelrecord" dataNode="dn1,dn2" rule="auto-sharding-long" />
#定义分片的表
8
9 <!-- global table is auto cloned to all defined data nodes ,so can join
10 with any table whose sharding node is in the same data node -->
11 <table name="company" primaryKey="ID" type="global" dataNode="dn1,dn2" />
#定义分片的表
12 <table name="goods" primaryKey="ID" type="global" dataNode="dn1,dn2" />
#定义分片的表
14 <!-- random sharding using mod sharind rule -->
15 <table name="hotnews" primaryKey="ID" dataNode="dn1,dn2"
#定义分片的表
16 rule="mod-long" />
17 <!-- <table name="dual" primaryKey="ID" dataNode="dnx,dnoracle2" type="global"
18 needAddLimit="false"/> <table name="worker" primaryKey="ID" dataNode="jdbc_dn 1,jdbc_dn2,jdbc_dn3"
19 rule="mod-long" /> -->
20 <table name="employee" primaryKey="ID" dataNode="dn1,dn2"
21 rule="sharding-by-intfile" />
#定义分片的表
22 <table name="customer" primaryKey="ID" dataNode="dn1,dn2"
23 rule="sharding-by-intfile">
24 <childTable name="orders" primaryKey="ID" joinKey="customer_id"
25 parentKey="id">
26 <childTable name="order_items" joinKey="order_id"
27 parentKey="id" />
28 </childTable>
29 <childTable name="customer_addr" primaryKey="ID" joinKey="customer_id"
30 parentKey="id" />
31 </table>
37 <dataNode name="dn1" dataHost="c1" database="db1" />
38 <dataNode name="dn2" dataHost="c2" database="db2" />
43 <dataHost name="c1" maxCon="1000" minCon="10" balance="0"
#定义分片使用的库，所在的物理主机，真正存储数据的db1库在物理主机c1上
44 writeType="0" dbType="mysql" dbDriver="native" switchType="1" slaveThreshold="100">
45 <heartbeat>select user()</heartbeat>
46 <!-- can have multi write hosts -->
47 <writeHost host="hostM1" url="192.168.4.21:3306" user="root"
48 password="123456">
49 <!-- can have multi read hosts -->
51 </writeHost>
52 <!--<writeHost host="hostS1" url="localhost:3316" user="root"
53 password="123456" /> -->
54 <!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -- >
55 </dataHost>
56 <dataHost name="c2" maxCon="1000" minCon="10" balance="0"
#定义分片使用的库，所在的物理主机，真正存储数据的db2库在物理主机c2上
57 writeType="0" dbType="mysql" dbDriver="native" switchType="1" slaveThreshold="100">
58 <heartbeat>select user()</heartbeat>
59 <!-- can have multi write hosts -->
60 <writeHost host="hostM1" url="192.168.4.22:3306" user="root"
61 password="123456"> #访问数据时 mycat服务器连接数据库服务器时使用的用户名和密码
62 <!-- can have multi read hosts -->
64 </writeHost>
65 <!--<writeHost host="hostS1" url="localhost:3316" user="root"
66 password="123456" /> -->
67 <!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -- >
68 </dataHost>
[root@host23 ~]# vim /etc/my.cnf
lower_case_table_names = 1 #表名区分字母大小写
启动mycat 当不是用rpm安装的java的jdk的时候，需要在mysql的指定它的路径，当然也可以使用系统自己带的
[root@host56 mycat]# which java
/usr/bin/java
[root@host56 ~]# which java
/usr/bin/java
[root@host56 ~]#sed -n '4.5p' /usr/local/mycat/conf/wrapper.conf
#Java Application
wrapper.java.command=java
[root@host56 ~]#echo "export PATH=/usr/local/mycat/bin" >>/etc/profile
[root@host56 ~]# /usr/local/mycat/bin/mycat --help
Usage: /usr/local/mycat/bin/mycat { console | start | stop | restart | status | dump }
[root@host56 ~]# whatis /usr/local/mycat/bin/mycat
/usr/local/mycat/bin/mycat：没有 appropriate。
[root@host56 ~]# /usr/local/mycat/bin/mycat start
Starting Mycat-server...
[root@host56 ~]# ls /usr/local/mycat/logs/
mycat.log mycat.pid wrapper.log
[root@host56 ~]# cat /usr/local/mycat/logs/mycat.pid
5907
[root@host56 ~]# ps -p 5907
PID TTY TIME CMD
5907 ? 00:00:00 wrapper-linux-x
[root@host56 ~]# netstat -utnlp | grrep :8066
[root@host56 ~]# netstat -utnlp | grep :8066
tcp6 0 0 :::8066 :::* LISTEN 5909/java
[root@host56 ~]# netstat -utnlp | grep java
tcp 0 0 127.0.0.1:32000 0.0.0.0:* LISTEN 5909/java
tcp6 0 0 :::45342 :::* LISTEN 5909/java
tcp6 0 0 :::1984 :::* LISTEN 5909/java
tcp6 0 0 :::8066 :::* LISTEN 5909/java
tcp6 0 0 :::37416 :::* LISTEN 5909/java
tcp6 0 0 :::9066 :::* LISTEN 5909/java
[root@host56 ~]# /usr/local/mycat/bin/mycat stop
Stopping Mycat-server...
Stopped Mycat-server.
[root@host56 ~]# netstat -utnlp | grep java
[root@host56 ~]# /usr/local/mycat/bin/mycat start
Starting Mycat-server...
[root@host56 ~]# rpm -qa | grep -i jdk
java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64
java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64
copy-jdk-configs-2.2-3.el7.noarch
[root@host56 ~]# cd /usr/local/
[root@host56 local]# ls
bin etc games include lib lib64 libexec mycat sbin share src
[root@host56 local]# cd mycat/
[root@host56 mycat]# ls
bin catlet conf lib logs version.txt
[root@host56 mycat]# ls bin/
mycat startup_nowrap.sh wrapper-linux-x86-32
rehash.sh wrapper-linux-ppc-64 wrapper-linux-x86-64
[root@host56 mycat]#
[root@host56 mycat]# ls conf/
autopartition-long.txt partition-range-mod.txt sequence_db_conf.properties
cacheservice.properties router.xml sequence_time_conf.properties
ehcache.xml rule.xml server.xml
log4j.xml schema.xml wrapper.conf
partition-hash-int.txt sequence_conf.properties
[root@host56 mycat]# ls
bin catlet conf lib logs version.txt
测试配置
[root@room9pc01 桌面]# mysql -h192.168.4.23 -P8066 -utest -ptest
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.5.8-mycat-1.4-beta-20150604171601 MyCat Server (OpenCloundDB)
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> show databases;
+----------+
| DATABASE |
+----------+
| TESTDB |
+----------+
1 row in set (0.01 sec)
MySQL [(none)]> use TESTDB
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MySQL [TESTDB]> show tables;
+------------------+
| Tables in TESTDB |
+------------------+
| company |
| customer |
| customer_addr |
| employee |
| goods |
| hotnews |
| orders |
| order_items |
| travelrecord |
+------------------+
9 rows in set (0.00 sec)
枚举规则：
[root@host56 ~]# cd /usr/local/mycat/
[root@host56 mycat]# ls
bin catlet conf lib logs version.txt
[root@host56 mycat]# cd conf/
[root@host56 conf]# vim rule.xml
[root@host56 conf]# pwd
/usr/local/mycat/conf
[root@host56 conf]# ls partition-hash-int.txt
partition-hash-int.txt
[root@host56 conf]# cat partition-hash-int.txt
10000=0
10010=1[root@host56 conf]#
[root@host56 conf]# vim partition-hash-int.txt
ID sharding_id
10000=0 dn1 c1 192.168.4.21 db1
100010=1 dn2 c2 192.168.4.22 db2
create table employee(
id int not null primary key,
name varchar(100),
age int(2),
sharding_id int not null);
insert into emplpyee(id ,name,age,sharding_id) values(1,"bob",18,10000);
insert into emplpyee(id ,name,age,sharding_id) values(2,"lucy",12,10010);
[root@room9pc01 桌面]# mysql -h192.168.4.23 -P8066 -utest -ptest
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.5.8-mycat-1.4-beta-20150604171601 MyCat Server (OpenCloundDB)
Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [TESTDB]> create table employee(
-> id int not null primary key,
-> name varchar(100),
-> age int(2),
-> sharding_id int not null);
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id: 3
Current database: TESTDB
Query OK, 0 rows affected (0.60 sec)
MySQL [TESTDB]> show tables
-> ;
+------------------+
| Tables in TESTDB |
+------------------+
| company |
| customer |
| customer_addr |
| employee |
| goods |
| hotnews |
| orders |
| order_items |
| travelrecord |
+------------------+
9 rows in set (0.00 sec)
MySQL [TESTDB]> desc employee
-> ;
+-------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| name | varchar(100) | YES | | NULL | |
| age | int(2) | YES | | NULL | |
| sharding_id | int(11) | NO | | NULL | |
+-------------+--------------+------+-----+---------+-------+
4 rows in set (0.01 sec)
MySQL [TESTDB]> insert into employee(id ,name,age,sharding_id) values(1,"bob",18,10000);
Query OK, 1 row affected (0.07 sec)
MySQL [TESTDB]> insert into employee(id ,name,age,sharding_id) values(2,"lucy",19,10010);
Query OK, 1 row affected (0.04 sec)
MySQL [TESTDB]> insert into employee(id ,name,age,sharding_id) values(3,"zhy",20,10000);
Query OK, 1 row affected (0.08 sec)
MySQL [TESTDB]> insert into employee(id ,name,age,sharding_id) values(4,"zhu",10,10010);
Query OK, 1 row affected (0.06 sec)
数据库服务器55测试结果
[root@host55 ~]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 23
Server version: 5.7.17 MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| db1 |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.01 sec)
mysql> use db1
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+---------------+
| Tables_in_db1 |
+---------------+
| employee |
+---------------+
1 row in set (0.00 sec)
mysql> desc employee
-> ;
+-------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| name | varchar(100) | YES | | NULL | |
| age | int(2) | YES | | NULL | |
| sharding_id | int(11) | NO | | NULL | |
+-------------+--------------+------+-----+---------+-------+
4 rows in set (0.00 sec)
mysql> select * from employee;
+----+------+------+-------------+
| id | name | age | sharding_id |
+----+------+------+-------------+
| 1 | bob | 18 | 10000 |
| 3 | zhy | 20 | 10000 |
+----+------+------+-------------+
2 rows in set (0.00 sec)
数据库服务器56测试结果
[root@host54 ~]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 25
Server version: 5.7.17 MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| db2 |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)
mysql> use db2
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+---------------+
| Tables_in_db2 |
+---------------+
| employee |
+---------------+
1 row in set (0.00 sec)
mysql> desc employee
-> ;
+-------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| name | varchar(100) | YES | | NULL | |
| age | int(2) | YES | | NULL | |
| sharding_id | int(11) | NO | | NULL | |
+-------------+--------------+------+-----+---------+-------+
4 rows in set (0.00 sec)
mysql> select * from employee;
+----+------+------+-------------+
| id | name | age | sharding_id |
+----+------+------+-------------+
| 2 | lucy | 19 | 10010 |
| 4 | zhu | 10 | 10010 |
+----+------+------+-------------+
2 rows in set (0.00 sec)