1. centos7下安装mariadb 10.5.3
1.1 下载安装包
# cd /opt
# wget http://mirrors.dotsrc.org/mariadb/mariadb-10.5.3/bintar-linux-systemd-x86_64/mariadb-10.5.3-linux-systemd-x86_64.tar.gz
# tar xf mariadb-10.5.3-linux-systemd-x86_64.tar.gz
# cp -a mariadb-10.5.3-linux-systemd-x86_64 /usr/local/mysql
1.2 创建数据目录，编辑配置文件
[root@localhost mariadb]# groupadd mysql
[root@localhost mariadb]# useradd -s /sbin/nologin -r -g mysql mysql
# mkdir /data/mariadb -p
# chown mysql.mysql /data/ -R
[root@db04 support-files]# cat /etc/my.cnf
[mysqld]
basedir=/usr/local/mysql/
datadir=/data/mariadb/
port=3360
pid-file=/data/mariadb/mysql.pid
socket=/tmp/mysql.sock
[mysqld_safe]
log-error=/data/mariadb/mysql.log
[client]
port=3360
socket=/tmp/mysql.sock
default-character-set=utf8
1.3 设置环境变量，数据库初始化,查看启动进程
[root@db04 ~]# tail -1 /etc/profile
export PATH=/usr/local/mysql/bin:$PATH
# ./mysql_install_db --user=mysql --defaults-file=/etc/my.cnf --datadir=/data/mariadb/ --basedir=/usr/local/mysql/
[root@db04 support-files]# mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
haven't set the root password yet, you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password or using the unix_socket ensures that nobody
can log into the MariaDB root user without the proper authorisation.
You already have your root account protected, so you can safely answer 'n'.
Switch to unix_socket authentication [Y/n] y
Enabled successfully!
Reloading privilege tables..
... Success!
You already have your root account protected, so you can safely answer 'n'.
Change the root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] n
... skipping.
By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] n
... skipping.
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] y
... Success!
Cleaning up...
All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!
[root@db04 support-files]# ./mysql.server restart
Shutting down MariaDB. SUCCESS!
Starting MariaDB.200608 17:02:52 mysqld_safe Logging to '/data/mariadb/mysql.log'.
200608 17:02:52 mysqld_safe Starting mysqld daemon with databases from /data/mariadb/
SUCCESS!
[root@db04 support-files]# ss -tlunp |grep mysql
tcp LISTEN 0 80 [::]:3360 [::]:* users:(("mysqld",pid=7419,fd=15))
2. spider存储引擎概述及原理
2.1 spider概述
spider是mariadb10.0.0.4版本推出的，首先spider提供的是从一台mariadb服务器访问另一台mariadb服务器的方法。spider提供内置分片（sharding）特性，和支持分区（partition）XA两阶段的分布式事务。
2.2 spider原理
当我们在spider存储引擎上创建一个表，该表会指向远程服务器中一个表或多个实例上的表，就像linux上的软件链接一样。而且远程服务器的表可以是任何存储引擎的表。链接对于同一个事务来讲都是共享的。
2.3 spider介绍
1. 表链接
简单来说就是在本地创建一个spider存储引擎的表时，该表指向远程服务器上对应的一张表或多个实例的表。
例如在远程服务器上创建一张表（192.168.214.133）： create table test (id int primary key,name varchar(32));
在spider上创建一张表指向该表：
create table test (id int primary key,name varchar(32)) engine=spider comment 'host "192.168.214.133",userr "spider",password "spider",port "3360"';
spider的系统表spider_tables记录了各个数据分片的位置信息。
2.事务
spider支持单机事务和XA事务；
Spider信赖后端数据节点保证事务的持久性以及隔离性，它负责开启事务，以及在适当的时机发送提交或者回滚事务的命令。如果单机事务涉及多个数据节点，spider需要将相应的连接保存在队列中。在事务提交或者回滚的时候，依次发送相应的命令。
spider参照分布式事务DTP／XA模型实现了分布式XA事务。在这个模型中，存在RM（resource manager资源管理器），TM（transaction manager，事务管理器）以及AP（application ，应用程序）三种角色，AP通过RM API来操作和管理资源，通过TM接口来开启／终止结束事务。RM和TM之间需要实现XA接口。XA接口定义了两阶段提交的必要步骤，以及RM和TM之间需要进行交互。spider扮演的是TM角色，而后端数据是RM 角色。
为使用分布式XA事务，业界定义的XA命令如下：
XA START ‘trx-id’: 开启事务
do actual work;实际折查询执行语句
XA END ’trx-id‘：XA事务结束
XA prepare ‘trx-id’：预提交
XA COMMIT ‘trx-id’: 提交
spider会在系统表spider_xa中记录XA事务的状态信息，同时会在spider_xa_members中记录XA事务的节点信息。
spiderXA事务有四种状态：NOT YET，PREPARED,ROLLBACK，COMMITTED。SPIDER在开始准备阶段会在系统表中记录该XA事务的状态为NOT YET，当所有数据节点接收到prepare消息以后，进阶到PREPARED阶段，如果某一节点在PREPAED阶段出现了故障，那么spider会回滚事务。相应的事务的状态会变为ROLLBACK。如果参与事务折节点都返回PREPARE OK的消息，则该事务进入提交阶段，spider向后端节点发送信息。
XA事务转换
Spider向后端节点发送XA START命令时会设置会话级别的事务特性，同时将XA事务ID发送到后端节点。因为XA事务ID由三部分组成，Spider会将这三个部分的解析出来，然后拼接成对应的字符串发送到后端节点。为了节省网络开销，Spider将XA END与XA PREPARE命令合并起来一起发送。也就是在这个阶段初始，Spider在系统表里面记录事务的状态。如果所有的RM都返回OK，那么Spider进入PREPARED状态，准备提交事务。否则，事务进入到回滚状态。
3.可插拔式引擎
在这个架构中，handler类定义了存储引擎提供的接口和功能。因为所有存储引擎从基类handler继承而来，所以它们能够提供相同的功能。总的来说handler类和handlerton 结构在整个体系结构中扮演了中间层的角色。
spider本身并不存放数据，而是类似于一个代理的功能将访问的请求转移到数据节点。
spider提供了两种途径访问后端节点存储数据：
a. 遵循传统的查询处理；SQL查询请求经过查询解析，查询重写，查询优化等，按照生成查询执行计划,spider从后端拉取数据交给mysql处理。spider查询处理框架有一缺点，不能很好地利用后端节点可并行化特性，同时需要对SQL进行两次解析，损耗性能严重；
b. spider开发团队提供了directSQL方式进行查询，DirectSQL的原理类似于MapReduce，将查询直接发到后端节点，不需要在mysql服务器层进行解析，后端节点将结果返回给spider,由spider合并结果集。这个方式有效的消除重复解析SQL语句的行为。
流程：client 发送sql语句－－>mysql的存储引擎层（spider）－－>通过spider进入handler类---->后端提取数据---->返回给spider，进行合并---->返回给客户端
4.读写流程
＊＊＊＊研究数据库系统的查询执行模型＊＊＊＊
数据库系统基本都采用迭代器模型处理查询，也叫volcano查询执行引擎。执行计划树的上层节点通过get_next方法驱动子节点获取一条元组，子节点递归调用。在叶子节点也就是基本表将数据返回。
模型缺点：函数的大量调用使得进程／线程上下文切换频繁，程序的局部性受到损害。因此，后来针对OLAP场景，采用了向量查询执行模型来减少进程上下文的切换以及保证高速缓存的命中率。
图：查询计划树示例
mysql体系的工作原理：查询执行计划由mysql server层生成，存储引擎受执行计划驱动而访问表。mysql的handler已经定义好表的访问方法的存储引擎就可以作为mysql的插件式引擎而存在。
a. select操作
由于spider存储引擎是作为一个代理存在的，不存放数据，因此spider处理select 语句时和update,delete语句类似，首先要将根据查询解析的信息生成一个select语句，发送到查询所涉及的后端节点，再由后端节点将数据返回到本地，进行处理。函数spider_db_append_select_column根据查询结果涉及的读和写获取相应的字段从后端数据拉取到本地。如果有多个分片，spider将从不同实例获取过来的结果集存放在不同的结果集spider_db_result中。类spider_db_fetch提供了fetch_next，current_row等方法供上层方法调用。SERVER层调用get_next方法驱动引擎获取下一条数据。
总结：首先
1. spider不存储任何数据，作为一个proxy；
1. spider在处理update,delete语句时和select语句类似，将信息解析成select语句，先发送到后端节点获取再将数据返回给本地；
1. spider_db_append_select_column函数用来将查询的读和写拉取到本地；
1. 如果有多个分片，不同的分片的结果集放在不同的spider_db_result中；
对于表访问方法，mysql实现了两种：
1). 索引扫描（ha_index_read）；对于切分为多个分片的DB，索引扫描需要借助队列。
如果是第一次使用索引扫描方法：需要遍历所有的分片读取一条记录，然后插入到优先队列；
如果是第一次调用访问远程实例表的方法：需要生成select语句，将远端实例的数据拉到本地存放。
在使用索引扫描方法时，mysql会为每个分片保留一个key buffer和record buffer。server 利用队列头部的m_top_entry 获得访问的分片ID.(分片可能是根据范围分片的也可以是根据hash分片)接着，调用get_next方法获取相应的元组，将返回的数据存放在record buffer，并插入到优先队列。函数最后将元组从优先队列返回。
为缓解内存等资源的压力，Spider实现全表扫描的方法是逐个分片串行扫描（为了加速，spider也提供了并行扫描数据节点的选项）。
- 随机访问（ha_rnd_next）
b. insert操作
mysql的handler类对insert操作提供的接口名是write_row。存储引擎若想支持insert操作必须实现write_row方法。
spider对insert的实现：根据查询解析信息生成一条insert语句，发往后端节点处理。如果是批量的操作，则需要和mysql server层配合，将批量的数据发往到后端节点。
批量插入的insert语句实现：
在执行批量插入insert语句时，如果是多个分片，values中的列必须包含分区键，分区键是自增列的情况除外。mysql_insert根据values包含的元组数目，判断是否需要进行批量插入操作。如果需要，mysql循环调用write_row方法触发spider生成insert语句。spider的write_row方法实现中会根据分区键将insert语句进行分组。
c. delete实现
spider实现delete操作必须实现mysql handler类的ha_delete_row方法。和insert的操作有所不同，delete操作需要生成一条select语句将查询涉及到的分区键拉到spider节点中。这是因为mysql server层的once-a-tuple的查询执行模型会驱动spider拼接delete语句，然后向后端节点发送。这时spider需要知道对应的delete语句该往哪个后端节点发送。为了减少网络开销，spider提供批量发送delete语句的功能。
UPDATE操作的实现类似DELETE，都需要Spider生成SELECT语句从后端节点拉取数据。只不过，UPDATE在更新区分键的时候，可能需要多一次DELETE操作（删除原来分区的数据，将新的数据插入到不同的分区）。UPDATE操作的实现类似DELETE，都需要Spider生成SELECT语句从后端节点拉取数据。只不过，UPDATE在更新区分键的时候，可能需要多一次DELETE操作（删除原来分区的数据，将新的数据插入到不同的分区）。
2.4 spider优劣对比
#优势分析：
a. 对业务完全透明，业务层不需要做任何修改；对于分库和分表的操作业务层不需要关系，只需要通过spider作为代理的入口，真实数据存储在哪台设备上，spider代理会自动进行路由；
b. 方便进行横向扩展，能解决单台mysql的性能和存储瓶颈；
c. 对后端的数据库引擎没有限制；
d. 实现垂直拆分和水平拆分功能，针对分表支持哈希，范围，列表等算法；
e. 完全兼容mysql协议；
#劣势分析：
a. spider本身不支持缓存和全文索引，只能在后端数据库实现全文搜索；
b. spider无法备份数据，只能对后端数据库做物理备份；
c. spider本身是单点的，无法做灾备，只能通过vip的方式自己实现；
d. 由于业务与数据库之间多了一层spider，在性能上多少会有些损耗。
3. spider存储引擎实操
3.1 spider单表连接
a. 安装spider存储引擎
如果是以rpm的方式安装，则安装路径为/usr/share/mysql中，脚本名是install_spider.sql
[root@server1 ~]# mysql
MariaDB [(none)]> source /usr/local/mysql/share/install_spider.sql
Query OK, 0 rows affected, 1 warning (0.002 sec)
Query OK, 0 rows affected (0.001 sec)
+------------------------------+
| @have_spider_i_s_plugin := 1 |
+------------------------------+
| 1 |
+------------------------------+
1 row in set (0.002 sec)
+--------------------------+
| @have_spider_plugin := 1 |
+--------------------------+
| 1 |
+--------------------------+
1 row in set (0.002 sec)
Query OK, 0 rows affected (0.002 sec)
Query OK, 0 rows affected (0.001 sec)
[root@server1 ~]# mysql -S /tmp/mysql.sock -e "show engines;" |grep SPIDER
SPIDER YES Spider storage engine YES YES NO
b. 环境说明
| 主机名 | ip地址 | mysql版本 | 是否装spider | 端口号 |
|---|---|---|---|---|
| server1 | 192.168.214.131 | 10.5.3-MariaDB | 是 | 3360 |
| server2 | 192.168.214.133 | 10.5.3-MariaDB | 否 | 3360 |
在远程服务器server2上创建一张表，且不需要spider
mysql> CREATE DATABASE spidertest;
Query OK, 1 row affected (0.00 sec)
mysql> use spidertest;
Database changed
mysql> CREATE TABLE customer(
id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
name VARCHAR(200) NOT NULL,
address VARCHAR(255) NOT NULL);
Query OK, 0 rows affected (0.02 sec)
mysql> INSERT INTO customer VALUES(NULL, 'John Doe', '1 Main Street');
Query OK, 1 row affected (0.01 sec)
mysql> INSERT INTO customer VALUES(NULL, 'Bob Smith', '45 Elm Street');
Query OK, 1 row affected (0.00 sec)
mysql> INSERT INTO customer VALUES(NULL, 'Jane Jones',
'18 Second Street');
Query OK, 1 row affected (0.00 sec)
c. 在server2上创建用户，保证可以让其他服务器访问自己
mysql> CREATE USER 'spider'@'192.168.214.131' IDENTIFIED BY 'spider';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL ON spidertest.* TO 'spider'@'192.168.214.131';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL ON mysql.* TO 'spider'@'192.168.214.131';
Query OK, 0 rows affected (0.00 sec)
d. 远程测试，在131(server1)主机上连通133 (server2)主机
[root@server1 ~]# mysql -uspider -h192.168.214.133 -P3360 -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 12
Server version: 10.5.3-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
e. 创建server1，访问我们在上面创建的表
// server1 上执行，连接远程server2
MariaDB [(none)]> CREATE SERVER if not exists server2 FOREIGN DATA WRAPPER mysql
OPTIONS(HOST '192.168.214.133', DATABASE 'spidertest', PORT 10482,
USER 'spider', PASSWORD 'spider');
MariaDB [(none)]> DROP DATABASE IF EXISTS spidertest;
Query OK, 0 rows affected, 1 warning (0.000 sec)
MariaDB [(none)]> CREATE DATABASE spidertest;
Query OK, 1 row affected (0.001 sec)
MariaDB [(none)]> use spidertest;
Database changed
MariaDB [spidertest]> CREATE TABLE customer(
id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,
name VARCHAR(200) NOT NULL) ENGINE=Spider
COMMENT = 'wrapper "mysql", srv "Server2"';
Query OK, 0 rows affected (0.132 sec)
f. 查看数据
MariaDB [spidertest]> SELECT * FROM customer;
+----+------------+
| id | name |
+----+------------+
| 1 | John Doe |
| 2 | Bob Smith |
| 3 | Jane Jones |
+----+------------+
3 rows in set (0.006 sec)
3.2 federation setup模式
a. 准备500万条数据
# sysbench /usr/share/sysbench/oltp_common.lua --mysql-host=127.0.0.1 --mysql-user=root --mysql-password='' --mysql-socket=/tmp/mysql.sock --tables=10 --table_size=5000000 --threads=4 --max-requests=0 prepare
b. 首先来进行第一个基准测试，检查一个刚初始化的单一节点性能
[root@server1 ~]# sysbench /usr/share/sysbench/oltp_read_only.lua --mysql-host=127.0.0.1 --mysql-user=root --mysql-password='' --mysql-socket=/tmp/mysql.sock --tables=10 --table_size=5000000 --threads=4 --max-requests=0 run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
Running the test with following options:
Number of threads: 4
Initializing random number generator from current time
Initializing worker threads...
Threads started!
SQL statistics:
queries performed:
read: 78414
write: 0
other: 11202
total: 89616
transactions: 5601 (559.03 per sec.)
queries: 89616 (8944.55 per sec.)
ignored errors: 0 (0.00 per sec.)
reconnects: 0 (0.00 per sec.)
General statistics:
total time: 10.0175s
total number of events: 5601
Latency (ms):
min: 2.19
avg: 7.15
max: 38.41
95th percentile: 10.27
sum: 40036.32
Threads fairness:
events (avg/stddev): 1400.2500/7.36
execution time (avg/stddev): 10.0091/0.01
c. 通过别名定义访问mariadb和mysql客户端的快捷方式
[root@server1 ~]# alias backend1='/usr/local/mysql/bin/mysql --user=spider --password='spider' --host=192.168.214.133 --port=3360'
[root@server1 ~]# alias spider1='/usr/local/mysql/bin/mysql --user=spider --password='spider' --host=192.168.214.131 --port=3360'
d. 在客户端192.168.214.133上创建存储的数据表并插入数据
backend1 <<EOF
create database if not exists spidertest;
use spidertest;
CREATE TABLE if not exists sbtest (
id int(10) unsigned NOT NULL AUTO_INCREMENT,
k int(10) unsigned NOT NULL DEFAULT '0',
c char(120) NOT NULL DEFAULT '',
pad char(60) NOT NULL DEFAULT '',
PRIMARY KEY (id),
KEY k (k)
) ENGINE=InnoDB;
EOF
MariaDB [test]> insert into spidertest.sbtest values(1,1,'a','b');
Query OK, 1 row affected (0.002 sec)
MariaDB [test]> select * from sbtest;
+----+---+---+-----+
| id | k | c | pad |
+----+---+---+-----+
| 1 | 1 | a | b |
+----+---+---+-----+
1 row in set (0.001 sec)
e. 在spider1端连接远程数据库，表的存储引擎是spider
# mysql -S /tmp/mysql.sock
CREATE SERVER if not exists backend1
FOREIGN DATA WRAPPER mysql
OPTIONS(
HOST '192.168.214.133',
DATABASE 'spidertest',
USER 'spider',
PASSWORD 'spider',
PORT 3360
);
CREATE TABLE if not exists spidertest.sbtest
(
id int(10) unsigned NOT NULL AUTO_INCREMENT,
k int(10) unsigned NOT NULL DEFAULT '0',
c char(120) NOT NULL DEFAULT '',
pad char(60) NOT NULL DEFAULT '',
PRIMARY KEY (id),
KEY k (k)
) ENGINE=spider COMMENT='wrapper "mysql",srv "backend1"';
f. 在远程中插入一条数据，在本地中查看
192.168.214.133
MariaDB [test]> use spidertest;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [spidertest]> insert into sbtest values (33,5,'c','z');
Query OK, 1 row affected (0.001 sec)
192.168.214.131
MariaDB [spidertest]> select * from sbtest;
+----+---+---+-----+
| id | k | c | pad |
+----+---+---+-----+
| 1 | 2 | a | c |
| 2 | 2 | a | c |
| 33 | 5 | c | z |
+----+---+---+-----+
3 rows in set (0.008 sec)
g. 在spider1查看servers信息
MariaDB [spidertest]> select * from mysql.servers;
+-------------+-----------------+------------+----------+----------+-------+--------+---------+-------+
| Server_name | Host | Db | Username | Password | Port | Socket | Wrapper | Owner |
+-------------+-----------------+------------+----------+----------+-------+--------+---------+-------+
| Server20 | 192.168.214.133 | spidertest | spider | spider | 10482 | | mysql | |
| server2 | 192.168.214.133 | spidertest | spider | spider | 10482 | | mysql | |
| backend | 192.168.214.133 | test | spider | spider | 3360 | | mysql | |
| backend1 | 192.168.214.133 | spidertest | spider | spider | 3360 | | mysql | |
+-------------+-----------------+------------+----------+----------+-------+--------+---------+-------+
4 rows in set (0.001 sec)
3.3 分片模式
水平分表：
环境准备
| 主机名 | ip地址 | 端口号 | 数据库版本 |
|---|---|---|---|
| spider1 | 192.168.214.131 | 3360 | 10.5.3-MariaDB |
| backend1 | 192.168.214.133 | 3360 | 10.5.3-MariaDB |
| backend2 | 192.168.214.129 | 3360 | 10.5.3-MariaDB |
a. 配置好backend2环境
//在backend2上创建用户
[root@backend2 scripts]# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 5
Server version: 5.5.5-10.5.3-MariaDB MariaDB Server
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> create user 'spider'@'localhost' identified by 'spider';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on spidertest.* to 'spider'@'localhost';
Query OK, 0 rows affected (0.00 sec)
mysql> create user 'spider'@'192.168.214.%' identified by 'spider';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on spidertest.* to 'spider'@'192.168.214.%';
Query OK, 0 rows affected (0.01 sec)
//创建数据库，数据表sbtest
mysql> create database spidertest;
Query OK, 1 row affected (0.00 sec)
mysql> use spidertest;
Database changed
mysql> CREATE TABLE sbtest ( id int(10) unsigned NOT NULL AUTO_INCREMENT, k int(10) unsigned NOT NULL DEFAULT '0', c char(120) NOT NULL DEFAULT '', pad char(60) NOT NULL DEFAULT '', PRIMARY KEY (id), KEY k (k) ) ENGINE=InnoDB;
// 创建软件连接
[root@backend2 scripts]# alias backend1='/usr/local/mysql/bin/mysql --user=spider --password='spider' --host=192.168.214.133 --port=3360'
[root@backend2 scripts]# alias spider1='/usr/local/mysql/bin/mysql --user=spider --password='spider' --host=192.168.214.131 --port=3360'
[root@backend2 scripts]# alias backend2='/usr/local/mysql/bin/mysql --user=spider --password='spider' --host=192.168.214.129 --port=3360'
b. 在spider上配置远程连接backend1和backend2
CREATE SERVER if not exists backend1
FOREIGN DATA WRAPPER mysql
OPTIONS(
HOST '192.168.214.133',
DATABASE 'spidertest',
USER 'spider',
PASSWORD 'spider',
PORT 3360
);
create server if not exists backend2
foreign data wrapper mysql
options(
host '192.168.214.129',
database 'spidertest',
user 'spider',
password 'spider',
port 3360);
CREATE TABLE if not exists spidertest.sbtest
(
id int(10) unsigned NOT NULL AUTO_INCREMENT,
k int(10) unsigned NOT NULL DEFAULT '0',
c char(120) NOT NULL DEFAULT '',
pad char(60) NOT NULL DEFAULT '',
PRIMARY KEY (id),
KEY k (k)
) ENGINE=spider COMMENT='wrapper "mysql",table "sbtest"'
partition by key (id)
(
partition pt1 comment=' srv "backend1"',
partition pt2 comment=' srv "backend2"');
c. 验证spider1是否可以远程连接backend2和backend1
[root@spider1 mysql]# mysql -uspider -p -h192.168.214.131 -P3360
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 1556
Server version: 10.5.3-MariaDB-log MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]>
[root@spider1 mysql]# mysql -uspider -p -h192.168.214.129 -P3360
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 11
Server version: 10.5.3-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> exit
//如果不能连接，在backend1和backend2上配置
mysql> grant all on spidertest.* to spider@'192.168.214.129' identified by 'spider';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> select user,host from mysql.user;
+-------------+-----------------+
| User | Host |
+-------------+-----------------+
| spider | 192.168.214.% |
| spider | 192.168.214.129 |
| spider | 192.168.214.131 |
| spider | 192.168.214.133 |
| | db05 |
| | localhost |
| mariadb.sys | localhost |
| mysql | localhost |
| root | localhost |
+-------------+-----------------+
mysql> grant all on spidertest.* to spider@'192.168.214.131' identified by 'spider';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on spidertest.* to spider@'192.168.214.129' identified by 'spider';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on spidertest.* to spider@'192.168.214.133' identified by 'spider';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
d. 验证测试
在本地插入两条数据
ariaDB [spidertest]> insert into sbtest values (1,1,'e','c');
Query OK, 1 row affected (0.011 sec)
MariaDB [spidertest]> insert into sbtest values (2,1,'e','c');
Query OK, 1 row affected (0.011 sec)
在backend1中查看：
mysql> select * from sbtest;
+----+---+---+-----+
| id | k | c | pad |
+----+---+---+-----+
| 1 | 1 | e | c |
+----+---+---+-----+
2 rows in set (0.00 sec)
在backend2中查看：
mysql> select * from spidertest.sbtest;
+----+---+---+-----+
| id | k | c | pad |
+----+---+---+-----+
| 2 | 1 | e | c |
+----+---+---+-----+
2 rows in set (0.00 sec)
垂直分表：
a. 在远程服务器backend1中创建sbtest;
# backend1 <<EOF
use spidertest;
CREATE TABLE if not exists `sbtest` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`k` int(10) unsigned NOT NULL DEFAULT 0,
`c` char(120) NOT NULL DEFAULT '',
`pad` char(60) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `k` (`k`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
EOF
b. 在spider1端建立连接
//spider11
CREATE SERVER if not exists backend1
FOREIGN DATA WRAPPER mysql
OPTIONS(
HOST '192.168.214.133',
DATABASE 'spidertest',
USER 'spider',
PASSWORD 'spider',
PORT 3360
);
CREATE TABLE if not exists spidertest.sbtest
(
id int(10) unsigned NOT NULL AUTO_INCREMENT,
k int(10) unsigned NOT NULL DEFAULT '0',
c char(120) NOT NULL DEFAULT '',
pad char(60) NOT NULL DEFAULT '',
PRIMARY KEY (id),
KEY k (k)
) ENGINE=spider COMMENT='wrapper "mysql",table "sbtest",srv "backend1"';
hash分区表：
//spider1
CREATE SERVER if not exists backend1
FOREIGN DATA WRAPPER mysql
OPTIONS(
HOST '192.168.214.133',
DATABASE 'spidertest',
USER 'spider',
PASSWORD 'spider',
PORT 3360
);
CREATE SERVER if not exists backend2
FOREIGN DATA WRAPPER mysql
OPTIONS(
HOST '192.168.214.129',
DATABASE 'spidertest',
USER 'spider',
PASSWORD 'spider',
PORT 3360
);
CREATE TABLE if not exists spidertest.sbtest
(
id int(10) unsigned NOT NULL AUTO_INCREMENT,
k int(10) unsigned NOT NULL DEFAULT '0',
c char(120) NOT NULL DEFAULT '',
pad char(60) NOT NULL DEFAULT '',
PRIMARY KEY (id),
KEY k (k)
) ENGINE=spider COMMENT='wrapper "mysql",table "sbtest"'
PARTITION BY HASH (id)
( PARTITION pt1 COMMENT = 'srv "backend1',
PARTITION pt2 COMMENT = 'srv "backend2"') ;
range分区表：
CREATE TABLE if not exists spidertest.sbtest
(
id int(10) unsigned NOT NULL AUTO_INCREMENT,
k int(10) unsigned NOT NULL DEFAULT '0',
c char(120) NOT NULL DEFAULT '',
pad char(60) NOT NULL DEFAULT '',
PRIMARY KEY (id),
KEY k (k)
) ENGINE=spider COMMENT='wrapper "mysql",table "sbtest"'
PARTITION BY range(id)
( PARTITION pt1 values less than (10000) COMMENT = 'srv "backend1',
PARTITION pt2 values less than (20000) COMMENT = 'srv "backend2"') ;
4. 性能测试
把最初的sysbench表里的数据拷贝到spider表里
[root@spider1 ~]# mysqldump --user=spider --password=spider --host=192.168.214.133 --port=3360 --no-create-info spidertest sbtest >spider
查看两个节点分布的数据：
[root@spider1 ~]# backend1 -e "select count(*) from spidertest.sbtest;"
+----------+
| count(*) |
+----------+
| 1001 |
+----------+
[root@spider1 ~]# backend2 -e "select count(*) from spidertest.sbtest;"
+----------+
| count(*) |
+----------+
| 803 |
+----------+
结果如下：
[root@spider1 ~]# sysbench /usr/share/sysbench/oltp_read_only.lua --mysql-host=192.168.214.131 --mysql-user=spider --mysql-password='spider' --mysql-db=spidertest --mysql-port=3360 --table_size=2000000 --tables=5 --threads=100 --max-requests=100 run
WARNING: --max-requests is deprecated, use --events instead
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
Running the test with following options:
Number of threads: 100
Initializing random number generator from current time
Initializing worker threads...
Threads started!
SQL statistics:
queries performed:
read: 1400
write: 0
other: 200
total: 1600
transactions: 100 (373.34 per sec.)
queries: 1600 (5973.44 per sec.)
ignored errors: 0 (0.00 per sec.)
reconnects: 0 (0.00 per sec.)
General statistics:
total time: 0.2667s
total number of events: 100
Latency (ms):
min: 9.76
avg: 224.70
max: 265.92
95th percentile: 257.95
sum: 22470.10
Threads fairness:
events (avg/stddev): 1.0000/0.00
execution time (avg/stddev): 0.2247/0.05
响应时间减少到了0.05。正如预期，因为查询延迟是由多个网络往返增加的，并且条件下推尚未实现。sysbench执行了大量的范围查询。现在考虑一下，这个范围查询可能是一个比较差的优化查询。
参数详解：
[root@spider1 ~]# sysbench /usr/share/sysbench/oltp_read_only.lua \
--mysql-host=192.168.214.131 --mysql-user=spider --mysql-password='spider'\
--mysql-db=spidertest --mysql-port=3360 --table_size=2000000 --tables=5 \
--threads=100 --max-requests=100 run
# --mysql-host:mysql主机名
--mysql-user:用户名
--mysql-password:密码信息
--table-size:每张表的行数
--tables:表的数量
--threads:线程数
--max-requests:最大请求数