MyCat separate read and write sub-library + + global table sub-table

Foreword

mycat powerful, simple configuration, as the database middleware, is a very good open source products, deployed on MyCat installation, refer Bowen: MyCat deployment installation .

This post will show the following functions mycat of:

• Separate read and write;
• Sub-library sub-table;
• Global table;

Environment are as follows:

system	IP	CPU name	service
CentOS 7.5	192.168.20.2	mysql01	MySQL 5.7.24
CentOS 7.5	192.168.20.3	mysql02	MySQL 5.7.24
CentOS 7.5	192.168.20.3	mysql03	MySQL 5.7.24
CentOS 7.5	192.168.20.4	mycat	Mycat

Note: The host mysql01 and mysql02 master-slave replication relationship , you can refer Bowen MySQL high availability solutions - double main (Note: You only need to make a reference Bowen from the effects of the main, do not need to double the Lord, you do not need keepalived high availability) to build. mycat service please refer Bowen MyCat deploy installation to build, the new host mysql03 here just to build a good mysql server, you can refer Bowen Centos deploy MySQL 5.7 to build.

1, mycat separate read and write

There is no host mysql03 used, separate read and write is easy to implement, requires only the following two modifications of the configuration files mycat. as follows:

1) modify the server.xml file

[root@mycat mycat]# pwd
/usr/local/mycat   <!--此处为mycat的家目录-->
[root@mycat mycat]# vim conf/server.xml
<!--指定client连接mycat的用户名及密码，此处的账号密码与MySQL数据库无关-->
[root@mycat mycat]# vim conf/server.xml    <!--定位到80行左右，修改如下-->
        <user name="mycat" defaultAccount="true">  <!--mycat为用户名-->
                <property name="password">pwd@123</property>   <!--此处为用户mycat的登录密码-->
                <property name="schemas">test_mycat</property>    <!--此处为逻辑库名--

                <!-- 表级 DML 权限设置 -->
                <!--            
                <privileges check="false">
                        <schema name="TESTDB" dml="0110" >
                                <table name="tb01" dml="0000"></table>
                                <table name="tb02" dml="1111"></table>
                        </schema>
                </privileges>           
                 -->
        </user>

2) modify the file schema.xml

<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
<!--下面的schema必须和第一个文件中的schema name（逻辑库名）一致。-->
        <schema name="test_mycat" checkSQLschema="false" sqlMaxLimit="100" dataNode="dn1">
        </schema>
        <dataNode name="dn1" dataHost="host1" database="test" />    <!--这里的database是指定要连接后端的哪个数据库，这里连接的是test库-->
                <!--下面是指定后端真实的MySQL主机，关于下面的balance值，有三个可选值，将在下面写下来-->
        <dataHost name="host1" maxCon="1000" minCon="10" balance="3"
                          writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
                <heartbeat>select user()</heartbeat>
                <!-- can have multi write hosts -->
                                <!--指定后端哪台主机用来写数据-->
                <writeHost host="hostM1" url="192.168.20.2:3306" user="root"
                                   password="123.com">
                        <!-- can have multi read hosts -->
                                                <!--指定后端哪台主机用来读数据-->
                        <readHost host="hostS2" url="192.168.20.3:3306" user="root" password="123.com" />
                </writeHost>
        </dataHost>
</mycat:schema>

NOTE: DataHost field balance load balancing type mentioned above, currently there are the following four values:

1. balance = "0", separate read and write mechanism is not enabled, all reads are sent to the currently available writeHost (default value).
2. balance = "1", and all readHost stand by writeHost participate in load balancing select statement, simply, when the dual master-slave mode bis (M1-> S1, M2-> S2, and M1 and M2 mutual backup) , under normal circumstances, M2, S1, S2 are involved in the select statement to load balancing.
3. balance = "2", all read operations are random in writeHost, the distribution readhost.
4. balance = "3", all read requests to a random distribution corresponding readhost wiriterHost performed, writerHost pressure without the burden of reading, only pay attention to balance = 3 and later versions 1.4, 1.3 no.

writeTyep field 3 following values:

1, writeType = "0", all write operations to the first transmission writeHost configuration, the first cut to hang still exist writeHost second, whichever is the later after the restart has been switched, the switching is recorded in the configuration file in: dnindex.properties.
2, writeType = "1", all writes are sent to writeHost random configuration, after discarding 1.5 is not recommended. switchType property

• -1 means no automatic switching.
• A default value, automatic switching.
• 2 based on the main MySQL decide whether to switch from the state of synchronization.

The revised full configuration file follows (in order to avoid redundancy, removed some comments):

server.xml configuration file as follows:

[root@mycat mycat]# vim conf/server.xml 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mycat:server SYSTEM "server.dtd">
<mycat:server xmlns:mycat="http://io.mycat/">
        <system>
        <property name="useSqlStat">0</property>  <!-- 1为开启实时统计、0为关闭 -->
        <property name="useGlobleTableCheck">0</property>  <!-- 1为开启全加班一致性检测、0为关闭 -->

                <property name="sequnceHandlerType">2</property>
                <property name="processorBufferPoolType">0</property>
                <property name="handleDistributedTransactions">0</property>

                <property name="useOffHeapForMerge">1</property>
                <property name="memoryPageSize">1m</property>
                <property name="spillsFileBufferSize">1k</property>

                <property name="useStreamOutput">0</property>
                <property name="systemReserveMemorySize">384m</property>

                <property name="useZKSwitch">true</property>

        </system>
        <user name="mycat" >
                <property name="password">pwd@123</property>
                <property name="schemas">test_mycat</property>
        </user>
</mycat:server>

schema.xml configuration file as follows:

[root@mycat mycat]# vim conf/schema.xml

<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">

        <schema name="test_mycat" checkSQLschema="false" sqlMaxLimit="100" dataNode="dn1">
        </schema>
        <dataNode name="dn1" dataHost="host1" database="test" />
        <dataHost name="host1" maxCon="1000" minCon="10" balance="3"
                          writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
                <heartbeat>select user()</heartbeat>
                <!-- can have multi write hosts -->
                <writeHost host="hostM1" url="192.168.20.2:3306" user="root"
                                   password="123.com">
                        <!-- can have multi read hosts -->
                        <readHost host="hostS2" url="192.168.20.3:3306" user="root" password="123.com" />
                </writeHost>
        </dataHost>
</mycat:schema>

Note: At this point, the back-end database to create their own user-related permissions, user such as I specified above datahost field writehost and readhost field is root, the root account database have corresponding logon rights (default root only allows users to log on localhost, so to modify their own root privileges or replaced by other authorized users).

Summary:
1. Specify the back-end database login user, you must have the appropriate permissions can;
2. specified above back-end database for the real test, so the test must have a back-end database;

3) Start and test write separation mycat

[root@mycat mycat]# mycat start       #启动mycat
[root@mycat mycat]# mysql -umycat -ppwd@123 -h 192.168.20.4 -P 8066   #登录到mycat
#192.168.20.4为mycat主机IP，mycat监听端口为8066，使用的登录用户为server.xml文件中创建的用户
#创建表并插入一些测试数据
mysql> use test_mycat;
mysql> create table t1 (id int,name varchar(4));
mysql> insert into t1 values(1,'张三'),(2,'李四'),(3,'王五');
#此时，去后端数据库上查看，应该是有上面这些数据的。
#现在可以暂时关闭后端数据库的主从复制，然后再mycat主机上继续插入数据，测试是否可以读到
#关闭主从复制命令：stop slave;
#以下操作还是在mycat主机进行
mysql> insert into t1 values(4,'赵六');    #插入一条数据
Query OK, 1 row affected (0.01 sec)

mysql> select * from t1;      #查询不到刚插入的数据，说明读写操作没有在同一台主机上进行
+------+--------+
| id   | name   |
+------+--------+
|    1 | 张三   |
|    2 | 李四   |
|    3 | 王五   |
+------+--------+
#可以去后端两个数据库分别进行查看数据
#mysql01主机的数据如下（说明写操作是在mysql01）：
mysql> select * from t1;
+------+--------+
| id   | name   |
+------+--------+
|    1 | 张三   |
|    2 | 李四   |
|    3 | 王五   |
|    4 | 赵六   |
+------+--------+
4 rows in set (0.00 sec)
#mysql02主机的数据如下：
mysql> select * from t1（说明读操作是在mysql02）;
+------+--------+
| id   | name   |
+------+--------+
|    1 | 张三   |
|    2 | 李四   |
|    3 | 王五   |
+------+--------+
3 rows in set (0.00 sec)

Read and write separation effect to this end. (Does not affect the later tests, the best recovery and mysql02 main mysql01 from host replication).

2, mycat achieve sub-libraries

A database consists of many tables, each table corresponds to a different business, so-called sub-library is classified according to the business table, do not get anything above a different database, so there is pressure to share data or to a different vault above the principle of sub-library: there is a close relationship should be associated table in a library, little or no mutual relationship table can be assigned to a different library.

Sub-libraries, for example:

Suppose there are four tables: customer, orders, orders_detail, dict_order_type, each table has millions of data, then this table as to achieve four sub-libraries, it can be separated to a separate customer database table, the other three separate tables in another database.

1) modify the server.xml file (to change a bit the logical library name)

[root@mycat mycat]# cat conf/server.xml 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE mycat:server SYSTEM "server.dtd">
<mycat:server xmlns:mycat="http://io.mycat/">
    <system>
    <property name="useSqlStat">0</property>  <!-- 1为开启实时统计、0为关闭 -->
    <property name="useGlobleTableCheck">0</property>  <!-- 1为开启全加班一致性检测、0为关闭 -->

        <property name="sequnceHandlerType">2</property>
        <property name="processorBufferPoolType">0</property>
        <property name="handleDistributedTransactions">0</property>

        <property name="useOffHeapForMerge">1</property>
        <property name="memoryPageSize">1m</property>
        <property name="spillsFileBufferSize">1k</property>

        <property name="useStreamOutput">0</property>
        <property name="systemReserveMemorySize">384m</property>

        <property name="useZKSwitch">true</property>

    </system>
    <user name="mycat" >
        <property name="password">pwd@123</property>
        <property name="schemas">test_db</property>
    </user>
</mycat:server>

2) modify the file schema.xml

[root@mycat mycat]# cat conf/schema.xml
<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">

    <schema name="test_db" checkSQLschema="false" sqlMaxLimit="100" dataNode="dn1">
        <table name="customer" dataNode="dn2"/>   <!--这里指定customer在dn2节点上-->
    </schema>
    <dataNode name="dn1" dataHost="host1" database="test" />
    <dataNode name="dn2" dataHost="host2" database="test" />
    <dataHost name="host1" maxCon="1000" minCon="10" balance="3"
              writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>select user()</heartbeat>
        <!-- can have multi write hosts -->
        <writeHost host="hostM1" url="192.168.20.2:3306" user="root"
                   password="123.com">
            <!-- can have multi read hosts -->
            <readHost host="hostS2" url="192.168.20.3:3306" user="root" password="123.com" />
        </writeHost>
    </dataHost>
    <!--下面添加host2主机字段，指定的主机就是mysql03-->
    <dataHost name="host2" maxCon="1000" minCon="10" balance="0"
              writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>select user()</heartbeat>
        <!-- can have multi write hosts -->
        <writeHost host="hostM2" url="192.168.20.21:3306" user="root"
                   password="123.com">
        </writeHost>
    </dataHost>
</mycat:schema>

3) Restart mycat

[root@mycat mycat]# mycat restart

Note: the back-end database must be specified real test database, because dataNode field above schema.xml file specifies the test is to connect back-end database.

4) testing the effect of sub-libraries

#登录到mycat
[root@mycat mycat]# mysql -umycat -ppwd@123 -h 192.168.20.4 -P 8066
#创建相应的表
mysql> use test_db;
mysql> CREATE TABLE customer(
    -> id_a INT AUTO_INCREMENT,
    -> NAME VARCHAR(200),
    -> PRIMARY KEY(id_a)
    -> );
Query OK, 0 rows affected (0.02 sec)

mysql> CREATE TABLE orders(
    -> id_b INT AUTO_INCREMENT,
    -> order_type INT,
    -> customer_id INT,
    -> amount DECIMAL(10,2),
    -> PRIMARY KEY(id_b)
    -> );
Query OK, 0 rows affected (0.01 sec)

mysql> CREATE TABLE orders_detail(
    -> id_c INT AUTO_INCREMENT,
    -> detail VARCHAR(2000),
    -> order_id INT,
    -> PRIMARY KEY(id_c)
    -> );
Query OK, 0 rows affected (0.02 sec)

mysql> CREATE TABLE dict_order_type(
    -> id_d INT AUTO_INCREMENT,
    -> order_type VARCHAR(200),
    -> PRIMARY KEY(id_d)
    -> );
Query OK, 0 rows affected (0.01 sec)

#登录到后端数据库上，查看表是否按照预计的效果拆分
[root@mycat mycat]# mysql -uroot -p123.com -h 192.168.20.2
mysql> use test;

mysql> show tables;             #可以看到主机mysql01这里有三个表
+-----------------+
| Tables_in_test  |
+-----------------+
| dict_order_type |
| orders          |
| orders_detail   |
+-----------------+
3 rows in set (0.00 sec)

#登录到mysql03上查看
[root@mycat mycat]# mysql -uroot -p123.com -h 192.168.20.21
mysql> use test;

mysql> show tables;     #可以看到这里只有customer表
+----------------+
| Tables_in_test |
+----------------+
| customer       |
+----------------+
1 row in set (0.00 sec)

So far, the library has achieved points, multiple tables in the different libraries (above chestnuts, the realization that multiple tables placed on different hosts), but for customers who log in through mycat, or a library, the library has four tables.

3, mycat achieve sub-table

Here is split to a new table emp.

1) schema.xml file is modified as follows (full profile)

[root@mycat mycat]# cat conf/schema.xml
<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">

    <schema name="test_db" checkSQLschema="false" sqlMaxLimit="100" dataNode="dn1">
        <table name="customer" dataNode="dn2"/>
        <!--只需要在原来的配置文件中增加下面一行即可，表示将emp表分在了dn1和dn2这两个节点上-->
        <table name="emp" dataNode="dn1,dn2" rule="mod_rule"/>
    </schema>
    <dataNode name="dn1" dataHost="host1" database="test" />
    <dataNode name="dn2" dataHost="host2" database="test" />
    <dataHost name="host1" maxCon="1000" minCon="10" balance="3"
              writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>select user()</heartbeat>
        <!-- can have multi write hosts -->
        <writeHost host="hostM1" url="192.168.20.2:3306" user="root"
                   password="123.com">
            <!-- can have multi read hosts -->
            <readHost host="hostS2" url="192.168.20.3:3306" user="root" password="123.com" />
        </writeHost>
    </dataHost>
    <dataHost name="host2" maxCon="1000" minCon="10" balance="0"
              writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>select user()</heartbeat>
        <!-- can have multi write hosts -->
        <writeHost host="hostM2" url="192.168.20.21:3306" user="root"
                   password="123.com">
        </writeHost>
    </dataHost>
</mycat:schema>

2) rule.xml file add the following

[root@mycat mycat]# vim conf/rule.xml      #修改rule.xml文件
        <tableRule name="mod_rule">    <!--这里指定的name必须和schema.xml中指定的name一致-->
                <rule>
                        <columns>id</columns>    <!--这里指定的是以哪里列进行分表的-->
                        <algorithm>mod_long</algorithm>  <!--这里指定的是分片方法-->
                </rule>
        </tableRule>
                        ...................
                                                <!--下面的name必须和上面的分片方法一致-->
        <function name="mod_long" class="io.mycat.route.function.PartitionByMod">
                <!-- how many data nodes -->
                <property name="count">2</property>    <!--这里指定的是node节点数量-->
        </function>

3) Restart mycat and log on to mycat

[root@mycat mycat]# mycat restart

4) Log in to the test data insertion mycat effect sub-table

[root@mycat mycat]# mysql -umycat -ppwd@123 -h 192.168.20.4 -P 8066
mysql> use test_db;
#创建emp表（后端两个数据库节点上都会有这张表）
mysql> create table emp (
    -> id int,
    -> name varchar(10)
    -> );
Query OK, 0 rows affected (0.41 sec)
#插入多条数据（对于这种分表，插入数据需要指定列名，否则会报错）
mysql> insert into emp(id,name) values(1,'张三'),(2,'李四'),(3,'王五'),(4,'赵六');
#查询新插入的数据
mysql> select * from emp;    
+------+--------+
| id   | name   |
+------+--------+
|    2 | 李四   |
|    4 | 赵六   |
|    1 | 张三   |
|    3 | 王五   |
+------+--------+
4 rows in set (0.04 sec)
mysql> select * from emp order by id;    #对结果进行排序
+------+--------+
| id   | name   |
+------+--------+
|    1 | 张三   |
|    2 | 李四   |
|    3 | 王五   |
|    4 | 赵六   |
+------+--------+
4 rows in set (0.01 sec)

#后端mysql01主机查询到该表的数据如下：
mysql> select * from emp;
+------+--------+
| id   | name   |
+------+--------+
|    2 | 李四   |
|    4 | 赵六   |
+------+--------+
2 rows in set (0.00 sec)
#mysql03主机查询到的数据如下：
mysql> select * from emp;          
+------+--------+
| id   | name   |
+------+--------+
|    1 | 张三   |
|    3 | 王五   |
+------+--------+
2 rows in set (0.00 sec)

You can see the data is the average spread over different nodes to save.

But after ordinary table fragmentation, and can not directly join the other tables.

To solve the problem of multi-table join, schema.xml also need to modify the configuration file, add childTable field in the table field points table, how to achieve specific, still studying it. However, the type of part tables may be set to the global table, it is a relatively simple implementation.

3, global table

After the points table, how to join with other tables became more difficult problem, we can choose to divide the table set the global table, which is a method (but not the only way) to solve a join, but there are certain limitations global table sex, as follows:

• Global table changes frequently;
• Little change in the overall amount of data;
• Data is small, few ultra hundreds of thousands of records.

Global table has the following features:

• Inserting the global table, the update operation will be performed in real time on all nodes, data consistency of each fragment;
• Global table query, only get from a node;
• Global table JOIN operation can be performed with any one table.

If you want to create a global table, the table field just increase type = "global" can be. as follows:

1) Define the global table

[root@mycat conf]# cat schema.xml
<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">

    <schema name="test_db" checkSQLschema="false" sqlMaxLimit="100" dataNode="dn1">
        <table name="customer" dataNode="dn2"/>
        <!--在下面的table字段增加global属性即可-->
        <table name="emp" dataNode="dn1,dn2" type="global">
        </table>
    </schema>
    <dataNode name="dn1" dataHost="host1" database="test" />
    <dataNode name="dn2" dataHost="host2" database="test" />
    <dataHost name="host1" maxCon="1000" minCon="10" balance="3"
              writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>select user()</heartbeat>
        <!-- can have multi write hosts -->
        <writeHost host="hostM1" url="192.168.20.2:3306" user="root"
                   password="123.com">
            <!-- can have multi read hosts -->
            <readHost host="hostS2" url="192.168.20.3:3306" user="root" password="123.com" />
        </writeHost>
    </dataHost>
    <dataHost name="host2" maxCon="1000" minCon="10" balance="0"
              writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
        <heartbeat>select user()</heartbeat>
        <!-- can have multi write hosts -->
        <writeHost host="hostM2" url="192.168.20.21:3306" user="root"
                   password="123.com">
        </writeHost>
    </dataHost>
</mycat:schema>

2) Verify that the global table

mysql> drop table emp;    #需要删除之前创建的emp表
Query OK, 0 rows affected (0.03 sec)
#重新创建emp表
mysql> create table emp(
    -> id int,
    -> name varchar(10)
    -> );
Query OK, 0 rows affected (0.38 sec)
#插入数据测试
mysql>  insert into emp(id,name) values(1,'张三'),(2,'李四'),(3,'王五'),(4,'赵六');
#在mycat上查询插入的数据如下：
mysql> select * from emp;
+------+--------+
| id   | name   |
+------+--------+
|    1 | 张三   |
|    2 | 李四   |
|    3 | 王五   |
|    4 | 赵六   |
+------+--------+
4 rows in set (0.02 sec)
#在后端主机上分别查看emp表中的数据：
#mysql01主机上数据如下：
mysql> select * from emp;
+------+--------+
| id   | name   |
+------+--------+
|    1 | 张三   |
|    2 | 李四   |
|    3 | 王五   |
|    4 | 赵六   |
+------+--------+
4 rows in set (0.00 sec)
#mysql03主机上数据如下：
mysql> select * from emp;
+------+--------+
| id   | name   |
+------+--------+
|    1 | 张三   |
|    2 | 李四   |
|    3 | 王五   |
|    4 | 赵六   |
+------+--------+
4 rows in set (0.00 sec)