Article directory

Table of contents

Article directory

foreword

1. The principle of mycat

Two, mycat completes read and write separation

2.1 Build the master-slave mode of MySQL

2.2 Realize read-write separation

3. Mycat vertical sub-library

Fourth, the level table

Five, the table connection operation of the horizontal split table

Six, the global table

Summarize

foreword

Mycat is == database middleware ==, the so-called middleware is a kind of computer software that connects software components and applications to facilitate communication between software components.

For example tomcat, web middleware. The database middleware is == software that connects Java applications and databases ==

Tight coupling of Java and database

High traffic and high concurrency put pressure on the database

Inconsistent read and write request data

Our common Java applications are directly connected to the MySQL software for read and write operations, that is, our configuration files in Java define the data source of mysql, and are directly connected to our mysql software, but when some circumstances Next, we may need to use multiple databases. At this time, we may need to configure multiple data sources to connect to our multiple databases. At this time, it will be very troublesome when we perform sql operations, because Java and the database have a close relationship. The degree of coupling, but if we use mycat between the Java application and mysql, we only need to access mycat. As for the data source and other issues, mycat will help us directly.

Let’s talk about high traffic and high concurrency. We all know that == mysql database actually has a bottleneck in data query ==. When we have too much data, when there are high concurrency requests on the Internet , At this time, the pressure on our mysql is very large. When the number of visits is large, there may be no data to be found, and the response time is too long. At this time, we may need to have == multiple servers to read the database Write separation ==, and cluster == to the == database. At this time, our sql statements need to be classified, and which sql statement needs to access which database. At this time, we only need to hand it over to the middleware.

1. The principle of mycat

The most important verb in the principle of Mycat is == "intercept" ==, which intercepts the SQL statement sent by the user, and first makes some specific analysis on the SQL statement: such as == fragmentation analysis ==, == Routing analysis ==, == read-write separation analysis ==, cache analysis, etc., and then send this SQL to the == real database == on the back end, and properly process the returned results, and finally return them to the user.

Two, mycat completes read and write separation

2.1 Build the master-slave mode of MySQL

Preparatory conditions Two virtual machines with database installed
1. Start mysql -----systemctl start mysqld
2. Check whether mysql is set up for remote connection. Navicat in windows connects to mysql.

Note that if the second virtual machine is the first clone machine with mysql installed, the following operations need to be performed

修改mysql的UUID保证不一样。
第一步 查看uuid命令要登录到mysql里面 show variables like '%server_uuid%';
第二步 找到 find / -name auto.cnf  并删除文件
第三步   重新启动mysql -----systemctl start mysqld

Build the master node

Choose a machine as the master node

 在master 的配置文件（/etc/my.cnf）中，配置如下内容：
#mysql 服务ID,保证整个集群环境中唯一
server-id=1

#mysql binlog 日志的存储路径和文件名
log-bin=/var/lib/mysql/mysqlbin

#错误日志,默认已经开启
#log-err

#mysql的安装目录
#basedir

#mysql的临时目录
#tmpdir

#mysql的数据存放目录
#datadir

#是否只读,1 代表只读, 0 代表读写
read-only=0

#忽略的数据, 指不需要同步的数据库
binlog-ignore-db=mysql

#指定同步的数据库
#binlog-do-db=db01

After execution, you need to restart Mysql: systemctl restart mysqld

View master status: show master status;

Configure slave nodes

In the /etc/my.cnf configuration file on the slave side, configure the following content:

#mysql server ID, unique
server-id=2

#Specify binlog log
log-bin=/var/lib/mysql/mysqlbin

  执行完毕之后，需要重启Mysql：
systemctl restart mysqld
然后连接到从节点的数据库里执行---指定主从关系。
change master to master_host= '192.168.91.129', master_user='root', master_password='admin', master_log_file='mysqlbin.000001', master_log_pos=154;
指定当前从库对应的主库的IP地址，用户名，密码，从哪个日志文件开始的那个位置开始同步推送日志。

Enable sync operation

start slave;

check status

show slave status\G;

stop sync operation

stop slave;
reset master;

2.2 Realize read-write separation

Open a new virtual machine with a database and install the mycat middleware to decompress it and use it

Modify the configuration file

schema.xml: Define our logic library and node information

rule.xml: define the rules for sub-tables

server.xml defines the mycat account and password

schema.xml

/*
**There are two parameters to pay attention to, balance and switchType. **

Among them, ==**balance refers to the load balancing type, and there are currently 4 values **==:

1. **balance="0",** does not enable the read-write separation mechanism, and all read operations are sent to the currently available writeHost.
2. **balance="1", **All readHost and stand by writeHost participate in the load balancing of the select statement,
* Simply put, when the dual-master and dual-slave mode (M1->S1, M2->S2, and M1 Mutual master and backup with M2), under normal circumstances, M2, S1, and S2 all participate in the load balancing of the select statement.
3. **balance="2", **All read operations are randomly distributed on writeHost and readhost.
4. **balance="3", **All read requests are randomly distributed to the readhost corresponding to the wiriterHost for execution, and the writerHost does not bear the read pressure
*/

<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
    <!-- schema:定义逻辑库的标签
	        name: 逻辑库的名称，
			checkSQLschema: 是否检查sql表结构
			sqlMaxLimit: sql最大的显示条数 根据业务以及服务器配置。
			dataNode: 数据节点的名称。 
	-->
	<schema name="TESTDB" checkSQLschema="true" sqlMaxLimit="100" dataNode="dn1">
		
	</schema>
	<!-- dataNode: 定义数据节点的。
	       name: 数据节点的名称要和上面schema标签中的dataNode属性值一致.
		   dataHost: 数据主机名
		   database: 真实数据库名
	 -->
	<dataNode name="dn1" dataHost="host1" database="qy158" />
	<!-- dataHost:定义数据主机
	        name:主机名称保证和上面dataNode标签中的datHost属性值相同
	 -->
	<dataHost name="host1" maxCon="1000" minCon="10" balance="3"
			  writeType="0" dbType="mysql" dbDriver="jdbc" switchType="1"  slaveThreshold="100">
		<!-- 主从之间的维护靠心跳 -->	  
		<heartbeat>select user()</heartbeat>
		
		<!-- 定义主节点的标签 -->
		<writeHost host="hostM1" url="192.168.223.158:3306" user="root"
				   password="123456">
			<readHost host="slave1" url="192.168.223.159:3306" user="root" password="123456"></readHost>	   
		</writeHost>
		<!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -->
	</dataHost>
	
</mycat:schema>

Modify server.xml to define logical accounts and passwords

<?xml version="1.0" encoding="UTF-8"?>
<!-- - - Licensed under the Apache License, Version 2.0 (the "License"); 
	- you may not use this file except in compliance with the License. - You 
	may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 
	- - Unless required by applicable law or agreed to in writing, software - 
	distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT 
	WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the 
	License for the specific language governing permissions and - limitations 
	under the License. -->
<!DOCTYPE mycat:server SYSTEM "server.dtd">
<mycat:server xmlns:mycat="http://io.mycat/">

	<!-- user定义逻辑账户的标签
	         name:表示逻辑账户名
	 -->
	<user name="abc">
		<!-- 逻辑密码 -->
		<property name="password">123456</property>
		<!-- 关联的逻辑库schema中的名称对应 -->
		<property name="schemas">TESTDB</property>
	</user>
</mycat:server>

Start mycat and enter the decompression directory of mycat in its bin directory./mycat console

3. Mycat vertical sub-library

A database consists of == many tables ==, each table corresponds to a different business, vertical segmentation refers to classifying the tables according to the business, and distributing them to different databases, so that the data or pressure is shared among the different libraries above,

The principle of sub-library:

A question: Can the tables in the two databases on the two hosts be associated with queries?

cannot

The principle of sub-library: tables with close relationship should be in one library, and tables that are not related to each other can be divided into different libraries.

close master-slave first

stop slave; close the master-slave relationship

Modify the mycat configuration file --schema.xml file mainly to explain which table to put in which library

<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
    <!-- schema:定义逻辑库的标签
	        name: 逻辑库的名称，
			checkSQLschema: 是否检查sql表结构
			sqlMaxLimit: sql最大的显示条数 根据业务以及服务器配置。
			dataNode: 数据节点的名称。 
	-->
	<schema name="TESTDB" checkSQLschema="true" sqlMaxLimit="100" dataNode="dn1">
		 <!-- customer表 放入dn2-->
		 <table name="customer" dataNode="dn2"></table>
	</schema>
	<!-- dataNode: 定义数据节点的。
	       name: 数据节点的名称要和上面schema标签中的dataNode属性值一致.
		   dataHost: 数据主机名
		   database: 真实数据库名
	 -->
	<dataNode name="dn1" dataHost="host1" database="mydb01" />
	<dataNode name="dn2" dataHost="host2" database="mydb02" />
	<!-- dataHost:定义数据主机
	        name:主机名称保证和上面dataNode标签中的datHost属性值相同
	 -->
	<dataHost name="host1" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<!-- 主从之间的维护靠心跳 -->	  
		<heartbeat>select user()</heartbeat>
		<!-- 定义主节点的标签 -->
		<writeHost host="hostM1" url="192.168.223.158:3306" user="root"
				   password="123456">
		</writeHost>
		<!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -->
	</dataHost>
	
	<dataHost name="host2" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<!-- 主从之间的维护靠心跳 -->	  
		<heartbeat>select user()</heartbeat>
		<!-- 定义主节点的标签 -->
		<writeHost host="hostM1" url="192.168.223.159:3306" user="root"
				   password="123456">
		</writeHost>
		<!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -->
	</dataHost>
	
</mycat:schema>

Create different libraries on the DN1 node and DN2 node respectively

dn1=====>mydb01
dn2=====>mydb02

Complete the creation of the table through mycat

mysql -uabc -P 8066 -h 192.168.223.160 -p123456

-u: mycat logical account

-P: port number of mycat

-h: the ip of mycat server

-p: logical password of mycat

#客户表 rows:20万
CREATE TABLE customer(
 id INT AUTO_INCREMENT,
 NAME VARCHAR(200),
 PRIMARY KEY(id)
);
#订单表 rows:600万
CREATE TABLE orders(
 id INT AUTO_INCREMENT,
 order_type INT,
 customer_id INT,
 amount DECIMAL(10,2),
 PRIMARY KEY(id) 
); 
#订单详细表 rows:600万
CREATE TABLE orders_detail(
 id INT AUTO_INCREMENT,
 detail VARCHAR(2000),
 order_id INT,
 PRIMARY KEY(id)
);
#订单状态字典表 rows:20
CREATE TABLE dict_order_type(
 id INT AUTO_INCREMENT,
 order_type VARCHAR(200),
 PRIMARY KEY(id)
);

Fourth, the level table

Compared with vertical splitting, horizontal splitting is not to classify tables, but to disperse them into multiple libraries according to a certain rule==of a certain field, and each==table contains a part of data==. To put it simply, we can understand the horizontal segmentation of data as == segmentation according to the data row, that is, some rows in the table are segmented into one database, and some other rows are segmented into other in the database ==

CREATE TABLE orders(

 id INT AUTO_INCREMENT,

 order_type INT,

 customer_id INT,

 amount DECIMAL(10,2),

 PRIMARY KEY(id) 

); 

按照一列进行拆分。

按照id---- 查询订单时---查询最近的订单数据。---之前的订单很少有人访问

按照订单的日期---双11 双12 ---这种日期的数据存入一张表---该表的记录会非常多。

客户id---->比较均匀的分到相应的表中，而且访问也比较均匀。

Modifying the schema.xml file is mainly to define a rule

<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
    <!-- schema:定义逻辑库的标签
	        name: 逻辑库的名称，
			checkSQLschema: 是否检查sql表结构
			sqlMaxLimit: sql最大的显示条数 根据业务以及服务器配置。
			dataNode: 数据节点的名称。 
	-->
	<schema name="TESTDB" checkSQLschema="true" sqlMaxLimit="100" dataNode="dn1">
		 <!-- customer表 放入dn2-->
		 <table name="customer" dataNode="dn2"></table>
		 <!-- rule:规则:表示orders表中的记录按照该规则分配到dn1和dn2节点上-->
		 <table name="orders" dataNode="dn1,dn2" rule="mod_rule"></table>
	</schema>
	<!-- dataNode: 定义数据节点的。
	       name: 数据节点的名称要和上面schema标签中的dataNode属性值一致.
		   dataHost: 数据主机名
		   database: 真实数据库名
	 -->
	<dataNode name="dn1" dataHost="host1" database="mydb01" />
	<dataNode name="dn2" dataHost="host2" database="mydb02" />
	<!-- dataHost:定义数据主机
	        name:主机名称保证和上面dataNode标签中的datHost属性值相同
	 -->
	<dataHost name="host1" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<!-- 主从之间的维护靠心跳 -->	  
		<heartbeat>select user()</heartbeat>
		<!-- 定义主节点的标签 -->
		<writeHost host="hostM1" url="192.168.223.158:3306" user="root"
				   password="123456">
		</writeHost>
		<!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -->
	</dataHost>
	
	<dataHost name="host2" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<!-- 主从之间的维护靠心跳 -->	  
		<heartbeat>select user()</heartbeat>
		<!-- 定义主节点的标签 -->
		<writeHost host="hostM1" url="192.168.223.159:3306" user="root"
				   password="123456">
		</writeHost>
		<!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -->
	</dataHost>
	
</mycat:schema>

Modify rule.xml

<tableRule name="mod_rule"> 
	<rule> Algorithm rule 
                <algorithm>mod-long</algorithm> 
		assigned to 
                the specified field 
		<columns>customer_id</columns> 
	</rule> 
</tableRule>

Add data to the orders table through mycat. level sub-table

Note that when inserting data into separate tables, the list of columns must be increased. Because if you don't provide it, people can't know the value corresponding to that column

Five, the table connection operation of the horizontal split table

join: join table query.

Thinking: If the order details table and the order table perform joint table query, since the records in the order table are split into two libraries, and our order details table exists in one library, then if you really want to join the table query , the order details will only be associated with the order table in the current library.

in principle:

Use the ER table to solve the problem of the associated query of the above sub-table, which depends on the storage location of the == sub-table to the main table ==, and it is physically adjacent to the storage, so it completely solves the efficiency and performance problems of JION. According to this idea , a data fragmentation strategy based on the ER relationship is proposed, and the records of the child table and the associated parent table records are stored in == the same data fragment ==.

modify schema.xml

<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
    <!-- schema:定义逻辑库的标签
	        name: 逻辑库的名称，
			checkSQLschema: 是否检查sql表结构
			sqlMaxLimit: sql最大的显示条数 根据业务以及服务器配置。
			dataNode: 数据节点的名称。 
	-->
	<schema name="TESTDB" checkSQLschema="true" sqlMaxLimit="100" dataNode="dn1">
		 <!-- customer表 放入dn2-->
		 <table name="customer" dataNode="dn2"></table>
		 <!-- rule:规则:表示orders表中的记录按照该规则分配到dn1和dn2节点上-->
		 <table name="orders" dataNode="dn1,dn2" rule="mod_rule">
			  <!-- childTable:orders的子表
			           name:子表的名称
					   primaryKey: 子表的主键
					   joinKey: 子表中的外键列
					   parentKey: 关联的父表的主键
			  -->
			  <childTable name="orders_detail" primaryKey="id" joinKey="order_id" parentKey="id"/>
		 </table>
	</schema>
	<!-- dataNode: 定义数据节点的。
	       name: 数据节点的名称要和上面schema标签中的dataNode属性值一致.
		   dataHost: 数据主机名
		   database: 真实数据库名
	 -->
	<dataNode name="dn1" dataHost="host1" database="mydb01" />
	<dataNode name="dn2" dataHost="host2" database="mydb02" />
	<!-- dataHost:定义数据主机
	        name:主机名称保证和上面dataNode标签中的datHost属性值相同
	 -->
	<dataHost name="host1" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<!-- 主从之间的维护靠心跳 -->	  
		<heartbeat>select user()</heartbeat>
		<!-- 定义主节点的标签 -->
		<writeHost host="hostM1" url="192.168.223.158:3306" user="root"
				   password="123456">
		</writeHost>
		<!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -->
	</dataHost>
	
	<dataHost name="host2" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<!-- 主从之间的维护靠心跳 -->	  
		<heartbeat>select user()</heartbeat>
		<!-- 定义主节点的标签 -->
		<writeHost host="hostM1" url="192.168.223.159:3306" user="root"
				   password="123456">
		</writeHost>
		<!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -->
	</dataHost>
	
</mycat:schema>

Create order details table on server 159

CREATE TABLE orders_detail(
id INT AUTO_INCREMENT,
detail VARCHAR(2000),
order_id INT,
PRIMARY KEY(id)
);

Add the record of the order details table through mycat

INSERT INTO orders_detail(id,detail,order_id) values(7,'detail1',1);
INSERT INTO orders_detail(id,detail,order_id) VALUES(8,'detail1',2);
INSERT INTO orders_detail(id,detail,order_id) VALUES(9,'detail1',3);
INSERT INTO orders_detail(id,detail,order_id) VALUES(10,'detail1',4);
INSERT INTO orders_detail(id,detail,order_id) VALUES(11,'detail1',5);
INSERT INTO orders_detail(id,detail,order_id) VALUES(12,'detail1',6);

Six, the global table

dict---->status such as payment status gender 0 1

Order data dictionary table.---Store order status---->Payment not paid, canceled pending shipment, shipped and confirmed. . . .

Since the order data dictionary table is needed on each node. So we define the data dictionary table as a global table.

What kind of table is suitable for global table.

Changes infrequently

The overall data volume has not changed much

The data size is small, rarely exceeding hundreds of thousands of records.

In view of this, Mycat defines a special table called "global table". The global table has the following characteristics:

The insert and update operations of the global table will be executed on == all nodes in real time == to maintain the data consistency of each shard

The query operation of the global table is only obtained from one node

A global table can perform JOIN operations with any table. Defining a dictionary table or some tables conforming to the characteristics of a dictionary table as a global table solves the problem of data JOIN from another aspect. Through the global table + fragmentation strategy based on ER relationship, Mycat can meet more than 80% of enterprise application development

modify schema.xml

<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">
    <!-- schema:定义逻辑库的标签
	        name: 逻辑库的名称，
			checkSQLschema: 是否检查sql表结构
			sqlMaxLimit: sql最大的显示条数 根据业务以及服务器配置。
			dataNode: 数据节点的名称。 
	-->
	<schema name="TESTDB" checkSQLschema="true" sqlMaxLimit="100" dataNode="dn1">
		 <!-- customer表 放入dn2-->
		 <table name="customer" dataNode="dn2"></table>
		 <!-- rule:规则:表示orders表中的记录按照该规则分配到dn1和dn2节点上-->
		 <table name="orders" dataNode="dn1,dn2" rule="mod_rule">
			  <!-- childTable:orders的子表
			           name:子表的名称
					   primaryKey: 子表的主键
					   joinKey: 子表中的外键列
					   parentKey: 关联的父表的主键
			  -->
			  <childTable name="orders_detail" primaryKey="id" joinKey="order_id" parentKey="id"/>
		 </table>
		 <!-- type:global表示全局表的意思-->
		 <table name="dict_order_type" dataNode="dn1,dn2" type="global"></table>
	</schema>
	<!-- dataNode: 定义数据节点的。
	       name: 数据节点的名称要和上面schema标签中的dataNode属性值一致.
		   dataHost: 数据主机名
		   database: 真实数据库名
	 -->
	<dataNode name="dn1" dataHost="host1" database="mydb01" />
	<dataNode name="dn2" dataHost="host2" database="mydb02" />
	<!-- dataHost:定义数据主机
	        name:主机名称保证和上面dataNode标签中的datHost属性值相同
	 -->
	<dataHost name="host1" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<!-- 主从之间的维护靠心跳 -->	  
		<heartbeat>select user()</heartbeat>
		<!-- 定义主节点的标签 -->
		<writeHost host="hostM1" url="192.168.223.158:3306" user="root"
				   password="123456">
		</writeHost>
		<!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -->
	</dataHost>
	
	<dataHost name="host2" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<!-- 主从之间的维护靠心跳 -->	  
		<heartbeat>select user()</heartbeat>
		<!-- 定义主节点的标签 -->
		<writeHost host="hostM1" url="192.168.223.159:3306" user="root"
				   password="123456">
		</writeHost>
		<!-- <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> -->
	</dataHost>
	
</mycat:schema>

Create a global table through mycat

Note that if you want to automatically lowercase the table name at this time

The first step to realize the lowercase table name is added in
vi /etc/my.cnf

#Change database case sensitivity
lower_case_table_names=1

The second step is to restart the service application and change
systemctl restart mysqld
The third step
is to change all other virtual machines with mysql to test it

The field query in the global table of mycat is tested to be case-insensitive

CREATE TABLE dict_order_type(
id INT AUTO_INCREMENT,
order_type VARCHAR(200),
PRIMARY KEY(id)
);

INSERT INTO dict_order_type(id,order_type) VALUES(101,'Pay');
INSERT INTO dict_order_type(id,order_type) VALUES(102,'NotPay');

There will be these two data in the global table

mysql optimization - mycat database middleware

Article directory

foreword

1. The principle of mycat

Two, mycat completes read and write separation

2.1 Build the master-slave mode of MySQL

2.2 Realize read-write separation

3. Mycat vertical sub-library

Fourth, the level table

Five, the table connection operation of the horizontal split table

Six, the global table

Summarize

Guess you like