[Mycat] Mycat master-slave replication, vertical sub-library

1. What is Mycat.

Mycat is database middleware . The so-called middleware is a type of computer software that connects software components and applications to facilitate communication between software components.

For example, tomcat, the middleware of the web. Database middleware is software that connects Java applications and databases.

2. Why use Mycat

• Tight coupling between Java and database
• High access volume and high concurrency put pressure on the database
• Read and write request data are inconsistent

Our common Java applications are directly connected to the MySQL software for read and write operations. That is, our configuration file in Java defines the data source of mysql and is directly connected to our mysql software. However, in some cases We may need to use multiple databases. At this time, we may need to configure multiple data sources to connect to our multiple databases. At this time, it will be very troublesome when we perform sql operations, because Java and the database have a close relationship. Coupling, but if we use mycat between the Java application and mysql, we only need to access mycat. As for data sources and other issues, mycat will directly help us.

Let’s talk about high access and high concurrency. We all know that the mysql database actually has a bottleneck in data query. When we have too much data, when there are high concurrency requests on the Internet, it is very difficult for us at this time. The pressure on mysql is very high. When the number of visits is large, the data may not be found, the response time is too long, etc. At this time, we may need to have multiple servers to separate the reading and writing of the database, and to perform maintenance on the database. Cluster, at this time our sql statements need to be classified, which sql statement needs to access which database, this time just hand it over to mycat.


Finally, when using multiple databases, we will encounter a problem of inconsistent read and write data. At this time, mycat can also perform master-slave replication to ensure data consistency.

Types of database middleware:

​ 1、Cobar；2、DRDS；3、MyCat；4、Atlas；5、OneProxy。 6. Sharding-JDBC

1. Separation of reading and writing

2. Data fragmentation

Vertical split (sub-database), horizontal split (sub-table)

First, our database has multiple tables

When we have enough tables, it will also cause a bottleneck in the entire database. At this time, the query is very slow. At this time, we may need to vertically split the database, that is, sub-databases.

We need to split tables 4 5 6 vertically and put them in another database.

After we split the data vertically, the data in a single table may reach more than 10 million, which creates a bottleneck for the table. At this time, we split the table.

We can split part of the data in the table to another database.

principle

The most important verb in Mycat's principle is"interception", it intercepts the SQL statement sent by the user, and first performs some specific analysis on the SQL statement: such asSharding analysis、Route analysis、Read and write separation analysis, cache analysis, etc., and then send this SQL to the backendreal database, and perform appropriate processing on the returned results, and finally return them to the user.

Prepare environment

2 servers - master node 192.168.179.131 slave node 192.168.179.132

Mysql installation steps
http://t.csdn.cn/dAUx2

Everyone prepares two mysql servers - the master node and the slave node.

Build mysql cluster – master-slave mode

① MySQL master-slave replication principle

From a high-level perspective, replication is divided into three steps:

• When the Master database commits a transaction, it will record the data changes as Events in the binary log file Binlog.

• The master library pushes the log events in the binary log file Binlog to the relay log Relay Log of the slave library.

• The slave redoes events in the relay log, and the changes will reflect its own data.

②Copying advantages

The advantages of MySQL replication mainly include the following three aspects:

• If there is a problem with the main library, you can quickly switch to the slave library to provide services.

• You can perform query operations on the slave database and update from the master database to achieve read-write separation and reduce the access pressure on the master database.

• Backups can be performed in the slave database to avoid affecting the services of the master database during the backup.

③Building steps

【1】master

1) In the master configuration file (/etc/my.cnf), configure the following content:

#mysql 服务ID,保证整个集群环境中唯一
server-id=1

#mysql binlog 日志的存储路径和文件名
log-bin=/var/lib/mysql/mysqlbin

#错误日志,默认已经开启
#log-err

#mysql的安装目录
#basedir

#mysql的临时目录
#tmpdir

#mysql的数据存放目录
#datadir

#是否只读,1 代表只读, 0 代表读写
read-only=0

#忽略的数据, 指不需要同步的数据库
binlog-ignore-db=mysql
#指定同步的数据库
#binlog-do-db=db01

2) After execution, you need to restart Mysql:

systemctl restart mysqld

3) Create an account for synchronizing data and perform authorization operations:
set up remote access. (If you have set up to allow mysql remote access - then this step can be omitted)

grant replication slave on *.* to 'root'@'192.168.179.131' identified by 'root';	

flush privileges;

4) Check master status:

show master status;

Field meaning:

File : 从哪个日志文件开始推送日志文件 
Position ： 从哪个位置开始推送日志
Binlog_Ignore_DB : 指定不需要同步的数据库

【2】slave

1) In the slave configuration file, configure the following content:

#mysql服务端ID,唯一
server-id=2

#指定binlog日志
log-bin=/var/lib/mysql/mysqlbin

2) After execution, you need to restart Mysql:

systemctl restart mysqld

The UUID of the master and slave cannot be the same.
Query the location of the auto.cnf file

find / -name auto.cnf

Go to the folder /var/lib/mysql and delete the auto.cnf file

rm -rf auto.cnf

Just delete it and it will be automatically recreated next time. Otherwise, if the two uuid are the same, an error will be reported.
Detailed reference: https://blog.csdn.net/cnds123321/article/details/117925881

3) Execute the following instructions:

change master to master_host= '192.168.179.131', master_user='root', master_password='123456', master_log_file='mysqlbin.000002', master_log_pos=154;

master_host: IP of the master node

master_user: master node account

master_password: Password of the master node

master_log_file: the master node binary file name, consistent with the results of the master node show master status; query

master_log_pos: synchronization position

Specify the IP address, username, and password of the master database corresponding to the current slave database, and the location from which log file to start pushing the log synchronously.

4) Turn on synchronization operation

start slave;

show slave status\G;

5) Stop synchronization operation

stop slave;

reset master;

verify:

Create a library on the master node - the slave node will automatically create a corresponding library

Complete reading and writing separation through mycat

Upload mycat to the corresponding server and decompress it

Unzip

tar -zxvf Mycat-server.......（文件名）

conf directory:

1. server.xml configures virtual accounts and passwords
2. schema.xml defines the corresponding relationship between virtual libraries and real libraries
3. rule.xml fragmentation rules

1. Modify the contents of the schema.xml file

<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">

    	<!-- 
            schema:  
                   name: 定义虚拟库的名称
                   checkSQLschema: 是否检查sql语句
                   sqlMaxLimit: 设置sql语句的最大条数
                   dataNode: 指向哪个数据节点
        -->
        <schema name="TESTDB" checkSQLschema="true" sqlMaxLimit="100" dataNode="dn1">
        </schema>

        <!-- dataNode: 定义数据节点
                 name: 节点的名称-该名称必须和schema中的dataNode的值必须一致。
                         dataHost: 数据主机的名称  
                         database: 真实的mysql中的数据库名
         -->
        <dataNode name="dn1" dataHost="host1" database="mydb" />

    <!-- dataHost: 定义数据主机的相关信息
                name: 数据主机的名称 该名称必须和上面dataNode中dataHost名称一致
        -->
        <dataHost name="host1" maxCon="1000" minCon="10" balance="3"
                          writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
            <!-- heartbeat:以心跳模式监控mysql集群的主机-->
                <heartbeat>select user()</heartbeat>
                <!-- writeHost：mysql写主机的信息 -->
                <writeHost host="hostM1" url="192.168.179.131:3306" user="root"
                                   password="123456">
                                <readHost host="hostS1" url="192.168.179.132:3306" user="root" password="123456"/>
                </writeHost>
        </dataHost>

</mycat:schema>

in,Balance refers to the load balancing type. There are currently 4 values.：

1. balance=“0”, the read-write separation mechanism is not enabled, and all read operations are sent to the currently available writeHost.
2. **balance="1", **All readHost and stand by writeHost participate in the load balancing of the select statement. Simply put, when the dual master dual slave mode (M1->S1, M2->S2, and M1 and M2 interact with each other) (active/standby). Under normal circumstances, M2, S1, and S2 all participate in the load balancing of the select statement.
3. **balance="2", **All read operations are randomly distributed on writeHost and readhost.
4. **balance="3", **All read requests are randomly distributed to the readhost corresponding to writerHost for execution. writerHost does not bear the reading pressure.

Generally in enterprises, the balance setting is either 1 (dual master and dual slave) or 3 (single master and single slave).

If you want mycat to complete the read and write separation operation of sql. Change balance to 3.

2. Modify the server.xml file

<?xml version="1.0" encoding="UTF-8"?>
<!-- - - Licensed under the Apache License, Version 2.0 (the "License"); 
	- you may not use this file except in compliance with the License. - You 
	may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 
	- - Unless required by applicable law or agreed to in writing, software - 
	distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT 
	WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the 
	License for the specific language governing permissions and - limitations 
	under the License. -->
<!DOCTYPE mycat:server SYSTEM "server.dtd">
<mycat:server xmlns:mycat="http://io.mycat/">

	<!-- user:定义虚拟账户
	        name: 虚拟账户的名称
	 -->
	<user name="mycat">
		<property name="password">123456</property>
		<property name="schemas">TESTDB</property>
	</user>

</mycat:server>

3. Start mycat

bin directory

4. Testing in java

Configure data source:

spring:
  # 数据源
  datasource:
    driver-class-name: com.mysql.jdbc.Driver
    # 192.168.179.130：mycat的服务器地址 8066：mycat的默认端口 TESTDB：虚拟数据库
    url: jdbc:mysql://192.168.179.130:8066/TESTDB?serverTimezone=Asia/Shanghai
    # username：虚拟数据库名称 password：虚拟数据库密码（二者在mycat中配置）
    username: mycat
    password: 123456

Use normally when using

    @Autowired
    private StudentMapper studentMapper;
    @Test
    void contextLoads() {
    
    
        List<Student> students = studentMapper.selectList(null);
        System.out.println("student = " + students);
    }

Vertical sub-library

A database is composed of many tables, each table corresponds to different businesses. Vertical segmentation refers toAccording to the business tableClassify and distribute it to different databases, thus sharing the data or pressure to different libraries, as shown below

How to divide a table

A question: Can tables in two databases on two hosts be linked and queried?
Answer: Related queries are not possible.

The principle of database sharding: Tables that are closely related should be in one database, and tables that are not related to each other can be divided into different databases.

<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">

	<schema name="TESTDB" checkSQLschema="true" sqlMaxLimit="100" dataNode="dn1">
		  <table name="tbl_consumer" dataNode="dn2"></table>
	</schema>
	
	<dataNode name="dn1" dataHost="host1" database="orders" />
	
	<dataNode name="dn2" dataHost="host2" database="consumer" />
	
	<dataHost name="host1" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<heartbeat>select user()</heartbeat>
		<writeHost host="hostM1" url="192.168.179.131:3306" user="root"
				   password="123456">
		</writeHost>
	</dataHost>
	
	<dataHost name="host2" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="native" switchType="1"  slaveThreshold="100">
		<heartbeat>select user()</heartbeat>
		<writeHost host="hostM2" url="192.168.179.132:3306" user="root"
				   password="123456">
		</writeHost>
	</dataHost>

</mycat:schema>

The orders database is created in 192.168.179.131, and the consumer database is created in 192.168.179.132.

You need to create the database first, otherwise it doesn't seem to work (it seems like it can't be opened when connecting to mycat through navicat)

Create the corresponding table through mycat.The table must be created in mycat, because it is a sub-library specified in mycat.

#客户表 rows:20万
CREATE TABLE tbl_consumer(
 id INT AUTO_INCREMENT,
 NAME VARCHAR(200),
 PRIMARY KEY(id)
);
#订单表 rows:600万
CREATE TABLE tbl_orders(
 id INT AUTO_INCREMENT,
 order_type INT,
 customer_id INT,
 amount DECIMAL(10,2),
 PRIMARY KEY(id) 
); 
#订单详细表 rows:600万
CREATE TABLE tbl_orders_detail(
 id INT AUTO_INCREMENT,
 detail VARCHAR(2000),
 order_id INT,
 PRIMARY KEY(id)
);
#订单状态字典表 rows:20
CREATE TABLE tbl_dict_order_type(
 id INT AUTO_INCREMENT,
 order_type VARCHAR(200),
 PRIMARY KEY(id)
);

The specified tbl_consumer table is created in 192.168.179.132. Tables that are not specified default to 192.168.179.131.