23_MySQL middleware mycat-01

mycat basic concepts

1. What is MyCat

1. A completely open source large database cluster for enterprise application development
2. Support transactions, ACID, and an enhanced database that can replace MySQL
3. An enterprise-level database that can be regarded as a MySQL cluster to replace the expensive Oracle cluster
4. A new SQL Server integrating memory caching technology, NoSQL technology, and HDFS big data
5. A new generation of enterprise-level database products that combine traditional databases and new distributed data warehouses
6. A novel database middleware product

2. Why use MyCat

Nowadays, with the development of the Internet, the magnitude of data also supports exponential growth, from GB to TB to PB. Various operations on data are becoming more and more difficult. Traditional relational databases can no longer meet the needs of fast query and data insertion. At this time, the emergence of NoSQL temporarily solved this crisis. It achieves performance improvements by reducing data security, reducing support for transactions, and reducing support for complex queries. However, in some occasions, NoSQL compromises cannot satisfy usage scenarios. For example, some usage scenarios absolutely require transaction and security indicators. NoSQL is definitely not enough at this time, so you still need to use a relational database. How to use relational database to solve the problem of mass storage? At this time, a database cluster is needed. In order to improve query performance, the data of one database is distributed to different databases for storage. In response to this problem, MyCat

To sum up: Mycat's role is: can meet the large storage of database data; improve query performance

1. Read and write separation
2. Data slicing vertical split (sub-database), horizontal split (sub-table), vertical + horizontal split (sub-database and sub-table)
3. Multi-data source integration

3. Comparison of database middleware

① Cobar belongs to the Alibaba B2B business group. It started in 2008 and has served in Alibaba for more than 3 years. It has taken over 3000+ MySQL
database schemas, and the cluster processes more than 5 billion online SQL requests per day. Due to the resignation of Cobar promoters, Cobar ceased maintenance. ② Mycat is the secondary development of the open source community on the basis of Ali cobar, which solves the problems of cobar and adds many new functions to it. the student surpasses the master.
③ OneProxy is developed based on MySQL's official proxy idea using c. OneProxy is a commercial fee-based middleware
. Abandoning some functions, focus on performance and stability.
④ Kingshard is developed by a small team in go language, and it needs to be developed and continuously improved.
⑤ Vitess is used in Youtube production, and its structure is very complicated. The MySQL native protocol is not supported, and the use requires a lot of transformation costs.
⑥Atlas is a rewrite of the 360 ​​team based on mysql proxy. The function needs to be improved, and it is unstable under high concurrency.
⑦ MaxScale is a middleware developed by mariadb (a version maintained by the original author of MySQL)
⑧ MySQLRoute is a middleware released by the official MySQL company Oracle

4. The core technology (sub-database and sub-table)
database sharding refers to: through a certain condition, the data we store in one database is scattered and stored in different multiple databases (hosts), so as to achieve a decentralized single According to the slicing rules, the load of the device can be divided into the following two slicing modes. MyCAT implements sharding by defining table sharding rules. Each table can be bundled with a sharding rule, and each sharding rule specifies a sharding field and Bind a function to implement the dynamic fragmentation algorithm.

1. Schema: logic library, corresponding to the Database in MySQL, the included Table is defined in a logic library.
2. Table: Logical table, that is, a table stored in a physical database. Unlike traditional databases, the table here needs to declare the logical data node DataNode stored in it. Here you can specify the fragmentation rules of the table.
3. DataNode: the logical data node of MyCAT, is the specific physical node that stores the table, also called the shard node, which is related to a specific back-end database through the DataSource. 4. DataSource
: defines the access address of a physical library , Used to bind to Datanode
5. Fragmentation rules: Data segmentation was mentioned earlier. If a large table is divided into several fragmentation tables, certain rules are required to divide the data into a certain segment according to certain business rules. The rule of sharding is the sharding rule. It is very important to select the appropriate sharding rule for data segmentation, which will greatly avoid the difficulty of subsequent data processing.

Mycat installation and deployment

surroundings

CPU name	IP address
master	192.168.1.65
slave	192.168.1.67
mycat	192.168.1.1

1. Configure master-slave replication
//operate
on the master host, turn on the binary log

[root@master ~]# cat /etc/my.cnf 
[mysqld]
.....
server_id = 1 //不可和其他主机一直
log-bin = mysql-bin
!重启MySQL

Create user

mysql> grant replication slave on *.* to rep@'192.168.1.%' identified by '123.com';
Query OK, 0 rows affected, 1 warning (0.00 sec)

Check the master binary file and location, which will be used later

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000001 |      446 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

//slave host to operate

[mysqld]
...
server_id = 2 //不可与其他主机一致
relay_log= relay_log  //中继日志
！重启MySQL

mysql> change master to
    -> master_host='192.168.1.65',
    -> master_user='rep',
    -> master_password='123.com',
    -> master_log_file='mysql-bin.000001',
    -> master_log_pos=446;
Query OK, 0 rows affected, 2 warnings (0.01 sec)

mysql> start slave;
Query OK, 0 rows affected (0.01 sec)

View slave status requirements

mysql> show slave status\G
......
 Slave_IO_Running: Yes
 Slav_SQL_Running: Yes

1. Prerequisites for deploying mycat

1. jdk: requires jdk to be version 1.7 and above
2. Mysql: recommended mysql is 5.5 or above
3. Mycat: Mycat's official website: http://www.mycat.org.cn/ Download address: https://github.com/MyCATApache/Mycatdownload Mycat has multiple versions of windows and linux.

2. Download and deploy mycat

[root@mycat ~]# cd /usr/local/
[root@mycat local]# wget http://dl.mycat.org.cn/1.6.7.6/20201126013625/Mycat-server-1.6.7.6-release-20201126013625-linux.tar.gz

//Install jdk environment

[root@mycat ~]# yum -y install java-devel

[root@mycat ~]# cd /usr/local/
[root@mycat local]# tar zxf Mycat-server-1.6.7.6-release-202011260

//Optimize the path

[root@mycat ~]# ln -s /usr/local/mycat/bin/* /usr/local/bin/

3. Create a library on the master host
// for specifying in the mycat host configuration file

mysql> create database qin;
mysql> create table qin.t1 (id int primary key auto_increment,
    -> name char(22)
    -> );
mysql> insert into qin.t1(name) values ('zhangsan'),('lisi') ;
Query OK, 2 rows affected (0.06 sec)
Records: 2  Duplicates: 0  Warnings: 0

mysql> select * from qin.t1;
+----+----------+
| id | name     |
+----+----------+
|  1 | zhangsan |
|  2 | lisi     |
+----+----------+

3. Modify the configuration file
//modify the server.xml file

[root@mycat ~]# vim /usr/local/mycat/conf/server.xml
......
        <user name="mycat" defaultAccount="true">  //更改用户名，可不改
                <property name="password">123456</property>
                <property name="schemas">TESTDB</property>
                <property name="defaultSchema">TESTDB</property>

//Modify the schema.xml file

[root@mycat ~]# vim /usr/local/mycat/conf/schema.xml
<?xml version="1.0"?>
<!DOCTYPE mycat:schema SYSTEM "schema.dtd">
<mycat:schema xmlns:mycat="http://io.mycat/">

	<schema name="TESTDB" checkSQLschema="true" sqlMaxLimit="100" dataNode="dn1">
	</schema>
	<dataNode name="dn1" dataHost="host1" database="qin" /> //指定qin库
	<dataHost name="host1" maxCon="1000" minCon="10" balance="0"
			  writeType="0" dbType="mysql" dbDriver="jdbc" switchType="1"  slaveThreshold="100">
		<heartbeat>select user()</heartbeat>

		<writeHost host="hostM1" url="jdbc:mysql://192.168.1.65:3306" user="root"
				   password="123.com"> //指定master的ip和登入mysql的root用户和密码
		</writeHost>
	</dataHost>
</mycat:schema>

//Start mycat
{console | start | stop | restart | status |dump} The default port number of Mycat is: 8066
if it is running in the background, the start command can be used

[root@mycat ~]# mycat console
Running Mycat-server...
wrapper  | --> Wrapper Started as Console
wrapper  | Launching a JVM...
jvm 1    | Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
jvm 1    |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
jvm 1    | 
jvm 1    | MyCAT Server startup successfully. see logs in logs/mycat.log

4. The scp mysql command to the mycat host.
Since mycat does not have mysql installed but needs to use the mysql login command, please copy the mysql login command to the mycat host on any mysql host.

[root@slave ~]# scp /usr/local/mysql/bin/mysql [email protected]:/usr/local/bin/

5. Mycat host test login

[root@mycat ~]# mysql  -umycat -p123456 -h 192.168.1.61 -P 8066

6, any client test login

[root@slave ~]# mysql  -umycat -p123456 -h 192.168.1.61 -P 8066

7. Set root authority
root only has root authority locally, so you need to create a remote root account to facilitate changing data in the database. If you do not have authority to log in successfully on the client side, you can't change the data at all.

mysql> grant all on *.* to root@'192.168.1.%' identified by '123.com';
Query OK, 0 rows affected, 1 warning (0.00 sec)

8. Insert data test
//Client insert data

[root@slave ~]# mysql  -umycat -p123456 -h 192.168.1.61 -P 8066

mysql> show databases;
+----------+
| DATABASE |
+----------+
| TESTDB   |
+----------+
1 row in set (0.01 sec)

mysql> use TESTDB;
Database changed
mysql> show tables;
+---------------+
| Tables_in_qin |
+---------------+
| t1            |
+---------------+
1 row in set (0.02 sec)

mysql> insert into t1(name) values('zi');
Query OK, 1 row affected (0.08 sec)
 OK!

mysql> select * from t1;
+------+----------+
| id   | name     |
+------+----------+
|    1 | zhangsan |
|    2 | lisi     |
|    3 | zi       |
+------+----------+
3 rows in set (0.02 sec)

//master host view verification

mysql> select * from qin.t1;
+----+----------+
| id | name     |
+----+----------+
|  1 | zhangsan |
|  2 | lisi     |
|  3 | zi       |
+----+----------+
3 rows in set (0.00 sec)