MySQL study notes: Principles and construction of mycat2

mycat2

Official website: MyCat2 (mycatone.top)

Refer to the learning video. This may be the only tutorial on Bilibili that explains the database middleware Mycat2 in such an in-depth and thorough manner! _bilibili_bilibili

What is mycat2?

Mycat is a database middleware developed by the open source community on the basis of Alibaba Cobar . It is also the most popular database middleware written in Java language . It is an open source distributed database system and a server that implements the MySQL protocol. , front-end users can think of it as a database proxy, using MySQL client tools and command line access (SQLyog, navigate), while its back-end can use the MySQL native (Native) protocol to communicate with multiple MySQL servers, and can also use The JDBC protocol communicates with most mainstream database servers. Its core function is table and database partitioning, that is, a large table is horizontally divided into N small tables and stored in the back-end MySQL server or other databases. In conjunction with the master-slave mode of the database, read-write separation can also be achieved

Mycat has developed to the current version and is no longer a pure MySQL agent. Its backend can support mainstream databases such as MySQL, SQL Server, Oracle, DB2, PostgreSQL, etc. It also supports MongoDB, a new type of NoSQL storage , and will continue to do so in the future . Supports more types of storage.

In terms of working method, Mycat will be deployed as a MySQL service. Programmers only need to operate mycat like an ordinary stand-alone MySQL service. Mycat will then complete the sub-database and table functions. As for the back-end database product, it is recommended to use MySQL.

Why use mycat2?

1. High access volume and high concurrency put pressure on the database
2. Read and write request data are inconsistent

Function

1. Separation of reading and writing

2. Sub-database and sub-table

3. Integration of multiple data sources

principle

The most important verb in the principle of mycat is interception. It intercepts the SQL statement sent by the user. First, it performs some specific analysis on the SQL statement: such as fragmentation analysis, routing analysis, read-write separation analysis, cache analysis, etc. , and then send this SQL to the back-end real database, perform appropriate processing on the returned results, and finally return them to the user.

Related concepts

1. Sub-database and sub-table

According to certain rules, the tables in the database are split into multiple sub-tables with database instances, physical libraries, and physical table access paths.

Sub-library: An e-commerce project is divided into user database, order database, etc.

Table splitting: One order table contains millions of data, reaching the MySQL single table bottleneck and splitting it into multiple tables in multiple databases.

2. Logic library

A database in a database agent, which can contain multiple logical tables

The tables defined in mycat exist logically, but do not physically exist in MySQL. It is possible that multiple MySQL databases together form a logical library.

A mapping to the real MySQL database on the backend

3. Logic table

The tables in the database agent can map the physical tables in the database that the agent connects to, either one-to-one or one-to-many.

4.Physical library

MySQL real database

5.Physical table

Real data table in MySQL real database

6. Split key

Sharding key, a field that describes the data rules for splitting a logical table

For example, the order table can be split according to the user ID it belongs to, and the user ID is the split key.

7. Prototype library (prototype)

The prototype library is the database behind mycat2, such as the mysql library

The prototype library is the real database that stores data. The prototype library must be specified when configuring the data source.

Installation and deployment

1. Download the source code package from the official website

Core source package jar package Index of /2.0/1.22-release/ (mycat.org.cn)

Shell zip package Index of /2.0/install-template/ (mycat.org.cn)

2. Source code package integration

Unzip the zip package and put the jar package into the /mycat/lib directory of the decompressed zip package.

3. Upload the integrated mycat source code package to the /usr/local directory of Linux

[root@mysqlrouter1 lib]# ls
libwrapper-aix-ppc-32.a       libwrapper-linux-x86-32.so             libwrapper-macosx-universal-64.jnilib  mycat2-1.22-release-jar-with-dependencies-2022-10-13.jar
libwrapper-aix-ppc-64.a       libwrapper-linux-x86-64.so             libwrapper-solaris-sparc-32.so         wrapper.jar
libwrapper-hpux-parisc-64.sl  libwrapper-macosx-ppc-32.jnilib        libwrapper-solaris-sparc-64.so         wrapper-windows-x86-32.dll
libwrapper-linux-ppc-64.so    libwrapper-macosx-universal-32.jnilib  libwrapper-solaris-x86-32.so           wrapper-windows-x86-64.dll

4. Modify file permissions to prevent startup errors

[root@mysqlrouter1 mycat]# cd bin/
[root@mysqlrouter1 bin]# ls
mycat               wrapper-aix-ppc-64      wrapper-linux-x86-32   wrapper-macosx-universal-32  wrapper-solaris-sparc-64    wrapper-windows-x86-64.exe
mycat.bat           wrapper-hpux-parisc-64  wrapper-linux-x86-64   wrapper-macosx-universal-64  wrapper-solaris-x86-32
wrapper-aix-ppc-32  wrapper-linux-ppc-64    wrapper-macosx-ppc-32  wrapper-solaris-sparc-32     wrapper-windows-x86-32.exe
[root@mysqlrouter1 bin]# chmod 777 mycat
[root@mysqlrouter1 bin]# chmod 777 wrapper-linux-x86-64 
[root@mysqlrouter1 bin]# chmod 777 wrapper-linux-x86-32 
[root@mysqlrouter1 bin]# chmod 777 wrapper-linux-ppc-64 

5. Add authorized users to the MySQL database connected to mycat

Create a user with the username mycat, password 123456, and grant permissions·

root@(none) 01:14  mysql>create user 'mycat'@'%' identified by '123456';
Query OK, 0 rows affected (0.00 sec)

root@(none) 01:15  mysql>grant all on *.* to 'mycat'@'%';
Query OK, 0 rows affected (0.00 sec)

root@(none) 01:16  mysql>flush privileges;
Query OK, 0 rows affected (0.00 sec)

6. Modify the configuration of mycat prototype

Mycat2's native mysql environment configuration, because mycat2 is deployed as a mysql service, operating mycat2 is just like operating mysql

Before starting mycat, you need to confirm the mysql database configuration corresponding to the prototype data source, modify the corresponding user user, password, and ip in the url.

[root@mysqlrouter1 datasources]# vim prototypeDs.datasource.json 

{
        "dbType":"mysql",
        "idleTimeout":60000,
        "initSqls":[],
        "initSqlsGetConnection":true,
        "instanceType":"READ_WRITE",
        "maxCon":1000,
        "maxConnectTimeout":3000,
        "maxRetryCount":5,
        "minCon":1,
        "name":"prototypeDs",
        "password":"123456",---连接本机mysql数据库的用户密码
        "type":"JDBC",
        "url":"jdbc:mysql://192.168.31.153:3306/mydb1?useUnicode=true&serverTimezone=Asia/Shanghai&characterEncoding=UTF-8",---指定mysql主机的ip地址端口号和数据库
        "user":"mycat", ---远程连接mysql数据库的用户
        "weight":0
}

7. Test the connection to the remote user

[root@mysqlrouter1 datasources]# mysql -umycat -p'123456' -h 192.168.31.111
[root@mysqlrouter1 datasources]# mysql -umycat -p'123456' -h 192.168.31.153

8. Start mycat

[root@mysqlrouter1 bin]# pwd
/usr/local/mycat/bin
[root@mysqlrouter1 bin]# ./mycat start
Starting mycat2...

Configuration file

1. Service (server)

Service-related configuration (1) The default configuration
in the directory
mycat/conf is sufficient

2. User

Configure user-related information
(1) Directory
mycat/conf/users
(2) Naming method
{username}.user.json
(3) Configuration content

vim mycat/conf/users/root.user.json
{
"ip":null,
"password":"123456",
"transactionType":"xa",
"username":"root",
"isolation":3
}

#Field meaning
#ip: access the IP of the local mycat2 client. It is recommended to be empty. After filling in, the client IP will be restricted. Only any IP of the local machine can log in.

username: username, default is root

password: password, default is 123456

Isolation: Set the initial transaction isolation level

READ_UNCOMMITTED:1
READ_COMMITTED:2
REPEATED_READ:3,默认
SERIALIZABLE:4

transactionType: transaction type

Optional value:
proxy local transaction. In transactions involving more than 1 database, failure in the commit phase will lead to inconsistency, but
the compatibility best
. XA transaction, you need to confirm whether the storage node cluster type supports XA.
You can switch through the statement
set transaction_policy = ' xa'
set transaction_policy = 'proxy'
can be queried through the statement
SELECT @@transaction_policy

3. Data source

Configure data source information for Mycat connection

directory

mycat/conf/datasources

naming method

{data source name}.datasource.json

Configuration content

vim mycat/conf/datasources/ prototype. datasources.json
{
"dbType": "mysql",
"idleTimeout": 60000,
"initSqls": [],
"initSqlsGetConnection": true,
"instanceType": "READ_WRITE",
"maxCon": 1000,
"maxConnectTimeout": 3000,
"maxRetryCount": 5,
"minCon": 1,
"name": "prototype",
"password": "123456",
"type": "JDBC",
"url":
"jdbc:mysql://127.0.0.1:3306/mysql?useUnicode=true&serverTimezone=UTC",
"user": "root",
"weight": 0,
"queryTimeout":30,//mills
}

Field meaning

dbType: database type, mysql

name: Data source name, used when adding data sources to the cluster

Password: user password to connect to the real database on the backend

type: data source type, default JDBC

url: access database address, ip + port + specific database table (you can also ignore it)

idleTimeout: idle connection timeout

initSqls: initialize sql

initSqlsGetConnection: For jdbc, whether to execute initSqls every time to get a connection

instanceType: configure whether the instance is read-only or read-write

Note: In mycat2, when adding data source information in the cluster, when the data source instanceType is read and write, set it to the master node, then it can read and write; if it is set to slave, then it can only read but not write. That is to say, no matter what the instanceType is configured to, if the data source is used as a slave node, it can only read and write.

Optional values:
READ_WRITE,READ,WRITE
weight: load balancing weight

Connection related configuration

“maxCon”: 1000,
“maxConnectTimeout”: 3000,
“maxRetryCount”: 5,
“minCon”: 1,

4. Cluster

Configure cluster information

directory

mycat/conf/clusters

naming method

{cluster name}.cluster.json

Configuration content

vim mycat/conf/clusters/prototype.cluster.json
{
"clusterType":"MASTER_SLAVE",
"heartbeat":{
"heartbeatTimeout":1000,
"maxRetryCount":3,//2021-6-4前是maxRetry，后更正为
maxRetryCount
"minSwitchTimeInterval":300,
"slaveThreshold":0
},
"masters":[ //配置多个主节点,在主挂的时候会选一个检测存活的数据源作
为主节点
"prototypeDs"
],
"replicas":[//配置多个从节点
"xxxx"
],
"maxCon":200,
"name":"prototype",
"readBalanceType":"BALANCE_ALL",
"switchType":"SWITCH"
可选
//
,
"timer":{ //MySQL集群心跳周期,配置则开启集群心跳,Mycat主动检测主从延迟以
及高可用主从切换
"initialDelay": 30,
"period":5,
"timeUnit":"SECONDS"
},
//readBalanceName:"BALANCE_ALL",
//writeBalanceName:"BALANCE_ALL",
}

Field meaning

clusterType: cluster type

Optional values:
SINGLE_NODE: single node
MASTER_SLAVE: common master-slave
GARELA_CLUSTER: garela cluster/PXC cluster
MHA: MHA cluster
MGR: MGR cluster

readBalanceType: Query load balancing strategy

Optional values:
BALANCE_ALL (default value)
Gets all data sources in the cluster
BALANCE_ALL_READ
Gets the data sources that are allowed to be read in the cluster
BALANCE_READ_WRITE
Gets the data sources that are allowed to be read and written in the cluster, but the data sources that are allowed to be read take precedence
BALANCE_NONE
Gets the data sources that are allowed to be written in the cluster , that is, select from the master node

switchType: switch type

Optional values:
NOT_SWITCH: Do not perform master-slave switching
SWITCH: Perform master-slave switching

4. Logical library table (schema)

Configure logical library tables to implement sub-database and sub-table

directory

mycat/conf/schemas

naming method

{Library name}.schema.json

Configuration content

vim mycat/conf/schemas/mydb1.schema.json
#库配置
{
"schemaName": "mydb",
"targetName": "prototype"
}

schemaName: logical library name

targetName: destination data source or cluster

targetName automatically loads the physical table or view under the test library as a single table from the prototype target. The prototype
must be a mysql server.