MyCAT is an open source distributed database component. In the project, this component is generally used to realize the function of sub-database and sub-table for the database, thereby improving the access performance of data tables, especially large database tables. Moreover, in actual projects, MyCAT sub-database and sub-table components are generally integrated with MySQL and Redis components, so that the data access performance can be improved from the two dimensions of "reducing the size of data in the data table" and "caching data". .
1 Overview of sub-library sub-table
Let’s first look at the concept of sub-database and sub-table through an example. For example, in an e-commerce system, there is a flow table whose primary key is id. If the business volume of the e-commerce system is large, this flow table is very likely. Reach the scale of "100 million" or even larger. If you want to query data from this table, even if you use indexes and other database optimization measures, after all, the scale of the data table is too large, which will become a performance bottleneck, so you can split this large flow table according to the following ideas .
1 In 10 different databases, create these 10 running tables at the same time, and the table structure of these tables is exactly the same.
2 In database No. 1, store only the flow records with id%10 equal to 1, for example, store the flow records with id 1, 11, and 21, and store only the flow records with id%10 equal to 2 in the No. 2 database. analogy.
In other words, through the above steps, this flow table can be split into 10 word tables, and the MyCAT component can disperse the application's request for the flow table into 10 sub tables. The specific effect is shown in the figure below.
In actual projects, the number of sub-tables can be set according to actual needs. Since the data of a large table is dispersed into several sub-tables, the total amount of data faced by each data request can be effectively reduced. From this, everyone can feel the help of the "split table" approach to improve database access performance.
And in actual projects, the sub-tables will be created on different hosts as much as possible, rather than simply creating multiple sub-tables on the same host and the same database, that is to say, these sub-tables need to be scattered to different On the database, the specific effect is shown in the figure below.
Try to "sub-database" the sub-table is still out of the consideration of improving performance. Because a single database always has a performance bottleneck when processing requests, for example, it can handle up to 500 requests per second. If these sub-tables are placed on the same database on the same host, the request speed for the table still cannot break through the performance bottleneck of a single database. But if these sub-tables are distributed to different databases on different hosts, then the request for the table is equivalent to being effectively allocated to different databases, so that the effective load of the database can be increased by n times.
In actual projects, due to cost considerations, it may not be possible to allocate a host for each sub-table. In this case, you can put a second place to it. You can create different sub-tables in different databases on the same host. In short, try not to create different subtables on the same host and the same database.
In other words, through "splitting", the data size of large tables can be effectively reduced, and through "splitting", multiple databases can be integrated, which can increase the effective load of processing requests. The MyCAT distributed database component realizes the effect of "sub-database and sub-table", so it is usually called "MyCAT sub-database and sub-table component".
In fact, the MyCAT component can parse SQL statements and send the SQL to the corresponding sub-table for execution according to the preset sub-database fields and sub-database rules, and then return the executed results to the application program.
2 Use MyCAT component to realize sub-database and sub-table
As mentioned above, MyCAT can be used to achieve the effect of sub-database and sub-table. This component works on port 8066 by default. The relationship between it and the application and database is shown in the figure below. As you can see, Java applications are not directly connected to databases such as MySQL, but to MyCAT components. The application program sends the SQL request to MyCAT, and MyCAT sends the request to the corresponding database according to the configured sub-database and sub-table rules, and then returns the request to the application after the request is obtained.
In order to achieve the effect of sub-library and sub-table, it is generally necessary to configure the three files shown in the following table in the MyCAT component.
Here will take a MyCAT component to connect to three databases as an example to give specific examples of writing the above three configuration files.
First, the code of the server.xml configuration file is shown below.
1 <?xml version="1.0" encoding="UTF-8"?> 2 <!DOCTYPE mycat:server SYSTEM "server.dtd"> 3 <mycat:server xmlns:mycat="http://io.mycat/"> 4 <system> 5 <property name="serverPort">8066</property> 6 <property name="managerPort">9066</property> 7 </system> 8 <user name="root"> 9 <property name="password">123456</property> 10 <property name="schemas">redisDemo</property> 11 </user> 12 </mycat:server>
In lines 5 and 6, the working port and management port of the MyCAT component are configured as 8066 and 9066, respectively. In the code from lines 8 to 11, the user name for connecting the MyCAT component is configured as root , The connection password is 123456, and at the same time, after the root login, you can access the redisDemo database in the MyCAT component.
Please note that redisDemo is the database of MyCAT component, not MySQL. In practice, this database generally has the same name as MySQL.
Second, the code of the schema.xml configuration file is shown below.
1 <?xml version="1.0"?> 2 <!DOCTYPE mycat:schema SYSTEM "schema.dtd"> 3 <mycat:schema xmlns:mycat="http://io.mycat/"> 4 <schema name="redisDemo"> 5 <table name="student" dataNode="dn1,dn2,dn3" rule="mod-long"/> 6 </schema> 7 <dataNode name="dn1" dataHost="host1" database="redisDemo" /> 8 <dataNode name="dn2" dataHost="host2" database="redisDemo" /> 9 <dataNode name="dn3" dataHost="host3" database="redisDemo" /> 10 <dataHost name="host1" dbType="mysql" maxCon="10" minCon="3" balance="0" writeType="0" dbDriver="native"> 11 <heartbeat>select user()</heartbeat> 12 <writeHost host="hostM1" url="172.17.0.2:3306" user="root" password="123456"></writeHost> 13 </dataHost> 14 <dataHost name="host2" dbType="mysql" maxCon="10" minCon="3" balance="0" writeType="0" dbDriver="native"> 15 <heartbeat>select user()</heartbeat> 16 <writeHost host="hostM2" url="172.17.0.3:3306" user="root" password="123456"></writeHost> 17 </dataHost> 18 <dataHost name="host3" dbType="mysql" maxCon="10" minCon="3" balance="0" writeType="0" dbDriver="native"> 19 <heartbeat>select user()</heartbeat> 20 <writeHost host="hostM3" url="172.17.0.4:3306" user="root" password="123456"></writeHost> 21 </dataHost> 22 </mycat:schema>
In lines 4 to 6, the student table in the redisDemo database is defined, which will be distributed to the three database nodes dn1, dn2, and dn3 according to the mod-long rule. Subsequently, in the code from lines 7 to 9, the definitions of the three nodes dn1, dn2, and dn3 are given, which point to the redisDemo database of host1, host2 and host3 respectively.
In the code from line 10 to line 21, the definitions for host1 to host3 are given. Their configuration is very similar. Here, the host1 configuration from line 10 to line 13 is used to illustrate.
In the tenth, firstly, through the dbType parameter, it is defined that host1 is a mysql type database, and then the maximum and minimum number of connections to the host database are specified through the maxCon and minCon parameters, and the read and write to host1 are specified through the balance and writeType parameters. The request is actually sent to the definition on line 12. The url is the mysql database of 172.17.0.2:3306. At the same time, in line 12, the user name and password of the mysql database connected to 172.17.0.2:3306 are also specified. The heartbeat parameter defined in line 11 defines the MyCAT component to use the select user() sql statement to determine whether the host1 database is in the "connected" state. In other words, the dn1 node defined in line 5 ultimately points to the stduent table of the MySQL database where 172.17.0.2:3306 is located.
Similarly, in the definitions for host2 and host3 in lines 14 to 21, the specific URL addresses of these two databases are also defined respectively. In other words, the student table defined in the redisDemo database on line 4 will eventually be scattered to the three redisDemo databases of 172.17.0.2:3306, 172.17.0.3:3306 and 172.17.0.4:3306 according to the definition of dataNode stduent table inside.
Through the figure below, you can see the sub-database relationship defined by the relevant parameters in the configuration file more clearly.
In this example, Docker containers are used to create three MySQL instances on the same host, so 172.17.0.2:3306, 172.17.0.3:3306, and 172.17.0.4:3306 are the addresses of the three Docker containers on this machine. If in the project, the MySQL server is deployed on multiple hosts, then the corresponding addresses should be modified to the IP addresses of these hosts.
Third, the code of the rule.xml configuration file is shown below.
1 <?xml version="1.0" encoding="UTF-8"?> 2 <!DOCTYPE mycat:rule SYSTEM "rule.dtd"> 3 <mycat:rule xmlns:mycat="http://io.mycat/"> 4 <tableRule name="mod-long"> 5 <rule> 6 <columns>id</columns> 7 <algorithm>mod-long</algorithm> 8 </rule> 9 </tableRule> 10 11 <function name="mod-long" class="io.mycat.route.function.PartitionByMod"> 12 <property name="count">3</property> 13 </function> 14 </mycat:rule>
The mod-long rule is defined in line 4, which is used in line 5 of schema.xml, and combined with the code from lines 11 to 13, you can see that the rule is used to sub-database the student table At the time, the id will be processed modulo 3 first, and then based on the modulo result, it will be processed in the student library of the data table where host1 to host3 are located. The modulus value 3 here needs to be the same as the number of MySQL hosts.
The above three configuration files are combined to give the following definitions for the actions related to the sub-database and sub-table.
1. If the application wants to use MyCAT, it needs to connect to the MyCAT component with the root user name and the 123456 password.
2. For example, if you want to insert stduent data with id 1, according to the definition in schema.xml, the id will be processed mod-long according to the mod-long rule. The result is 1, so it will be inserted into the 172.17 defined by host2. In the student table of 0.3:3306 database, if you want to perform read, delete and update operations, the id will be modulo 3, and then the request will be sent to the corresponding database.
Here only one of the more commonly used rules of MyCAT sub-library (that is, modulo) is given, and the stduent table is scattered into three physical data tables. In fact, by writing the configuration, other algorithms can be used to let the MyCAT component The data table is scattered into more sub-tables.
3 Integration example of Java, MySQL and MyCAT
Here will take the requirement of "a MyCAT component to connect to three MySQL databases and sub-databases for student tables" as an example, combined with the three MyCAT configuration files given above, give the details of setting MyCAT sub-databases and sub-tables based on the Docker container Steps, and on this basis, give a code example of Java application connecting MyCAT to realize sub-library and sub-table.
Step 1: Prepare 3 Docker containers containing MySQL through the following 3 Docker commands.
1 docker run -itd -p 3306:3306 --name mysqlHost1 -e MYSQL_ROOT_PASSWORD=123456 mysql:latest 2 docker run -itd -p 3316:3306 --name mysqlHost2 -e MYSQL_ROOT_PASSWORD=123456 mysql:latest 3 docker run -itd -p 3326:3306 --name mysqlHost3 -e MYSQL_ROOT_PASSWORD=123456 mysql:latest
The three MySQL docker containers created here are called mysqlHost1, mysqlHost2, and mysqlHost3. They all work on port 3306 in the container, but they are mapped to ports 3306, 3316, and 3326 of the host, respectively. And, through the -e parameter, the passwords of the root user names of the three databases are specified as 123456.
After the creation is complete, use the following commands to observe the ip address of the docker container where they are located.
1 docker inspect mysqlHost1 2 docker inspect mysqlHost2 3 docker inspect mysqlHost3
The observed IP addresses are shown in the following table. When you operate on your computer, if you see other IP addresses, you need to change the relevant configuration items in the following steps.
Step two, enter the mysqlHost1 container through the docker exec -it mysqlHost1 /bin/bash command, and then use the mysql -u root -p command to enter the mysql database, the password you need to enter when entering is 123456, and then run the following command to create redisDemo Database and student table.
1 create database redisDemo; 2 use redisDemo; 3 create table student( id int not null primary key,name char(20),age int,score float);
Among them, the first statement is used to build the database, the second statement is to enter the redisDemo library, and the third statement is used to build the table.
After completion, enter the other two MySQL containers through the docker exec -it mysqlHost2 /bin/bash and docker exec -it mysqlHost3 /bin/bash commands, and enter the database through the mysql command, and then proceed through the above statement The action of creating a database and a data table. So far, the creation of the three MySQL databases has been completed.
Step 3: Download the image of the mycat component through the docker pull command. If you cannot download it, you can use the docker search mycat command to find the available image and download it.
Step 4: Create a new C:\work\mycat\conf directory, and put in it the three configuration files server.xml, rule.xml and schema.xml for MyCAT components given in section 10.2.2.
Among them, in the schema.xml, the definition of the database url is shown in the following lines 3, 7 and 11. Please note that they point to the MySQL IP address in the specific Docker container, and their values need to be consistent with the values given in Table 10.3. If you use the docker inspect command to observe that the addresses of the three Dockers have changed, you need to modify the url value in the first schema.xml.
1 <dataHost name="host1" dbType="mysql" maxCon="10" minCon="3" balance="0" writeType="0" dbDriver="native"> 2 <heartbeat>select user()</heartbeat> 3 <writeHost host="hostM1" url="172.17.0.2:3306" user="root" password="123456"></writeHost> 4 </dataHost> 5 <dataHost name="host2" dbType="mysql" maxCon="10" minCon="3" balance="0" writeType="0" dbDriver="native"> 6 <heartbeat>select user()</heartbeat> 7 <writeHost host="hostM2" url="172.17.0.3:3306" user="root" password="123456"></writeHost> 8 </dataHost> 9 <dataHost name="host3" dbType="mysql" maxCon="10" minCon="3" balance="0" writeType="0" dbDriver="native"> 10 <heartbeat>select user()</heartbeat> 11 <writeHost host="hostM3" url="172.17.0.4:3306" user="root" password="123456"></writeHost> 12 </dataHost>
Step 5, after ensuring that the My SQL contained in the above three Dockers are all available, start the docker container corresponding to MyCAT through the following Docker command.
1 docker run --name mycat -p 8066:8066 -p 9066:9066 -v C:\work\mycat\conf\server.xml:/opt/mycat/conf/server.xml:ro -v C:\work\mycat\conf\schema.xml:/opt/mycat/conf/schema.xml:ro -v C:\work\mycat\conf\rule.xml:/opt/mycat/conf/rule.xml:ro -d mycat:latest
Please pay attention to the following points of the docker command.
1 Use the -p parameter to map the working port 8066 and management port 9066 of the MyCAT component to the port with the same name in the host.
2 With three -v parameters, map the three MyCAT configuration files in the C:\work\mycat\conf\ directory outside the container to the /opt/mycat/conf/ directory inside the container, so that they can be read when starting To these three configuration files. The premise of this is that it has been confirmed in advance that the three configuration files such as server.xml in the container exist in the /opt/mycat/conf/ directory. If the three configuration files in some mycat images do not exist in this directory, You can use the docker exec -it mycat /bin/bash command to enter the mycat container first, find the corresponding locations of the three configuration files, and then rewrite the docker run command to start the mycat container.
3 Use the mycat:latest parameter to specify that the container is generated based on the mycat:latest image.
After running the above docker run command, you can use the docker logs mycat command to observe the startup log of the MyCAT component contained in the container. If it starts successfully, you can see a message indicating success in the log as shown in Figure 10.12 below. If there is an error, then either check the connection status of the three MySQL databases, or troubleshoot the problem according to the error message given in the log.
So far, the configuration of the MyCAT component and the three MySQL databases is completed. In the following MyCATSimpleDemo example, the method of inserting data into the MySQL database by the Java program through the MyCAT component will be given, and everyone can feel the effect of sub-database and sub-table.
1 import java.sql.*;
2 public class MyCATSimpleDemo {
3 public static void main(String[] args){
4 //Define the connection object and PreparedStatement object
5 Connection myCATConn = null;
6 PreparedStatement ps = null;
7 //Define Connection information
8 String mySQLDriver = "com.mysql.jdbc.Driver";
9 String myCATUrl = "jdbc:mysql://localhost:8066/redisDemo";
10 String user = "root";
11 String pwd = "123456";
12 try{
13 Class.forName(mySQLDriver);
14 myCATConn = DriverManager.getConnection(myCATUrl, user, pwd);
15 ps = myCATConn.prepareStatement("insert into student (id,name,age,score) values (?,'test',18,100)");
16 ps.setString(1,"11");
17 ps.addBatch();
18 ps.setString(1,"12");
19 ps.addBatch();
20 ps.setString(1,"13");
21 ps.addBatch();
22 ps.executeBatch();
23 } catch (SQLException se) {
24 se.printStackTrace();
25 } catch (Exception e) {
26 e.printStackTrace();
27 }
28 finally{
29 //如果有必要，释放资源
30 if(ps != null){
31 try {
32 ps.close();
33 } catch (SQLException e) {
34 e.printStackTrace();
35 }
36 }
37 if(myCATConn != null){
38 try {
39 myCATConn.close();
40 } catch (SQLException e) {
41 e.printStackTrace();
42 }
43 }
44 }
45 }
46 }
In the 14th line of this example, the connection object myCATConn pointing to the MyCAT component is created. Please note that it is pointing to port 8066 of localhost. Use root and 123456 to connect to the redisDemo database, which is consistent with the configuration in server.xml. In the following line 15, the PreparedStatement type ps object is created with myCATConn, and in the code from lines 16 to 21, three insert statements are assembled in batches through the addBatch method. Please note that their ids are 11, 12 and 13, finally in the 22nd line of the code, these three insert statements are executed through the executeBatch statement.
As you can see, the way to execute SQL statements through the MyCAT connection object is basically the same as the way to directly use the MySQL connection object, and when obtaining the MyCAT connection object, you only need to change the connection url accordingly. In other words, the MyCAT component is transparent to the application when it implements the sub-database and sub-table. It completely separates the "business action of data operation" and "the underlying realization of data operation", so if you want to in a system The introduction of MyCAT sub-database and sub-table components has very limited modification points and has little impact on the original business.
Let's look at the effect of sub-database and table again. Enter the mysqlHost1 container through the docker exec -it mysqlHost1 /bin/bash command, and then use the mysql -u root -p command to enter the mysql database, and use the redisDemo; command to enter the redisDemo database After executing the select * from student; command, only one piece of data can be seen, as shown in the figure below.
Similarly, in the database where mysqlHost2 and mysqlHost3 are located, you can only see one piece of data. The student data stored in these three databases is shown in the following table.
As you can see, according to the value of id modulo 3, the MyCAT component separates them into 3 databases. Since the focus of this book is on Redis, we will no longer give relevant examples of deleting, updating and querying operations with MyCAT components. However, if you use the myCATConn connection object in the above example and the ps object generated with it to implement related operations The effect is not difficult.
The data in the student table here is very small, in fact, it cannot reflect the advantages of sub-database and sub-table, but if the scale of this table is large, such as reaching a million level or even higher, then the effect of sub-database and sub-table is introduced through the MyCat component After that, it is equivalent to spreading the pressure on this large table evenly on several sub-tables, which can better cope with high-concurrency scenarios.