MyCat, Cobar, distributed database fragmentation (MySQL availability middleware)

Outline 

The first MYCAT background

The second development MYCAT

About a third MYCAT

The fourth characteristic MYCAT

The fifth MYCAT installation

Sixth MYCAT and compare COBAR

Seventh reference

 

I. Background

With traditional database technology matures, the rapid development and expansion of the scope of application of computer network technology, database applications have generally built on a computer network. Then a centralized database system demonstrated its shortcomings:

(1) centralized processing will inevitably result in a performance bottleneck;

(2) in the application set to run on a computer, once the computer fails, the whole system is affected, the reliability is not high;

(3) centralized processing system caused by the size and configuration are not flexible enough, poor system scalability.

In this situation, the centralized database will be distributed database development.

 

Second, the development

MyCAT of birth, and from its predecessor Amoeba Cobar start.

Amoeba (Amoeba) project, the open source framework in 2008 began publishing a Amoeba for Mysql software. This software is committed to MySQL distributed database front-end proxy layer, it is mainly in the application layer access MySQL SQL act as routing capabilities, focused on distributed database proxy layer (Database Proxy) development. Located and Client, between DB Server (s), transparent to the client. Having load balancing, high availability, the SQL filtration, separate read and write, may be routed to the relevant target database, the database may be multiple concurrent requests merged result. By Amoeba You can complete availability of multiple data sources, load balancing, data slice functions, currently Amoeba has been used in many business lines above.

Alibaba in 2012 Nian 6 Yue 19 Ri , outside the official database middleware open source Cobar, formerly known as Amoeba is already open source, but after its author Chen Siru went off grand, taking into account the internal Alibaba Amoeba stability, performance, and feature support as well as other factors, re-established a project group and changes its name to Cobar. Cobar is that it looks like a traditional database services, like providing massive amounts of data in a distributed environment by the MySQL open source distributed processing middleware Alibaba.

Cobar from the date of birth, sought after by the majority of programmers, but since 2013, almost no subsequent updates. In this case, MyCAT came into being, which is based on open source Ali Cobar product research and development, Cobar stability, reliability, excellent performance and architecture, as well as numerous sophisticated use cases makes MyCAT the outset have a good starting point, standing on the shoulders of giants, myCAT can see farther. The latest released version is 1.3.0.2 version MyCAT.

Three, MYCAT Profile

MyCAT is mysql middleware, formerly known as Ali famous Cobar, Cobar open after a period of time, nothing. So MyCAT taken up this banner, in the era of big data, its importance is increasingly apparent.

Mycat be described and defined as follows:

1 a completely open source database cluster for large enterprise application development

2 supports transactions, ACID, can replace the enhanced version of MySQL database

3 can be regarded as a cluster of MySQL enterprise database, Oracle cluster used to replace expensive

4 a fusion memory cache, NoSQL technology, HDFS large new SQL Server data

5 combination of traditional database and data warehouse generation of new distributed enterprise-level database products

6 a new database middleware products

 Four, Mycat Key Features 

Support SQL92 standard

1 Mysql native protocol compliance, cross-language, cross-platform, cross-database middleware general agent.

2 based heartbeat automatic failover support separate read and write, from the main support MySQL, galera cluster and the cluster.

3 supports Galera for MySQL Cluster, Percona Cluster or MariaDB cluster

4 based on Nio achieve effective management of threads, high concurrency issues.

5 a plurality of pieces of data to support automatic routing and aggregation functions to support conventional polymerization sum, count, max, etc., cross-database tab.

6 single library supports any internal join, cross-database 2 table join, even based on the multi-table join caltlet.

7 support through a global table, fragmentation strategy ER relations, enables efficient multi-table join queries.

8 supports multi-tenant solution.

9 support distributed transactions (weak xa).

10 support global serial number, primary key to solve the problem of distributed generation.

11 fragmentation rules rich, plug-in technology development, easy to expand.

12 Powerful web, command line monitoring.

13 support the front end as mysq common agent, the back-end JDBC way to support Oracle, DB2, SQL Server, mongodb, giant sequoias.

14 support password encryption

15 Support Services Downgrade

16 Support IP whitelist

17 blacklist support SQL, sql injection attack blocking

18 support points table (1.6)

19 ZooKeeper cluster-based management, online upgrade, expansion, intelligent optimization, large data processing (2.0 Developer Edition).

 

Fifth, install and use

 

A test environment:

There are three tables users, item, item_detail (item sub-table), and three databases db01, db02, db03

In the data tables in database db01, and item tables item_detail db01 and db02 users distributed memory storage 

Now in three databases are created the same table

sql

create database db01;  

create database db02;  

create database db03;  

  

CREATE TABLE users (  

   id INT NOT NULL AUTO_INCREMENT,  

   name varchar(50) NOT NULL default '',  

    indate DATETIME NOT NULL default '0000-00-00 00:00:00',  

    PRIMARY KEY (id)  

)AUTO_INCREMENT= 1 ENGINE=InnoDB DEFAULT CHARSET=utf8;  

 

CREATE TABLE item (  

   id INT NOT NULL AUTO_INCREMENT,  

    value INT NOT NULL default 0,  

    indate DATETIME NOT NULL default '0000-00-00 00:00:00',  

    PRIMARY KEY (id)  

)AUTO_INCREMENT= 1 ENGINE=InnoDB DEFAULT CHARSET=utf8;  

  

CREATE TABLE item_detail (  

    id INT NOT NULL AUTO_INCREMENT,  

    value INT NOT NULL default 0,  

    name varchar(50) NOT NULL default '',  

    item_id INT NOT NULL,  

   PRIMARY KEY (id),  

    key (item_id)  

)AUTO_INCREMENT= 1 ENGINE=InnoDB DEFAULT CHARSET=utf8;  

 

 Installation and Configuration B JDK

This does not install the demo

 

C Mycat installation configuration

1. cd /usr/local/src  
2. tar xvf Mycat-server-1.5.1-RELEASE-20160405120037-linux.tar.gz   
3. mv /usr/local/src/mycat /usr/local/  
4.   
5. groupadd mycat  
6. useradd -g mycat mycat  
7. passwd mycat  
8. chown -R mycat.mycat /usr/local/mycat  

 

Adding Environment Variables

1. vi /etc/profile  
2. vi ~ / .bash_profile  
3.   
4. export MYCAT_HOME=/usr/local/mycat  

Set to take effect (best to reboot the system)

 

1. source /etc/profile  
2. source ~/.bash_profile  

D server name and IP binding

 

1. vi /etc/hosts  
2. 192.168.100.50  server1  
3. 127.0.0.1       server1  

Java path is provided wrapper.java.command

 (And can be set from -Xmx -Xms, reference: myCat failed to start Exceeds The maximum size specified of The representable size )

 

1. we /usr/local/mycat/conf/wrapper.conf  
2.   
3. wrapper.java.command=%JAVA_HOME%/bin/java  

Configuring server.xml and schema.xml (in this case the bottom of the configuration information in the article)

 

1. we /usr/local/mycat/conf/server.xml  
2. we /usr/local/mycat/conf/schema.xml  

Start mycat service ./mycat {console | start | stop | restart | status | dump}

 

1. /usr/local/mycat/bin/mycat console  
2. #mycat process  
3. ps -ef | grep mycat  
4.   
5. # Day information  
6. tail -20 /usr/local/mycat/logs/wrapper.log  

 

E test

Access mycat (credentials reference /usr/local/mycat/conf/server.xml)

1. mysql -utest -ptest -h127.0.0.1 -P8066 -DTESTDB  
2. mysql> show databases;  
3. +----------+  
4. | DATABASE |  
5. +----------+  
6. | TESTDB   |   
7. +----------+  
8.   
9. mysql> show tables;  
10. +------------------+  
11. | Tables in TESTDB |  
12. +------------------+  
13. | item             |   
14. | item_detail      |   
15. | users            |   
16. +------------------+  

Insert and query data in mycat:

 

1. insert into users(name,indate) values('kk',now());  
2. insert into item(id,value,indate) values(1,100,now());  
3. insert into item_detail(value,name,item_id) values('pad',40,1);  
4. insert into item_detail(value,name,item_id) values('phone',50,1);  
5.   
6. insert into item(id,value,indate) values(999,100,now());  
7. insert into item_detail(value,name,item_id) values('pad',40,999);  
8. insert into item_detail(value,name,item_id) values('phone',50,999);  

 

1. mysql> select * from users;  
2. +----+------+---------------------+  
3. | id | name | indate              |  
4. +----+------+---------------------+  
5. | 1 | kk | 4/13/2016 6:28:00 |   
6. +----+------+---------------------+  
7.   
8. mysql> select * from item;  
9. +-----+-------+---------------------+  
10. | id  | value | indate              |  
11. +-----+-------+---------------------+  
12. | 999 |   100 | 2016-04-13 06:34:12 |   
13. |   1 |   100 | 2016-04-13 06:32:58 |   
14. +-----+-------+---------------------+  
15.   
16. mysql> select * from item_detail;  
17. +----+-------+------+---------+  
18. | id | value | name | item_id |  
19. +----+-------+------+---------+  
20. |  1 |     0 | 40   |     999 |   
21. |  2 |     0 | 50   |     999 |   
22. |  1 |     0 | 40   |       1 |   
23. |  2 |     0 | 50   |       1 |   
24. +----+-------+------+---------+  

Open another terminal, using the mysql account login access to view the table situation

1. mysql -uroot -ptest -p  

Specific situation data storage table:

 

1. -----------------------------------------  
2. mysql> select * from db01.users;  
3. +----+------+---------------------+  
4. | id | name | indate              |  
5. +----+------+---------------------+  
6. | 1 | kk | 4/13/2016 6:28:00 |   
7. +----+------+---------------------+  
8.   
9. mysql> select * from db02.users;  
10.   
11. mysql> select * from db03.users;  
12.   
13. -----------------------------------------  
14.   
15. mysql> select * from db01.item;  
16.   
17. mysql> select * from db02.item;  
18. +-----+-------+---------------------+  
19. | id  | value | indate              |  
20. +-----+-------+---------------------+  
21. | 999 |   100 | 2016-04-13 06:34:12 |   
22. +-----+-------+---------------------+  
23.   
24. mysql> select * from db03.item;  
25. +----+-------+---------------------+  
26. | id | value | indate              |  
27. +----+-------+---------------------+  
28. |  1 |   100 | 2016-04-13 06:32:58 |   
29. +----+-------+---------------------+  
30.   
31. -----------------------------------------  
32.   
33. mysql> select * from db01.item_detail;  
34.   
35. mysql> select * from db02.item_detail;  
36. +----+-------+------+---------+  
37. | id | value | name | item_id |  
38. +----+-------+------+---------+  
39. |  1 |     0 | 40   |     999 |   
40. |  2 |     0 | 50   |     999 |   
41. +----+-------+------+---------+  
42.   
43. mysql> select * from db03.item_detail;  
44. +----+-------+------+---------+  
45. | id | value | name | item_id |  
46. +----+-------+------+---------+  
47. |  1 |     0 | 40   |       1 |   
48. |  2 |     0 | 50   |       1 |   
49. +----+-------+------+---------+  

 

The test is completed, the data is a sub-library sub-table.

 server.xml configuration and schema.xml

1. # We /usr/local/mycat/conf/server.xml  
2.   
3. <?xml version="1.0" encoding="UTF-8"?>  
4. <!DOCTYPE mycat:server SYSTEM "server.dtd">  
5. <mycat:server xmlns:mycat="http://org.opencloudb/">  
6.         <system>  
7.             <!--   
8.                 <property name="processors">32</property>  
9.                 <property name="processorExecutor">32</property>   
10.                 <property name="bindIp">0.0.0.0</property>   
11.                 <property name="frontWriteQueueSize">4096</property>  
12.                 <property name="idleTimeout">300000</property>  
13.                 <property name="mutiNodePatchSize">100</property>  
14.             -->  
15.                 <property name="defaultSqlParser">druidparser</property>  
16.                 <property name="mutiNodeLimitType">1</property>  
17.                 <property name="serverPort">8066</property>  
18.                 <property name="managerPort">9066</property>   
19.         </system>  
20.         <-! Arbitrarily set landing mycat user name, password, database ->  
21.         <user name="test">  
22.                 <property name="password">test</property>  
23.                 <property name="schemas">TESTDB</property>  
24.         </user>  
25.   
26.         <user name="user">  
27.                 <property name="password">user</property>  
28.                 <property name="schemas">TESTDB</property>  
29.                 <property name="readOnly">true</property>  
30.         </user>  
31.         <!--   
32.         <quarantine>   
33.            <whitehost>  
34.               <host host="127.0.0.1" user="mycat"/>  
35.               <host host="127.0.0.2" user="mycat"/>  
36.            </whitehost>  
37.        <blacklist check="false"></blacklist>  
38.         </quarantine>  
39.         -->  
40. </mycat:server>  

 

1. # We /usr/local/mycat/conf/schema.xml  
2.   
3. <?xml version="1.0"?>  
4. <!DOCTYPE mycat:schema SYSTEM "schema.dtd">  
5. <mycat:schema xmlns:mycat="http://org.opencloudb/">  
6.   
7.     <! - storage of the setting table .schema name = "TESTDB" and set the same in server.xml TESTDB ->  
8.     <schema name="TESTDB" checkSQLschema="false" sqlMaxLimit="100">  
9.         <table name="users" primaryKey="id" type="global" dataNode="node_db01" />  
10.   
11.         <table name="item" primaryKey="id" dataNode="node_db02,node_db03" rule="mod-long">  
12.                 <childTable name="item_detail" primaryKey="id" joinKey="item_id" parentKey="id" />  
13.         </table>  
14.     </schema>  
15.   
16.     <! - dataNode provided corresponding database, and connection address dataHost mycat ->  
17.     <dataNode name="node_db01" dataHost="dataHost01" database="db01" />  
18.     <dataNode name="node_db02" dataHost="dataHost01" database="db02" />  
19.     <dataNode name="node_db03" dataHost="dataHost01" database="db03" />  
20.   
21.     <! -. DataHost physical host mycat corresponding logical host is also provided wherein a corresponding login information mysql ->  
22.     <dataHost name="dataHost01" maxCon="1000" minCon="10" balance="0" writeType="0" dbType="mysql" dbDriver="native">  
23.             <heartbeat>select user()</heartbeat>  
24.             <writeHost host="server1" url="127.0.0.1:3306" user="root" password="mysql"/>  
25.     </dataHost>  
26. </mycat:schema>  

Six, MYCAT and compare COBAR

1 mycat late than cobar launch, mycat the latter does not have a lot of functions

2 mycat to do cobar based improvements and optimization, system architecture are basically the same, are more mature.

3 in the main flow class name, an interface name, consistent relationship between the call and the function name.

4 for Cobar distal BIO implementations, a concurrent connection must block a thread, and the number of threads JVM is finite, slow execution and the DBServer Cobar, Cobar cause no response, i.e. suspended animation. To this end Mycat version made a big improvement, improved to AIO, over a million lines on the number of concurrent connections. In addition, NIO changed AIO, for the kernel slelect changed epoll: First, to further improve the efficiency of the IO: Second simplifies API calls, the application does not require at least one additional thread processes read cycle and so on.
5 Cobar just a simple set of results, and no sorting, merging functions and other functions, such as: max, count functions are not supported, Mycat for this purpose to increase DataMergeService

6 Cobar at a standstill, MyCAT community is very active

 

 

reference:

https://blog.csdn.net/pingdouble/article/details/79731677 

http://www.csdn.net/article/2015-07-16/2825228  (Principles and Applications)

http://www.songwie.com/articlelist/50 (theory and practice of writing good)

http://ly.blog.itpub.net/29733787/viewspace-2049139/ (example) MyCAT installation and simple test 

http://www.cnblogs.com/ivictor/p/5111495.html (example) MyCAT easy entry

http://www.2cto.com/database/201503/385641.html (Example) myCat level sub-table, the vertical sub-table Practice (2)

http://www.51testing.com/html/34/369434-3686088.html (example) using the separate read and write do simple Mycat (a)

"MyCat inAction Chinese version of" Master:

Mycat-server@github          :   https://github.com/MyCATApache/Mycat-Server.git
Mycat-web@github              :  https://github.com/MyCATApache/Mycat-Web.git
Mycat-doc@github               :  https://github.com/MyCATApache/Mycat-doc.git
Mycat-download@github     :  https://github.com/MyCATApache/

MyCAT Related:

Mycat Donation Address: http://www.mycat.io/donate.html

Mycat Zhuo official website: http://www.mycat.io/

Mycat Source: https://github.com/MyCATApache/Mycat-Server

Mycat Download: https://github.com/MyCATApache/Mycat-download

Mycat Definitive Guide Official Download: http://songwie.com/attached/file/mycat_1.5.2.pdf

Video Download: http://pan.baidu.com/s/1i4mgga5