I. Introduction
As the most popular distributed database middleware, Mycat has been implemented in many production projects, and Mycat will be used in more production projects over time.
This article mainly introduces the main configuration files of MyCat, and the author's understanding of these configurations.
2. Preface
This article mainly analyzes the three most commonly used files, server.xml, schema.xml, and rule.xml.
3. Server.xml
Server.xml saves all the system configuration information that mycat needs, and the code is mapped to the SystemConfig class.
There are four main tags:
system,user,firewarll,cluster.
Next, the four labels are explained
1. user tag
Content: <user name="test"> Explain that the user name is test.
Subtags :
property, privileges.
1) Contents of property tag
:
| content | illustrate |
|---|---|
<property name="password">test</property> |
User password is test |
<property name="schemas">db1, db2</property> |
The accessible schemas are db1, db2 |
<property name="readOnly">true</property> |
read-only |
<property name="benchmark">11111</property> |
Connection upper limit, downgrade weight. |
<property name="usingDecrypt">1</property> |
Whether to enable encryption. |
2) The privileges tag
performs refined DML permission control on the user's schema and table.
<privileges check="false">- 1
check indicates whether to enable DML permission checking. Default is off.
The server.dtd file <!ELEMENT privileges (schema)*> states that there can be multiple schema configurations.
dml sequence description:
insert,update,select,delete
<schema name="db1" dml="0110" > <table name="tb01" dml="0000"></table> <table name="tb02" dml="1111"></table> </schema>- 1
- 2
- 3
- 4
The authority of db1 is update, select.
The authority of tb01 is that nothing can be done.
The permissions of tb02 are insert, update, select, delete.
Other tables default to udpate, select.
2. system tag
All property tags nested within this tag are related to system configuration.
<property name="charset">utf8</property> - 1
character set
<property name="processors">1</property> - 1
The number of processing threads, the default is the number of CPUs.
<property name="processorBufferChunk">4096</property> - 1
The number of each read, the default is 4096.
<property name="processorBufferPool">409600</property>- 1
The total space required to create a shared buffer. processorBufferChunk*processors*100.
<property name="processorBufferPoolType">0</property>- 1
The default is 0. 0 means DirectByteBufferPool, 1 means ByteBufferArena.
<property name="processorBufferLocalPercent">100</property>- 1
The secondary shared buffer is the percentage of processorBufferPool, which is set here.
<property name="sequnceHandlerType">100</property>- 1
Global ID generation method. (0: for local file mode, 1: for database mode; 2: for timestamp sequence mode; 3: for ZK generation ID; 4: for ZK incremental ID generation.
<property name="useCompression">1</property>- 1
Whether to enable the mysql compression protocol. 1 is on, 0 is off, the default is off.
<property name="packetHeaderSize">4</property> - 1
Specifies the header length in the Mysql protocol. The default is 4.
<property name="maxPacketSize">16M</property>- 1
Specifies the maximum length of data that the Mysql protocol can carry. The default is 16M.
<property name="idleTimeout">1800000</property>- 1
Specifies the idle timeout for the connection. When a connection initiates an idle check and finds that the idle time has passed since the last use, the connection will be recycled, that is, it will be closed directly. The default is 30 minutes, in milliseconds.
<property name="txIsolation">3</property>- 1
The initialization transaction isolation level of the front-end connection is only used during initialization, and the back-end database connection will be synchronized according to the attributes passed by the client. The default is REPEATED_READ, and the setting value is the number default 3.
READ_UNCOMMITTED = 1;
READ_COMMITTED = 2;
REPEATED_READ = 3;
SERIALIZABLE = 4;
<property name="sqlExecuteTimeout">300</property>- 1
The time when the SQL execution times out, Mycat will check the time of the last execution of SQL on the connection, and if it exceeds this time, the connection will be closed directly. The default time is 300 seconds, in seconds.
<property name="processorCheckPeriod">1000</property>- 1
Clean up the interval between front-end and back-end idle, timeout and close connections on NIOProcessor. The default is 1 second,
in milliseconds.
<property name="dataNodeIdleCheckPeriod">300000</property> - 1
The interval between idle and timeout checks for backend connections, the default is 300 seconds, in milliseconds.
<property name="dataNodeHeartbeatPeriod">10000</property>- 1
The interval between heartbeats for all read and write libraries in the backend, the default is 10 seconds, in milliseconds.
<property name="bindIp">0.0.0.0</property>- 1
The IP address the mycat service listens on, the default value is 0.0.0.0.
<property name="serverPort">8066</property>- 1
Define the port used by mycat, the default value is 8066.
<property name="managerPort">9066</property>- 1
Defines the management port of mycat, the default value is 9066.
<property name="fakeMySQLVersion">5.6</property>- 1
The version number of mysql simulated by mycat, the default value is version 5.6. If it is not specially required, do not modify this value. Currently, versions 5.5, 5.6, and 5.7 are supported. Other versions may have problems.
<property name="useSqlStat">0</property> - 1
Whether to enable real-time statistics. 1 is on; 0 is off.
<property name="useGlobleTableCheck">0</property> - 1
Whether to enable global table consistency check. 1 is on; 0 is off.
<property name="handleDistributedTransactions">0</property>- 1
Distributed transaction switch. 0 means do not filter distributed transactions; 1 means filter distributed transactions; 2 means do not filter distributed transactions, but record distributed transaction logs.
<property name="maxStringLiteralLength">65535</property>- 1
The default is 65535. 64K is used for the maximum text length in sql parsing.
The properties in the above example are only a part, and there are many variables that can be configured. For details, you can view the properties of the SystemConfig class.
The properties under the System tab are generally after going online and need to be modified according to the actual running situation and after analysis and tuning.
3. Firewall tab
As the name implies, this is about the firewall settings, that is, to limit the requested addresses at the network layer, mainly from a security perspective to ensure that Mycat is not accessed by anonymous IPs
<firewall>
<whitehost>
<host host="127.0.0.1" user="mycat"/> <host host="127.0.0.2" user="mycat"/> </whitehost> <blacklist check="false"> </blacklist> </firewall>- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
The settings are very simple and easy to understand. As long as the whitelist is set, it means that the firewall is turned on, and only the whitelisted connections can be connected.
4. Cluster label
To be perfected.
4. Schema.xml
This file is the most important configuration file of MyCat, responsible for managing libraries, tables, sharding rules, DataNode, DataSource. Therefore, this configuration needs to be very clear about the meaning of each part.
schema.dtd
<!ELEMENT mycat:schema (schema*,dataNode*,dataHost*)>- 1
You can see that the included tags are:
Schema 、 dataNode 、 dataHost。
Next, each label will be explained in detail one by one.
1. Schema tag
<schema name="db1" checkSQLschema="false" sqlMaxLimit="100"></schema>- 1
The schema tag is used to define the logic library in the MyCat instance. MyCat can have multiple logic libraries, and each logic library has its own related configuration. These different libraries can be divided using schema tags.
<schema name="TESTDB" checkSQLschema="false" sqlMaxLimit="100"></schema> <schema name="USERDB" checkSQLschema="false" sqlMaxLimit="100"></schema>- 1
- 2
The configuration shown above configures two different logical libraries. The concept of a logical library is the same as that of a Database in the MYSQL database. When we query the tables in these two different logical libraries, we need to switch to this logical library. The required table can be queried.
The attribute checkSQLschema is false by default. The official document means whether to remove the name of the database in front of the table, "select * from db1.testtable", if it is set to true, db1 will be removed. But if the name of db1 is not the name of the schema, it will not be removed, so the official recommendation is not to use this syntax. Also set to false by default.
sqlMaxLimit when the value is set to a certain value. For each executed SQL statement, if the limit statement is not added, MyCat will automatically add the corresponding value. For example, set the value to 100 and execute "select * from test_table", the effect will be "selelct * from test_table limit 100".
Note: This property will not take effect if the running schema is non-split library.
1) Table tag
<table name="travelrecord" dataNode="dn1,dn2,dn3" rule="auto-sharding-long" ></table>- 1
The Table tag defines the logical table in MyCat, and all tables that need to be split need to be defined in this tag.
Relevant attributes of the table tag:
| Attributes | illustrate |
|---|---|
| name | This attribute defines the table name of the logical table |
| dataNode | This attribute defines the dataNode to which this logical table belongs, and the value of this attribute needs to correspond to the value of the name attribute in the dataNode tag. |
| rule | This attribute is used to specify the rule name to be used in the logic table. The rule name is defined in rule.xml and must correspond to the value of the name attribute in the tableRule tag. |
| ruleRequired | This property is used to specify whether the table is bound to a sharding rule. If the configuration is true, but no specific rule is configured, the program will report an error. |
| primaryKey | The logical table corresponds to the primary key of the real table, |
| type | This attribute defines the type of logical table. Currently, there are only two types of logical table: "global table" and "normal table". The global table defines type=”global”, and what is not defined is the ordinary table. |
| autoIncrement | Whether the primary key is self-incrementing. |
| subTables | Sub-table, sub-table currently does not support Join. |
| needAddLimit | Whether to automatically add limit, the default is on. Close with caution. |
2) childTable tag
<table name="customer" primaryKey="ID" dataNode="dn1,dn2" rule="sharding-by-intfile"> <childTable name="c_a" primaryKey="ID" joinKey="customer_id" parentKey="id" /> </table>- 1
- 2
The childTable tag is used to define the child table of the ER shard. It is associated with the parent table through attributes on the label.
Property description:
| Attributes | illustrate |
|---|---|
| name | the name of the subtable |
| joinKey | the name of the field in the subtable |
| parentKey | Field name in parent table |
| primaryKey | Same as Table |
| needAddLimit | Same as Table |
2. dataNode label
The dataNode tag defines the data nodes in MyCat, which is what we usually call data shards. A dataNode label is an independent data shard.
<dataNode name="dNode1" dataHost="dHost128" database="db1" ></dataNode>- 1
The meaning expressed in the example is: use the db1 physical database on the database instance named dHost128, which constitutes a data shard, and finally, we use the name dNode1 to identify this shard.
Relevant attributes of the dataNode tag:
| Attributes | illustrate |
|---|---|
| Name | Define the name of the data node, this name needs to be unique |
| dataHost | This property is used to define which database instance the shard belongs to |
| Database | This property is used to define the specific library on which specific database instance of the shard property |
3. dataHost tag
This tag defines the specific database instance, read-write separation configuration, and heartbeat statement.
<dataHost name="localhost1" maxCon="1000" minCon="10" balance="0" writeType="0" dbType="mysql" dbDriver="native"> <heartbeat>select user()</heartbeat> <writeHost host="hostM1" url="localhost:3306" user="root" password="123456"> </writeHost> </dataHost>- 1
- 2
- 3
- 4
- 5
Relevant attributes of the dataHost tag:
| Label | illustrate |
|---|---|
| name | Uniquely identifies the dataHost tag for use by upper-level tags |
| maxCon | Specifies the maximum connections per read-write instance connection pool. |
| minCon | Specify the minimum connection of each read-write instance connection pool, and initialize the size of the connection pool. |
| balance | 负载均衡类型，目前的取值有4 种： “0”, 不开启读写分离机制，所有读操作都发送到当前可用的 writeHost 上。 “1”，全部的 readHost 与 stand by writeHost 参与 select 语句的负载均衡，简单的说，当双主双从模式(M1->S1，M2->S2，并且 M1 与 M2 互为主备)，正常情况下，M2,S1,S2 都参与 select 语句的负载均衡。”2”，所有读操作都随机的在 writeHost、readhost 上分发。”3”，所有读请求随机的分发到 wiriterHost 对应的 readhost 执行，writerHost 不负担读压writeType 1. writeType=”0”, 所有写操作发送到配置的第一个 writeHost，第一个挂了切到还生存的第二个writeHost，重新启动后已切换后的为准，切换记录在配置文件中:dnindex.properties .2. writeType=”1”，所有写操作都随机的发送到配置的 writeHost，1.5 以后废弃不推荐。默认0就好了！ |
| dbType | 指定后端连接的数据库类型，目前支持二进制的 mysql 协议，还有其他使用 JDBC 连接的数据库。例如：mongodb、oracle、spark 等. |
| dbDriver | 指定连接后端数据库使用的 Driver，目前可选的值有 native 和 JDBC。使用 native 的话，因为这个值执行的是二进制的 mysql 协议，所以可以使用 mysql 和 maridb。其他类型的数据库则需要使用 JDBC 驱动来支持。 |
| switchType | “-1” 表示不自动切换； “1” 默认值，自动切换； “2” 基于 MySQL 主从同步的状态决定是否切换心跳语句为 show slave status； “3” 基于 MySQL galary cluster 的切换机制（适合集群）（1.4.1）心跳语句为 show status like ‘wsrep%’. |
| tempReadHostAvailable | 如果配置了这个属性 writeHost 下面的 readHost 仍旧可用，默认 0 可配置（0、1）。 |
1) heartbeat 标签
这个标签内指明用于和后端数据库进行心跳检查的语句。
例如：MYSQL 可以使用 select user()，Oracle 可以使用 select 1 from dual 等。
2) writeHost /readHost 标签
这两个标签都指定后端数据库的相关配置，用于实例化后端连接池。唯一不同的是，writeHost 指定写实例、readHost 指定读实例。
在一个 dataHost 内可以定义多个 writeHost 和 readHost。但是，如果 writeHost 指定的后端数据库宕机，那么这个 writeHost 绑定的所有 readHost 都将不可用。另一方面，由于这个 writeHost 宕机，系统会自动的检测到，并切换到备用的 writeHost 上去。这两个标签的属性相同，这里就一起介绍。
| 属性 | 说明 |
|---|---|
| host | 用于标识不同实例，一般 writeHost 我们使用*M1，readHost 我们用*S1。 |
| url | 后端实例连接地址。Native：地址：端口 JDBC：jdbc的url |
| password | 后端存储实例需要的密码 |
| user | 后端存储实例需要的用户名字 |
| weight | 权重 配置在 readhost 中作为读节点的权重 |
| usingDecrypt | 是否对密码加密，默认0。具体加密方法看官方文档。 |
五、 Rule.xml
rule.xml 里面就定义了我们对表进行拆分所涉及到的规则定义。我们可以灵活的对表使用不同的分片算法，或者对表使用相同的算法但具体的参数不同。
包含的标签 tableRule 和 function。
1. tableRule 标签
这个标签定义表规则。
定义的表规则，在 schema.xml：
<tableRule name="rule1">
<rule> <columns>id</columns> <algorithm>func1</algorithm> </rule> </tableRule>- 1
- 2
- 3
- 4
- 5
- 6
name 属性指定唯一的名字，用于标识不同的表规则。
内嵌的 rule 标签则指定对物理表中的哪一列进行拆分和使用什么路由算法。
columns 内指定要拆分的列名字。
algorithm 使用 function 标签中的 name 属性。连接表规则和具体路由算法。当然，多个表规则可以连接到
同一个路由算法上。table 标签内使用。让逻辑表使用这个规则进行分片。
2. function 标签
<function name="hash-int" class="org.opencloudb.route.function.PartitionByFileMap">
<property name="mapFile">partition-hash-int.txt</property> </function>- 1
- 2
- 3
name 指定算法的名字。
class 制定路由算法具体的类名字。
property 为具体算法需要用到的一些属性。
六、 后记
至此，MyCat三个主要的配置文件的说明已经完成，过程中参考了一些文章，在此不进行一一列举，同时参考了MyCat的源代码，以及官方文档。