On a simple and efficient implementation of mysql distributed and supported by id from sub-library sub-table method
Liwan Hong -2020-2-28
Database is the bottleneck where the IT system to improve database performance and storage capacity to implement distributed high-performance concurrent use mass data storage security is a very urgent and practical problems. Primary key Id is critical databases, primary keys using the number much faster than using a character and save storage space, Mysql auto-incrementing primary key is great, as long as the simple use of them, we can solve distributed id increment and support sub-library sub-table, cleverly use mysql own transaction management functions without the use of additional software or method is very complicated to solve, by contrast, such higher efficiency, lower costs, discussed in detail below.
One. Distributed correctly generated primary key
Distributed applications, there is inserted a plurality of data a user, to improve the speed, there is a plurality of users of the database data into one or more, i.e., a plurality of connection of the database while the database a plurality of insert If not treated, id incorrectly, which requires the use of a way to resolve.
Mysql principle of the self-energizing Id is inserted into the database, acquires the LAST_INSERT_ID (), and then generates id, id = LAST_INSERT_ID () + increment value.
First, when inserted into the database, you need to get LAST_INSERT_ID ()
And to ensure that the database id from the id must increase real-time updates to LAST_INSERT_ID ()
This is the only way to ensure a good increment id order, for which the following modifications inser sql statement:
<insert id="insertStudent" parameterType="com.czd.mybatis01.bean.Student">
INSERT stu(name)VALUES (#{name})
<selectKey keyProperty="id" order="AFTER" resultType="java.lang.Integer">
SELECT LAST_INSERT_ID()
</selectKey>
</insert>
This ensures that the latest id get carried insert, but id must be updated in real time to LAST_INSERT_ID (), will take to the correct id when so next time insertion. To do this you need to perform in the database:
Sequence ID the SET = the LAST_INSERT_ID the UPDATE (ID +. 1);
the UPDATE statement and initiator sequence counter increases to the LAST_INSERT_ID () of
A call for the return value after the upgrade. This SELECT statement by
mysql_insert_id () to retrieve the value, and then after the increment can get the correct id.
Calling the LAST_INSERT_ID () generated sequences, wherein the utility function ID value is stored in the server, as an automatically generated value. It is suitable for multiple users, because multiple users can use the UPDATE statement and a SELECT statement (or mysql_insert_id ()), get their own sequence value without affecting other produce their own sequence value of the client or other create their own sequence value of client impact. This would solve the high concurrent inserts distributed applications, correctly generate incremental id.
Note that, the mysql_insert_id () will only be upgraded later INSERT and UPDATE statements, so you can not execute the other corresponding value (expr) such as the SQL SELECT statement or SET used to retrieve the C API function LAST_INSERT_ID.
Mysql support of 4.2 billion data int id, sufficient to meet the business needs, the amount of data substantially lower than the 100 million necessary to take measures to ensure the performance of the database.
Second, the use of cluster mysql
Using the above method to generate incremental digital id, it can easily mysql cluster sub-library sub-table, as follows:
Embodied as:
We set the ID acquiring step is l, a total of n distribution tables id.
After initialization, id value of n in the table were 0, l, 2l, 3l ...
When the application ID acquired from any one table, this table of ID nl increase in the original ID values.
Illustrated, step 100, a total of four tables id
After initialization, the values of the four tables respectively, 1: 0, 2: 100, 3: 200, 4: 300
Obtained from Table 2 assume that the application of the id, the values of the four table becomes, 1: 0, 2: 500, 3: 200, 4: 300
In this way has the following advantages:
To achieve a globally unique ID.
It does not affect the business expansion of the database.
To get the ID have disaster recovery, can not access a single table does not affect the overall situation. If the master-slave cluster, you can solve this problem.
When the write database, do not do any other processing, direct write, since more than one mysql server, you need routing, load balancing, is simple: Database number = Random ()% n.
When reading, you need to make a simple calculation of the id, id has been found to save the database, as follows:
Database Number = Id% n * l / 100
The data in Table 2 of newly inserted id = 500,500 to 4 * 100 = 400 = 100, 100 take I / 100 = 1, that is a second database. Then use the approach of multiple data sources, can be selected.
This method supports mysql linear expansion, directly increase the database server, increasing the solution is simple, as written, remember maxID before expansion, id is calculated when read as follows:
If（id<=maxId）{
# N = number of servers before the expansion stage mysql
Database Number = Id% n * l / 100
}else
{
Database Number = Id% (n + 1) * l / 100
}
Later expansion, followed suit, remember maxId is the key.
This makes it easy expansion, increase the database server to support business growth, store more data. This approach supports multiple main mysql write, can also be achieved from the main separation using separate read and write to improve performance.
Three, mysql separate read and write from the cluster master,
A multi-master tape from the mysql binlog implemented using master-slave synchronization, not replicated among the plurality of main mysql, full use of storage, master-slave synchronization to ensure data redundancy, and easy to separate read and write, to improve performance. The method of writing data of the same application, it is necessary to select the address database corresponding to the primary database when reading data from the source, so that the system performance, data redundancy, safer and more effective.
1. Chart
Dump failed to re-upload canceled
MySQL to read and write separation refers to the process to make master write operation, so that slave processing read operation, ideal for larger scene than read operations, reduce stress master's.
Mysql-proxy implemented using separate read and write the mysql, mysql-proxy mysql is intended as a rear end of the master from the proxy server, which directly accepts the request of the client, analysis of the SQL statements, it is determined that a read or write operation, and distribution corresponding to the mysql server.
Preparing the environment:
Three hosts:
192.168.20.10 Master
192.168.20.11 Slave
192.168.20.12 Proxy
第一：192.168.20.10、11两台mysql搭建主从复制架构。
第二、在proxy机器上安装配置mysql-proxy，实现master/slave架构读写分离。
1、下载mysql-proxy-0.8.5-linux-el6-x86-64bit.tar.gz
2、安装mysql-proxy
[root@proxy install]# tar zxvf mysql-proxy-0.8.5-linux-el6-x86-64bit.tar.gz
mysql-proxy-0.8.5-linux-el6-x86-64bit/
mysql-proxy-0.8.5-linux-el6-x86-64bit/bin/
......
[root@proxy install]# mv mysql-proxy-0.8.5-linux-el6-x86-64bit /usr/local/
[root@proxy local]# ln -s mysql-proxy-0.8.5-linux-el6-x86-64bit/ mysql-proxy
[root@proxy local]# useradd -r mysql-proxy
[root@proxy local]# id mysql-proxy
uid=493(mysql-proxy) gid=486(mysql-proxy) groups=486(mysql-proxy)
3、添加mysql-proxy自启动
[root@proxy local]# cd /etc/init.d/
[root@proxy init.d]# vi mysql-proxy
!/bin/bash
mysql-proxy This script starts and stops the mysql-proxy daemon
chkconfig: - 78 30
processname: mysql-proxy
description: mysql-proxy is a proxy daemon for mysql
Source function library.
. /etc/rc.d/init.d/functions
prog="/usr/local/mysql-proxy/bin/mysql-proxy"
Source networking configuration.
if [ -f /etc/sysconfig/network ]; then
. /etc/sysconfig/network
fi
Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0
Set default mysql-proxy configuration.
ADMIN_USER="admin"
ADMIN_PASSWD="admin"
ADMIN_LUA_SCRIPT="/usr/local/mysql-proxy/share/doc/mysql-proxy/admin.lua"
PROXY_OPTIONS="--daemon"
PROXY_PID=/var/run/mysql-proxy.pid
PROXY_USER="mysql-proxy"
Source mysql-proxy configuration.
if [ -f /etc/sysconfig/mysql-proxy ]; then
. /etc/sysconfig/mysql-proxy
fi
RETVAL=0
start() {
echo -n prog: "
daemon PROXY_OPTIONS --pid-file= PROXY_ADDRESS" --user= ADMIN_USER" --admin-lua-script=" ADMIN_PASSWORD"
RETVAL= RETVAL -eq 0 ]; then
touch /var/lock/subsys/mysql-proxy
fi
}
stop() {
echo -n prog: "
killproc -p prog
RETVAL= RETVAL -eq 0 ]; then
rm -f /var/lock/subsys/mysql-proxy
rm -f $PROXY_PID
fi
}
See how we were called.
case " PROXY_PIDFILE PROXY_PID 0 {start|stop|restart|reload|status|condrestart|try-restart}"
RETVAL=1
;;
esac
exit $RETVAL
[root@proxy init.d]# chmod +x /etc/init.d/mysql-proxy
[root@proxy init.d]# chkconfig --add mysql-proxy
[root@proxy init.d]# vim /etc/sysconfig/mysql-proxy
Options for mysql-proxy
ADMIN_USER="admin"
ADMIN_PASSWORD="admin"
ADMIN_ADDRESS=""
ADMIN_LUA_SCRIPT="/usr/local/mysql-proxy/share/doc/mysql-proxy/admin.lua"
PROXY_ADDRESS=""
PROXY_USER="mysql-proxy"
PROXY_OPTIONS="--daemon --log-level=info --log-use-syslog --plugins=proxy --plugins=admin --proxy-backend-addresses=192.168.20.10:3306 --proxy-read-only-backend-addresses=192.168.20.11:3306 --proxy-lua-script=/usr/local/mysql-proxy/share/doc/mysql-proxy/rw-splitting.lua"
[root@proxy init.d]# vim /usr/local/mysql-proxy/share/doc/mysql-proxy/admin.lua
function set_error(errmsg)
proxy.response = {
type = proxy.MYSQLD_PACKET_ERR,
errmsg = errmsg or "error"
}
end
function read_query(packet)
if packet:byte() ~= proxy.COM_QUERY then
set_error("[admin] we only handle text-based queries (COM_QUERY)")
return proxy.PROXY_SEND_RESULT
end
local query = packet:sub(2)
local rows = { }
local fields = { }
if query:lower() == "select * from backends" then
fields = {
{ name = "backend_ndx",
type = proxy.MYSQL_TYPE_LONG },
{ name = "address",
type = proxy.MYSQL_TYPE_STRING },
{ name = "state",
type = proxy.MYSQL_TYPE_STRING },
{ name = "type",
type = proxy.MYSQL_TYPE_STRING },
{ name = "uuid",
type = proxy.MYSQL_TYPE_STRING },
{ name = "connected_clients",
type = proxy.MYSQL_TYPE_LONG },
}
for i = 1, #proxy.global.backends do
local states = {
"unknown",
"up",
"down"
}
local types = {
"unknown",
"rw",
"ro"
}
local b = proxy.global.backends[i]
rows[#rows + 1] = {
i,
b.dst.name, -- configured backend address
states[b.state + 1], -- the C-id is pushed down starting at 0
types[b.type + 1], -- the C-id is pushed down starting at 0
b.uuid, -- the MySQL Server's UUID if it is managed
b.connected_clients -- currently connected clients
}
end
elseif query:lower() == "select * from help" then
fields = {
{ name = "command",
type = proxy.MYSQL_TYPE_STRING },
{ name = "description",
type = proxy.MYSQL_TYPE_STRING },
}
rows[#rows + 1] = { "SELECT * FROM help", "shows this help" }
rows[#rows + 1] = { "SELECT * FROM backends", "lists the backends and their state" }
else
set_error("use 'SELECT * FROM help' to see the supported commands")
return proxy.PROXY_SEND_RESULT
end
proxy.response = {
type = proxy.MYSQLD_PACKET_OK,
resultset = {
fields = fields,
rows = rows
}
}
return proxy.PROXY_SEND_RESULT
end
4、启动proxy服务：
[root@proxy bin]# service mysql-proxy start
Starting /usr/local/mysql-proxy/bin/mysql-proxy: [ OK ]
[root@proxy bin]# ss -nalp|grep mysql
LISTEN 0 128 *:4041 : users:(("mysql-proxy",28520,11))
LISTEN 0 128 *:3306 : users:(("mysql-proxy",28520,10))
[root@proxy bin]# mysql -uadmin -padmin -h192.168.20.12 --port=4041
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.0.99-agent-admin
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SELECT * FROM backends;
+-------------+-----------------+---------+------+------+-------------------+
| backend_ndx | address | state | type | uuid | connected_clients |
+-------------+-----------------+---------+------+------+-------------------+
| 1 | 192.168.20.10:3306 | unknown | rw | NULL | 0 |
| 2 | 192.168.20.11:3306 | unknown | ro | NULL | 0 |
+-------------+-----------------+---------+------+------+-------------------+
2 rows in set (0.00 sec)
配置完成。
第三、测试
[root@master ~]# su – mysql
-bash-4.1$ mysql -uroot -p12345
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 22
Server version: 5.6.37-log MySQL Community Server (GPL)
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
mysql>
mysql> GRANT ALL ON . TO 'alex'@'192.168.20.%' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.01 sec)
mysql>
主从机器上：
tcpdump -i eth0 -nn -XX ip dst 192.168.20.10 and tcp dst port 3306
tcpdump -i eth0 -nn -XX ip dst 192.168.20.11 and tcp dst port 3306
进行抓包观察。
PROXY机器上操作：
[root@proxy bin]# mysql -ualex -p123456 -h192.168.20.12 --port=3306 （这个地方连接3306）
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 26
Server version: 5.6.37-log MySQL Community Server (GPL)
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>create database alex;
mysql>use mysql;
mysql>select * from user \G
[root@proxy ~]# mysql -uadmin -padmin -h192.168.20.12 --port=4041
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.0.99-agent-admin
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SELECT * FROM backends;
+-------------+-----------------+-------+------+------+-------------------+
| backend_ndx | address | state | type | uuid | connected_clients |
+-------------+-----------------+-------+------+------+-------------------+
| 1 | 192.168.20.10:3306 | up | rw | NULL | 0 |
| 2 | 192.168.20.11:3306 | up | ro | NULL | 0 |
+-------------+-----------------+-------+------+------+-------------------+
2 rows in set (0.00 sec)
成功实现读写分离。
总之，mysql开源且功能强大，作为业界使用量最广的数据库，采用这个办法简单巧妙地实现了采用自增id对数据库的分布式使用，轻松支持分库分表，线性扩容，还支持读写分离，而方法简单易行，把解决此问题的成本和复杂度降到0，是一个值得使用的好办法，必将为广大IT项目和企业带来益处。
