Kingshard is a high-performance MySQL Proxy project developed by Chen Fei (@flikecn) of Zhuhai Jinshan WPS cloud platform team using Go. Kingshard is committed to simplifying the operation of MySQL database and table separation in order to meet the basic functions of read-write separation; Easy and smooth expansion of MySQL database through kingshard.
Chen Fei (@flikecn). He graduated from the University of Electronic Science and Technology of China with a master's degree in 2013. In the same year, he joined the Web Platform Department of Qihoo 360, engaged in the design and development of Atlas database middleware. In early 2015, he joined the Zhuhai Jinshan WPS cloud platform team. The new team mainly uses Go as the development language. It is precisely because of the high development efficiency of the Go language in the new work that the idea of writing a simple and easy-to-use Go version of MySQL Proxy sprouted.
Operations supported by sharding
Currently kingshard sharding supports insert, delete, select, update and replace statements, and all these five types of operations support cross-sub-tables. However, write operations only support cross-subtables on a single node, and select operations can cross nodes and subtables.
Main functions of sharding:
1. Read and write separation.
2. Split tables across nodes.
3. Client IP access control.
4. Smooth on-line or off-line DB, no front-end application perception.
kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read/write sqls. Now it supports basic SQL statements (select, insert, update, replace, delete). The most important feature is the sharding function. Kingshard aims to simplify the sharding solution of MySQL. The Performance of kingshard is about 80% compared to connecting to MySQL directly.
Feature
1. Basic Function
- Splits reads and writes
- Client's ip ACL control.
- Transaction in single node.
- Support limitting the max count of connections to MySQL database.
- Support setting the backend database online or offline dynamically.
- Supports prepared statement: COM_STMT_PREPARE, COM_STMT_EXECUTE, etc.
- Support multi slaves, and loading banlance between slaves.
- Support reading master database forcely.
- Support last_insert_id().
- Support MySQL backends HA.
- Support set the charset of proxy.
- Support SQL blacklist.
- Support dynamically changing the config value of kingshard.
2. Sharding Function
- Support hash,range and date sharding across multiple nodes.
- Support sending sql to the specified node.
- Support most commonly used functions, such as
max, min, count, sum, and also supportjoin, limit, order by,group by.
Sub-table migration scheme based on kingshard
It is very convenient to dynamically migrate sub-tables through kingshard, so as to ensure that the load pressure of MySQL nodes is not too high. The general steps are as follows:
1) Start data migration through an automatic data migration tool.
2) The data difference is less than a certain critical value, blocking Laozi table write operation (read-only)
3) Wait for the new sub-table data to be synchronized
4) Change the routing rules of the corresponding sub-table in the kingshard configuration file.
5) Delete the child table on the old node.
Now more and more Internet companies still in heavy use MySQL to store various types of relational data. With the amount of data and traffic increasing, developers had to consider some new MySQL-related problems.
1)读写分离Read/Write Splitting. With the increasing traffic sent by the front-end applications, one instance of MySQL can not hold all the queries. At this time, we have to send the read queries to the slaves for load balance.
2)单表容量The capacity of one table in MySQL. If in the begining of system design, you have not considered the table sharding, that will make it difficult to keep your system high-performance.
3)MySQL的维护MySQL maintenance operation. Without proxy you should configure the master and slaves host in your source code. When you upgrade the MySQL server, the front-end applications have to make relevant regulation.
4)连接池Connection pool. The front-end applications send queries by creating a new connection with MySQL, and close the connection when they don't need to send queries any more. The extra performance cost of these operations can not be ignored. If a connection pool is added between the front-end applications and MySQL, and the front-end applications can pick a connection from the connection pool, it will enhance the performance of your system.
5)SQL日志SQL logs. When the program has problems, usually we want to get some SQL logs sent by the program. For example, We want to know which SQL was sent to which DB backend. By checking the log, it can help us locate the problem more quickly.
Faced with these problems, we can implement every function in the client code. But this also makes the client less flexible. I have been working on database development for years, and I believe we can use a MySQL proxy to solve the problems more effectively, which is why I created this project. In this document, I will show you how kingshard solve the above problems.
1)读写分离问题。由于前端应用访问量增加，单台MySQL不足以支撑整个系统的写入和查询操作。这时候，我们不得不将一些耗时的查询操作分散到多个slave上。
2)单表容量问题。如果在系统设计之初，没有考虑到分表问题。随着数据量的增长，单表容量越来越大。作者见过单表容量5亿条记录，然后一个简单的delete操作都会引起系统慢日志，而且有可能导致MySQL IO瞬发性的飙升。很多同学可能会想到，在查询的字段上加上索引，但当数据量增长到这么大的时候，即使加上索引效果也不明显了。归根结底，就是单表数据量太大，导致MySQL即使通过索引定位数据，仍然需要扫描很多记录。
3)数据库的运维问题。如果在代码中配置主库和从库host，系统运行当然也是没问题的。但这样大大增加了运维工作的压力，比如：MySQL数据库IO压力由于访问量的增加居高不下，DBA需要添加一台slave，这时候就不得不修改代码，然后打包并上线。还有很多非常实际的例子，在这就不一一列举。
4)连接池。前端应用频繁连接MySQL，由此给MySQL带来的额外性能消耗也是不容忽视的。如果通过增加一个连接池，每个DB缓存一定数量的MySQL连接，当有应用需要连接后端的MySQL，直接从连接池里取出一个已建好的连接来发送SQL请求，这样会大大加快数据查询速度。而且可以降低MySQL的性能消耗。
5)SQL日志。在程序出现问题时，我们希望得到一些SQL日志，比如，什么时刻哪条SQL发送到哪一台DB上了。通过查看这种日志能够帮助我们快速定位问题。