hello everybody junior partner Hello everyone, I am a small stack monarch, in this issue we share research on mysql middleware, that is, read and write data layer separation and load balancing, hoping to be able to help you a small partner in practical applications.
The next issue we will continue to share the series go to explain the language, as well as later in life, we will also share courses, including big data, artificial intelligence, block chain and so on, a lot of hope that we can learn and share to the side of a small partner, we together with the progress and growth.
mysql-proxy
MySQL Proxy is such an intermediate layer proxy, simply put, MySQL Proxy is a connection pool is responsible for connection requests will be forwarded to the foreground application background database, and by using lua script, you can implement complex connection control and filtering.
Enabling separate read and write and load balancing. For applications, MySQL Proxy MySQL Proxy is completely transparent to the more powerful function is to achieve a "read-write separation", the basic principle is to let the Lord handle transactional database queries.
Its execution flow as shown:
Let SELECT query processing from the library. Database replication is used to query the transaction due to changes sync to the cluster from the library.
mysql-proxy mysql official middleware can provide load balancing, separate read and write, and other failover, but it does not support a large amount of data sub-library sub-table and poor performance.
Official website:
https://downloads.mysql.com/archives/proxy/
Here are several that can replace the mysql open source middleware products, Atlas, cobar, tddl, let's see what some of their respective advantages and new features of it.
Atlas
Atlas is composed of Qihoo 360, Web Platforms Group infrastructure team development and maintenance of a data item interlayer MySQL protocol.
It is based on mysql-proxy 0.8.2 version on, it is optimized, added some new features. Atlas 360 Internal mysql service operation, read and write requests carried per day up of billions.
Source:
Github: https://github.com/Qihoo360/Atlas
But according to the latest news because Atlas has been able to meet most business needs, so had to stop updating.
Altas architecture:
Atlas is located between the application and MySQL, which implements the MySQL client-server protocol, as communication with the application server, as both a client and MySQL communication.
It shields the details of the application DB, and in order to reduce the burden of MySQL, it maintains a connection pool.
Some of the new features Altas:
1, does not affect the main library down to read down the main library, Atlas automatically removed down the main library, the write operation fails, read operations are not affected. Down from the library, Atlas automatically removed from the library is down, there is no impact on the application. In mysql official proxy in the main library down, nor available from the library.
2 via the management interface, to simplify management work, the upper and lower DB completely transparent to the application, while offline manually.
3. The read and write their separation, in order to solve the presence of separate read and write once read and written wanted time there may be a case where a master-slave synchronization delay, may increase of Altas / SQL statements before Master / read request can be made mandatory to the main library. Master library set number, separated by commas, can be provided from the library and the number of weights to achieve load balancing.
4. realize his sub-table, with the sub-table fields required. And supports SELECT, INSERT, UPDATE, DELETE, REPLACE statement. Finally, support merging and sorting multiple sub-query results table.
Here have Tucao points table function Atlas can not achieve distributed points table, all of the child table and all sub-tables must be completed in advance at the same table with a database DB's, Atlas does not automatically create a menu.
The main official function logic written in lua script 5. Before the use, low efficiency, Atlas rewritten in C, QPS (for a particular query server traffic measure how much of the processing within the specified time) increase, latency (an instruction issued to arrive elapsed time) decreases.
6. enhance security: Atlas user connected via pwds parameter configuration file access control. Ip is filtered by Atlas has permission to connect client-ips parameters.
Log records all SQL statements Altas process, including the IP client, DB actual execution of the statement, execution success, the time spent performing
7. GR smooth restart function is provided lvs-ips parameters in the configuration file, otherwise restart instantaneously Altas those SQL request will fail.
DBProxy
DBProxy review by the US Mission DBA team technical engineering development (Beijing) data maintained by a middle layer of MySQL protocol. It Atlas basis Qihoo 360 companies on open source, to modify some of the bug, and added many features.
Currently DBProxy widely used in the US group review, including the US group pay, hotels and tourism, take-away, buy and other product lines within the company to develop comprehensive DBProxy go on github, open source and internally consistent. Currently only supports MySQL (Percona) 5.5 and 5.6.
The main function:
Separate read and write
From the library load balancing
IP过滤
分表
DBA可平滑上下线DB
自动摘除宕机的DB
监控信息完备SQL filter
Configuring traffic from the library
After using DBProxy, applications only need to set the address in the connection string DBProxy, the entire database need not be concerned node cluster;
DBProxy internal load balancing, separate read and write;
Slave on offline by the DBA operation point on an automated system of the mouse operation can be completed.
This greatly reduces the DBA's work and application developers; but without DBProxy These work by RD to achieve, for the introduction of DBProxy manageability and convenience of the system has a very big help.
DBProxy improvement is based Qihoo 360 open source Altas, illustrates the point improvement on official documents are:
提供了丰富的监控信息，大量参数可配置化并且支持动态修改
对原有的诸如日志等模块进行了优化，性能提升明显
开源版本即为目前美团点评内部使用版本，并将一直对源码及其文档进 行维护
开源地址：
https://github.com/Meituan-Dianping/DBProxyAmoeba
Amoeba is a MySQL to as the underlying data store, and MySQL applications Proxy protocol interface. It is concentrated in response to the application request, the user according to the rules set in advance, the SQL execution request to a specific database.
Based on this load balancing can be achieved, separate read and write, and high availability requirements. Compared with the official MySQL MySQL Proxy, the authors emphasize that the amoeba convenient configuration (XML-based configuration file, with SQLJEP grammar writing rules, simpler than MySQL Proxy-based lua script).
Amoeba SQL requests corresponding to a router, for the purpose of load balancing, separate read and write, provides a mechanism for high availability, they are not fully realized.
Users need to combine the use of MySQL Replication and other mechanisms to achieve replica synchronization. amoeba to the underlying database connection management and routing mechanism is also used to achieve pluggable, third parties can develop more advanced strategies implemented instead of the class.
Amoeba advantages:
a). After a complex data source data segmentation Integration
b). Effect of providing data segmentation rules and reduce data segmentation rules to bring the database
c). lowered client database connection
d). Routing separate read and write
Amoeba Disadvantages:
a), is not yet supported Affairs
b), does not support stored procedures (near term support)
c), is not suitable for conducting data from the scene amoeba large amount of data query or queries is not appropriate (such as a request to return 10w even more than the case of data)
d), does not support the sub-library sub-table, Amoeba currently only do sub-database instance, each node is sliced need to maintain consistent database table structure:
Official Website:
http://wiki.hexnova.com/display/amoeba/Home
Unfortunately, age-old, last updated site is located 2012
alibaba.cobar
Cobar Alibaba (B2B) division develops distributed processing system for relational data, it may look like a traditional database, like to provide you with massive data services in a distributed environment. So why do we use specifically talk about it, or say cobar-- can you do?
cabar advantage summary:
And access data from a centralized to distributed change:
Cobar horizontal support table split into a plurality of parts into separate database table to achieve a level resolution
Cobar also supports different tables into different libraries
In most cases, users will note that the above two methods used in combination! : Cobar does not support a table, for example, test table split into test_1, test_2, test_3 ..... in the same library, you must watch after the split into separate libraries to implement distributed.
Solve the problem of excessive number of connections.
Less invasive for business code.
Provide data node failover, HA: (1) Cobar of two active switching triggering methods for a user to manually trigger a heartbeat Cobar statement is automatically triggered after the abnormality is detected.
Then, when an abnormal heartbeat to the host, switching to the standby machine, if the host recovery, require users to manually cut the work back to the host, of Cobar does not automatically switch back to the host when the host restored, unless the heartbeat preparation machine also returns an exception.
Check the MySQL standby Cobar only exception, do not care data synchronization between the primary, so the user needs to use the prior configuration in the standby Cobar MySQL bidirectional synchronization.
cobar drawback: the open source version of the database only supports mysql, and does not support separate read and write.
TDDL (core code is not open source)
Taobao, according to their operational characteristics developed TDDL (Taobao Distributed Data Layer nickname: big head) framework, mainly to solve the sub-library sub-table for data replication between heterogeneous database and application transparency, which is based on a jdbc datasource implemented centralized configuration, the main and backup, separate read and write, the dynamic database configuration functions.
TDDL located position (tddl universal data access layer deployed on the client jar package, for the user's route to the specified SQL database):
Taobao is too long to process the data library, the upper system to connect multiple databases, called DBRoute the middle of the route to unified access to data.
DBRoute operate on the data, the integration of multi-database data, so that the upper system like operating a database as operating multiple libraries. But with the amount of data grows, database tables for the points system have higher requirements.
For example, your product data to the ten billion level, when any of the library can not be stored, and then divided into two, four, eight, 16, 32 ...... until 1024, 2048.
Into so much data can be stored, then how query it?
At this time, middleware, data queries will be able to carry this burden, and its upper layer, it must be like to query a database to query the same data, but also as a database query as fast (each query is completed within a few milliseconds) , TDDL to undertake such a work.
The main advantage:
Standby database and dynamically switch
With weight separate read and write
Single-threaded read retry
Centralized data source information management and dynamic change
Stable jboss datasource
Mysql and oracle database support
Jdbc-based specifications, can be easily extended to support the achievement jdbc specification of data sources
No server, client-jar forms exist, direct application database
Read and write times, concurrency process control, dynamically changeable 10. The log may be analyzed print log flow control, dynamically changing TDDL must rely diamond distribution center (diamond is a system management internal Taobao use durable configuration, the current internal Taobao absolute most systems are configured to carry out the unified management by the diamond, while diamond has open source).
Source:
https://github.com/alibaba/tb_tddl
TDDL complexity is relatively high. And has fallen into disrepair the
Less current document published by the only open source dynamic data sources, sub-table sub-library section is not yet open, but also need to rely on diamond, is not recommended.
Today's share went to here came to an end, if you like my share of trouble you click again, share, or comment, I was Xiaozhan king, we see the next issue, thanks to a worship -
This article from the blog article multiple platforms OpenWrite release!