MySQL read-write separation and load balancing

Author: Zen and the Art of Computer Programming

1 Introduction

With the increasing traffic of Internet websites, more and more websites are facing the challenge of database pressure. A single server cannot handle such a large amount of visits, so the database needs to be split horizontally so that the server can handle more requests at the same time. Under normal circumstances, MySQL master-slave replication is a method to achieve read-write separation, that is, the master server is responsible for writing and synchronizing to the slave server, and the slave server is only responsible for reading. However, due to problems such as delay and asynchronous replication in master-slave replication, performance is poor in high concurrency scenarios. In order to solve the problem of MySQL master-slave replication and improve the processing capabilities of the database, the official MySQL team launched an open source software called mysql-proxy as middleware. It can realize the function of separation of reading and writing, thereby reducing the pressure on the database and improving the processing capacity of the database. This article will introduce mysql-proxy in detail, including its working principle, configuration method and specific optimization methods.

2. Separation of reading and writing

Read Write Splitting is a database optimization strategy commonly used in database clusters to improve the processing capabilities of the database server. Under normal circumstances, for a master database (Master), the load is distributed among multiple slave databases (Slave) according to a certain strategy. When a query statement needs to access the master database, it is only sent to the master database for execution; while for updates Or insert a statement, all slave libraries will be notified for execution at the same time. The benefits of doing this mainly include the following aspects:

1. Improve the processing capacity of the database server: Due to load balancing, database requests can be distributed to different database servers, thereby improving the processing capacity of the entire database cluster.

2. Improve the availability of database services: When a database server fails, only the slave database on that server is affected, and other servers can still provide normal services.

3. Increase disaster tolerance: When the database in a certain area fails, you only need to shut down the server in that area, and other servers can still provide normal services.

4. Reduce the pressure on the main database: Since all queries are sent only to the main database, the pressure on the main database will be reduced and the database will respond faster.

Advantages of MySQL read-write separation:

1. Share the pressure on the main library: After reading and writing are separated, the write operation pressure on the main library will be halved. At the same time, since all data operations are performed through the slave library, the pressure on the main library will be greatly reduced.

2. Improve database processing capabilities: After reading and writing are separated, the database processing capabilities are significantly improved.

3. Improve data backup and disaster recovery capabilities: By backing up the data of the master database to the slave database, the risk of data loss after a catastrophic failure of the master database can be effectively avoided.

3.Introduction to mysql-proxy

mysql-proxy is an open source database middleware developed by Alibaba Group and has the following characteristics:

1. Supports read-write separation: mysql-proxy can realize the read-write separation function of the database, that is, it can direct some query requests to specific slave libraries to reduce the load on the main library.

2. Load balancing: mysql-proxy supports two load balancing strategies based on connection and SQL parsing. It can automatically identify load imbalances and automatically adjust the back-end connection pool information.

3. Support hot switching: mysql-proxy can dynamically add or delete slave database nodes without stopping the service, realizing dynamic switching of read-write separation.

4. Data integrity guarantee: mysql-proxy uses the binlog dump protocol to synchronize data changes in the main database to the slave database in real time to ensure data consistency.

5. Strong protocol compatibility: mysql-proxy maintains good compatibility with the native MySQL client. You can directly connect to mysql-proxy by modifying the client's connection parameters.

6. Flexible configuration: mysql-proxy supports hot loading of configuration files, allowing dynamic modification of configuration information during operation to achieve seamless switching.

4.mysql-proxy installation and deployment

4.1 Install the compilation environment

If your system does not have gcc or make tools installed, please install them first, and then install the mysql-proxy source package.

yum install -y gcc make

4.2 Download source code

The latest version of mysql-proxy is 1.9. We can also choose other versions to download, and then compile and install:

wget https://github.com/mysql-net/MySqlConnector/releases/download/v1.3.14/mysql-connector-net_1.3.14.tar.gz
tar zxvf mysql-connector-net_1.3.14.tar.gz && cd mysql-connector-net_1.3.14/
./build.sh
cp MySqlConnector.dll /usr/local/lib/

4.3 Configuration file description

After the installation is complete, go to the /etc/myproyx/ directory and view the my.ini file. The key information is as follows:

[server]
# server_id is used to identify this proxy instance among a set of proxies
server_id = 1

[mysql]
host=127.0.0.1 # 连接到的MySQL主机地址，通常设置为master节点的IP地址
port=3306 # MySQL端口号
user=root # 登录用户名
password=<PASSWORD> # 登录密码
schema=test # 设置默认的数据库名

[readconnroute]
# readconnroute specifies the destination of SELECT queries
rule1=slave|localhost:3306|test

[writeconnroute]
# writeconnroute specifies the destination of INSERT, UPDATE and DELETE queries
rule1=master|localhost:3306|test

[app]
enable_heartbeat=true # 是否开启心跳检测功能

Among them, hostthe field specifies which MySQL node to connect to, which is usually set to the IP address of the master node. The portfield specifies the MySQL port number, userand passwordthe user name and password used to log in to MySQL respectively, which schemaare used to set the default database name. readconnrouteand writeconnroutetwo subkeys define the target nodes of the SELECT and INSERT/UPDATE/DELETE statements respectively. The format is: {type}|{IP}:{port}|{database name}. What we need to pay attention to here is rule1Article 1, which represents the matching rule. The first part slaveindicates the type, indicating that the rule applies to SELECT requests; the latter part localhost:3306indicates the target node, which is the address to which requests sent to the master node are forwarded; the third parameter testindicates which database the request will be sent to (the default we set database). Other sub-items include:

• charset: Specifies the character encoding format.
• sqlblacklist: You can specify that certain SQL statements are prohibited from being sent to the slave database.
• tls: Configure TLS encrypted transmission.

4.4 Start mysql-proxy

Start mysql-proxy:

/usr/local/mysql-proxy/bin/myproyx --defaults-file=/etc/myproyx/my.ini

If an abnormal log appears, you can follow the prompts to troubleshoot the cause of the error. After successful startup, you can see relevant information in the command line window, which means it is running normally.

4.5 Testing

4.5.1 View master-slave status

First, we check the master-slave status through the mysql command line tool:

mysql -h 127.0.0.1 -P 3306 -u root -p123456 -e "show slave status\G"

Seconds_Behind_MasterIf the value in the returned result is not NULL, it indicates that the current master-slave replication status is normal.

4.5.2 Query data

Next we test whether the effect of read-write separation is correct. First open another window and connect to the machine where mysql-proxy is located:

mysql -h 127.0.0.1 -P 4040 -u root -p123456 -D test

Note that you need to specify the connection to port 4040, not port 3306. Then try running some simple queries and insert statements:

-- 查询数据
select * from users where id=1;

-- 插入数据
insert into users (name, age) values ('Tom', 25);

It can be seen that all queries will only be sent to the main database for execution, and all insert statements will be sent to both the main database and the slave database for execution. We can repeat this process to verify the effect of read and write separation.