Article directory

MySQL database - master-slave replication and read-write separation

MySQL database - master-slave replication and read-write separation

Preface, common application scenarios of Mysql replication

• Read-write separation improves query access performance and effectively reduces access pressure on the main database.

• Real-time disaster recovery, when the primary database fails, it can quickly switch to the secondary database.

• Data aggregation, which can synchronously aggregate multiple master databases into one slave database, which is convenient for statistical analysis of data.

1. Introduction to the principle of Mysql master-slave replication

1. Mysql asynchronous and semi-synchronous replication

Traditional Mysql replication provides a simple master-slave replication approach, with one master, and one or more slaves. The master executes and commits transactions, then sends them (asynchronously) to slaves for re-execution (in statement-based replication) or application (in row-based replication). This is a shared-nothing system, and by default all server members have a complete copy of the data. (Version 5.7 asynchronous performance is very good, no delay)

There is also semi-synchronous replication, which adds a synchronization step to the protocol, which means that the master node needs to wait for the slave node to confirm that it has received the transaction when submitting, and only then can the master node continue to submit the operation. (The delay is relatively high, which affects the read and write performance of the master. If you want to ensure high data consistency, choose to use semi-synchronization.)

In the above two pictures, you can see the graphical display of the traditional asynchronous MySQL replication protocol (and semi-synchronous). The blue arrows indicate the information exchange between different servers or between server and client applications.

2.Mysql master-slave replication process

• Enable the binlog log, enable the relaylog log from the slave library, and re-parse the application to the slave library by transferring the binlog of the master library to the slave library.

• Replication requires three threads (dump, io, sql) to complete.

• Replication is an asynchronous process. Master-slave replication is asynchronous logical SQL statement-level replication.

3. Master-slave replication prerequisites

• The master server must have binary logging turned on.

• Two servers (or multiple instances) are required.

• The slave server requires a data initialization.

• If the master and slave servers are newly built, initialization can be omitted.

• If the master server has been running for a long time, you can restore the master database data to the slave database through backup.

• The master must have a user that replicates requests to the slave.

• The slave library needs to have relay-log settings to store the binary log sent from the master libraryshow variables like'%relay%;

• During the first replication, the slave library needs to change the master to connect to the master library,

• Change master information needs to be stored in master.infoshow variables like%master info%';

• How does the slave library know that new changes have taken place in the master library? The applied relay-log information recorded by relay-log.info,

• Threads involved in the replication process:

❁ The slave library will open a lO thread (thread), which is responsible for connecting to the main library, requesting binlog, receiving binlog and writing to relay-log.

❁ The slave library will open a SQL thread (thread), which is responsible for executing the time in the relay-log.

❁ The main library will open a dump thread (thread) responsible for responding to requests from the IO thread.

4.Mysql master-slave replication implementation

★★Process★★:

• Via the binary log

• At least two (Master, Slave)

• The binary log of the master server is "taken" to the slave server and run again

• Connecting two machines through the network generally has a delayed state, which can also be said to be asynchronous.

• The slave library is linked to the master library by manually executing the change master to statement, providing all the user, password, port, and ip conditions for the connected user.

• And let the slave library know the starting position of the binary log (file name position number).

• Start slave synchronization service start slave.

• A link is established between the IO of the slave library and the dump thread of the main library.

• According to the file number and position number provided by the change master to statement from the slave library, the IO thread initiates a binlog request to the master library.

• The main library dump thread sends the local binlog to the slave library IO thread in the form of events according to the slave library's request.

• Receive binlog events from the library IO thread and store them in the local relay-log. The transmitted information will be recorded in master.info.

• Apply the relay-log from the library SQL thread, and record the applied record in relay-log.info. By default, the applied relay will be automatically cleaned and purged.

5. Three core formats of Mysql replication

The working principle of replication is that the database modification is recorded to the binlog log and passed to the slave, and then the slave restores it locally. The format of time recorded to binlog will be different.

① statement based replication (statement based replication)

Based on the main library, SQL statements are written to the bin log to complete the replication.

②row based replication

Based on the main library, the information of each row of data changes is written into the bin log as the time to complete the log. The default is row-level replication because it is more logical than statement replication.

③Mixed based replication

A combination of the two above. By default, statement-based replication is preferred, and only when some statements are incomplete based on statement-based replication will they be automatically switched to row-based data replication.

6. Mysql master-slave replication setup (one master and one slave)

Environment deployment :

master：192.168.222.20

slave：192.168.222.10

Port: 3306

master server master configuration

#配置my.cnf文件
[root@master ~]# vim /etc/my.cnf
[mysqld]
…………
server-id = 1
log-bin=master-bin				#添加，主服务器开启二进制日志
binlog_format = MIXED
log-slave-updates=true			#添加，允许slave从master复制数据时可以写入到自己的二进制日志

#重启mysqld
[root@master ~]# systemctl restart mysqld
[root@master ~]# mysql -u root -p

#Mysql授权远程主机登录
mysql> GRANT REPLICATION SLAVE ON *.* TO 'myslave'@'192.168.222.%' IDENTIFIED BY '111';	 #给从服务器授权
mysql> FLUSH PRIVILEGES; 						#刷新权限

insert image description here

Slave configuration from server

[root@slave ~]# vim /etc/my.cnf
…………
server-id = 2								#修改，注意id与Master的不同，两个Slave的id也要不同
relay-log=relay-log-bin						#添加，开启中继日志，从主服务器上同步日志文件记录到本地
relay-log-index=slave-relya-bin.index		#添加，定义中继日志文件的位置和名称，一般和relay-log在同一目录
relay_log_recovery = 1                      #选配项
#当 slave 从库宕机后，假如 relay-log 损坏了，导致一部分中继日志没有处理，则自动放弃所有未执行的 relay-log，并且重新从 master 上获取日志，这样就保证了relay-log 的完整性。默认情况下该功能是关闭的，将 relay_log_recovery 的值设置为 1 时， 可在 slave 从库上开启该功能，建议开启。
#重启服务
[root@slave ~]# systemctl restart mysqld
#进入数据库
[root@slave ~]# mysql -u root -p

mysql> CHANGE master to master_host='192.168.222.20',master_user='myslave',master_password='111',master_log_file='master-bin.000004',master_log_pos=604;					#配置同步，注意 master_log_file 和 master_log_pos 的值要与Master查询的一致

mysql> start slave;						#启动同步，如有报错执行 reset slave;
mysql> show slave status\G					#查看 Slave 状态
//确保 IO 和 SQL 线程都是 Yes，代表同步正常。
Slave_IO_Running: Yes				#负责与主机的io通信
Slave_SQL_Running: Yes				#负责自己的slave mysql进程

insert image description here

If it is not YES then how to troubleshoot?

Slave_IO线程相对比较简单，一般不容易出错。如果显示为No，则有可能以下原因导致：
    * 网络问题（检查网络连接是否有问题）
    * 权限问题，例如在配置slave同步时因为slave访问master没有权限导致
    * master上的binlog文件误删或者其他问题导致的master库突然停止更新binlog日志。解决方案是找到同步的点和binlog文件，重新change master
 
相对的Slave_SQL线程就比较容易出错。例如人为的在从库插入一条数据，造成的主从库不一致。但此时两个线程的状态仍然是正常的，主库插入数据时，到从库同步时，从库会出现主键重复的错误。此时Slave_SQL_Running的状态变为No
 
而Last_SQL_Error, Last_SQL_Error_Timestamp会记录错误的原因和发生的时间

Slave_SQL_Running线程报错之后，会停止后续的SQL执行，因为向后执行会导致错误修复的难度增加
-- 先停止slave
stop slave;
 
-- 跳过执行语句数量
-- 此时需要查看错误日志去修复报错信息
set global sql_slave_skip_counter=1;
 
-- 开始slave
start slave;
 
-- 然后再检查一下 slave status

Verify master-slave replication effect

Create a table in the primary database and insert data to check whether the secondary database is replicated.

insert image description here

2. Mysql read and write separation

1. The principle of read-write separation

• Write only on the master server, read only on the slave server.

• The master database handles transactional queries and the slave database handles SELECT queries.

• Database replication is used to synchronize changes from transactional queries to slave databases in the cluster.

insert image description here

2. Read-write separation scheme

• Internal implementation based on program code

• Implementation based on intermediate proxy layer

▶MySQL Proxy——MySQL Official

▶Atlas - Qihoo 360

▶DBProxy——Meituan Dianping

▶Amoeba - early Alibaba

▶cober——Alibaba

▶MyCat——Alibaba

3. Amoeba+M+S cluster construction

insert image description here

3.1 Prepare the environment

Amoba server: 192.168.10.21 jdk1.6, Amoeba

Master database: 192.168.10.10 MySQL5.7

Slave1 database: 192.168.10.12 MySQL5.7

Slave2 database: 192.168.10.17 MySQL5.7

Amoba server configuration

#因为 Amoeba 基于是 jdk1.5 开发的，所以官方推荐使用 jdk1.5 或 1.6 版本，高版本不建议使用。
#将jdk-6u14-linux-x64.bin 和 amoeba-mysql-binary-2.2.0.tar.gz.0 上传到/opt目录下。

[root@amoeba ~]# cd /opt/
[root@amoeba opt]# cp jdk-6u14-linux-x64.bin /usr/local/

[root@amoeba opt]# cd /usr/local/

[root@amoeba local]# chmod +x jdk-6u14-linux-x64.bin 

[root@amoeba local]# ./jdk-6u14-linux-x64.bin
按空格到最后一行
按yes，按enter

[root@amoeba local]# mv jdk1.6.0_14/ /usr/local/jdk1.6

[root@amoeba local]# vim /etc/profile  ##存放的是jdk环境变量
export JAVA_HOME=/usr/local/jdk1.6
export CLASSPATH=$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/bin/:$PATH:$HOME/bin
export AMOEBA_HOME=/usr/local/amoeba
export PATH=$PATH:$AMOEBA_HOME/bin

[root@amoeba local]# source /etc/profile  # 刷新环境变量
[root@amoeba local]# java -version # 查看java jdk

##安装 Amoeba软件##
[root@amoeba local]# mkdir /usr/local/amoeba
[root@amoeba local]# tar zxvf /opt/amoeba-mysql-binary-2.2.0.tar.gz -C /usr/local/amoeba/

[root@amoeba local]# chmod -R 755 /usr/local/amoeba/

[root@amoeba local]# /usr/local/amoeba/bin/amoeba
##如显示amoeba start|stop 说明安装成功

Amoeba read-write separation is configured on the server, and two Slaves read load balancing

mysql> grant all on *.* to 'test'@'192.168.222.%' identified by '111';
#在一主两从上都要配置,amoeba详单与一个用户，可以操作每个数据库的操作

#amoeba服务器配置
[root@amoeba local]# cd /usr/local/amoeba/conf/

[root@amoeba conf]# cp amoeba.xml amoeba.xml.bak
[root@amoeba conf]# vim amoeba.xml				#修改amoeba配置文件

#30修改
<property name="user">amoeba</property>

#32修改
<property name="password">abc123</property>

#115修改
<property name="defaultPool">master</property>

#117去掉注释–
<property name="writePool">master</property>
<property name="readPool">slaves</property>
[root@amoeba conf]# cp dbServers.xml dbServers.xml.bak
[root@amoeba conf]# vim dbServers.xml			#修改数据库配置文件

--23行--注释掉  作用：默认进入test库 以防mysql中没有test库时，会报错
<!-- <property name="schema">test</property> -->
--26--修改
<property name="user">test</property>
--28-30--去掉注释
<property name="password">123.com</property>
--45--修改，设置主服务器的名Master
<dbServer name="master"  parent="abstractServer">
--48--修改，设置主服务器的地址
<property name="ipAddress">192.168.10.10</property>
--52--修改，设置从服务器的名slave1
<dbServer name="slave1"  parent="abstractServer">
--55--修改，设置从服务器1的地址
<property name="ipAddress">192.168.10.12</property>
--58--复制上面6行粘贴，设置从服务器2的名slave2和地址
<dbServer name="slave2"  parent="abstractServer">
<property name="ipAddress">192.168.10.17</property>
--65行--修改
<dbServer name="slaves" virtual="true">
--71行--修改
<property name="poolNames">slave1,slave2</property>

[root@amoeba conf]# /usr/local/amoeba/bin/amoeba start &	#启动Amoeba软件，按ctrl+c 返回
[root@amoeba conf]# netstat -anpt | grep java				#查看8066端口是否开启，默认端口为TCP 8066

Test read-write separation

做下面的实验时，记得在从服务器上把所有的同步关掉！！

#在一台其他机器上使用yum快速安装MySQL虚拟客户端
[root@client ~]# yum install -y mariadb-server mariadb
[root@client ~]# systemctl start mariadb.service
#测试：
#通过amoeba服务器代理访问mysql ，在通过客户端连接mysql后写入的数据只有主服务会记录，然后同步给从服务器
[root@client ~]# mysql -u amoeba -pabc123 -h 192.168.10.21 -P8066

END