Introduction: Earlier we had an article on how to implement MySQL master-slave replication. Today, we will introduce how to ensure the consistency of MySQL master-slave data. To put it bluntly, the principle of MySQL master-slave replication is that when the master writes data, it will leave a write log, and the slave will write data according to the log left by the master to imitate its data execution process. After understanding the principle of MySQL master-slave replication, you can clearly understand that there are two steps that may lead to master-slave inconsistency:
1、mater日志写入不成功导致slave不能正常模仿。
2、slave根据master日志模仿时写入不成功。
Today we will solve the problem of inconsistency between master and slave from these two dimensions.
1. Ensure the unity of MySQL (master side) log and data, and handle abnormal situations such as power failure and downtime.
A few more beeps :
1. As a pluggable database system, MySQL supports a plug-in storage engine. It is designed to be divided into the Server layer and the Storage Engine layer.
2. At the Server layer, MySQL records the Binlog binary log of various operations of the database in the form of events. Its basic core functions are: replication and backup. In addition, we combined the needs of diversified business scenarios and built a powerful MySQL ecosystem based on the characteristics of Binlog, such as: DTS, unitization, real-time synchronization between heterogeneous systems, etc. Binlog has long become an indispensable part of the MySQL ecosystem. Module.
3. At the Storage Engine layer, InnoDB, as a more general storage engine, has a good balance between high availability and high performance, and has long become the first choice for using MySQL (PS: the official starting from MySQL 5.5.5, will InnoDB serves as the default storage engine for MySQL). Like most relational databases, InnoDB uses WAL technology, that is, InnoDB Redo Log records physical changes to data files, and ensures that the log is always first. Before persisting the data file, ensure that the previous redo log has been written to disk. Whether Binlog and InnoDB Redo Log are placed on the disk will directly affect the extent to which the instance's data can be recovered after an abnormal downtime. InnoDB provides corresponding parameters to control the log writing method and strategy when the transaction is committed, for example:
innodb_flush_method:控制innodb数据文件、日志文件的打开和刷写的方式，建议取值：fsync、O_DIRECT。
innodb_flush_log_at_trx_commit:控制每次事务提交时，重做日志的写盘和落盘策略，可取值：0，1，2。
当innodb_flush_log_at_trx_commit=1时，每次事务提交，日志写到InnoDB Log Buffer后，会等待Log Buffer中的日志写到Innodb日志文件并刷新到磁盘上才返回成功。
sync_binlog：控制每次事务提交时，Binlog日志多久刷新到磁盘上，可取值：0或者n(N为正整数)。
不同取值会影响MySQL的性能和异常crash后数据能恢复的程度。当sync_binlog=1时，MySQL每次事务提交都会将binlog_cache中的数据强制写入磁盘。
innodb_doublewrite：控制是否打开double writer功能，取值ON或者OFF。
当Innodb的page size默认16K，磁盘单次写的page大小通常为4K或者远小于Innodb的page大小时，发生了系统断电/os crash ，刚好只有一部分写是成功的，则会遇到partial page write问题，从而可能导致crash后由于部分写失败的page影响数据的恢复。InnoDB为此提供了Double Writer技术来避免页断裂(partial write)的发生。
innodb_support_xa:控制是否开启InnoDB的两阶段事务提交.默认情况下，innodb_support_xa=true，支持xa两段式事务提交。
Through the above beeps, we get the following configuration:
#以下配置保证bin-log写入后事务提交流程会变成两阶段提交，这里的两阶段提交并不涉及分布式事务，mysql把它称之为内部xa事务
innodb_support_xa=ON
#以下配置能够保证不论是MySQL Crash 还是OS Crash 或者是主机断电重启都不会丢失数据
innodb_doublewrite=ON
#以下配置保证每次事务提交后，都能实时刷新到磁盘中，尤其是确保每次事务对应的binlog都能及时刷新到磁盘中，只要有了binlog，InnoDB就有办法做数据恢复，不至于导致主从复制的数据丢失
innodb_flush_log_at_trx_commit = 1
sync_binlog = 1
mysql configuration
2. Ensure that MySQL (slave side) synchronizes with the master side during synchronization.
1. Asynchronous replication
The main library will immediately return the result to the client after executing the transaction submitted by the client, and does not care whether the slave library has received and processed it, so there will be a problem. If the master crashes, the master has already The submitted transaction may not be transmitted to the slave library. If at this time, the forced slave will be promoted to the master, which may lead to "data inconsistency". Early MySQL (before 5.5) only supported asynchronous replication.
2. Semi-synchronous replication
MySQL introduced semi-synchronous replication in 5.5. The master library needs to ensure that at least one slave library receives and writes to the relay log before responding to the transaction submitted by the client. Semi-synchronous replication uses the rpl_semi_sync_master_wait_point parameter to control the link in which the master receives the slave ack. After receiving the ack, the master returns the status to the client. There are two options for this parameter: AFTER_SYNC & AFTER_COMMIT.
rpl_semi_sync_master_wait_point=WAIT_AFTER_COMMIT
When rpl_semi_sync_master_wait_point is WAIT_AFTER_COMMIT, the call of commitTrx is after the engine layer commit, that is, while waiting for the Slave ACK, although the current client is not returned, the transaction has been committed, and other clients will read the committed transaction. If the Slave side has not read the events of the transaction, and the main library crashes at the same time, then switch to the standby library. Then the transaction read before is gone, and there is a problem of data inconsistency. If the main library can never be started, then the transaction that has actually been successfully committed in the main library cannot be found on the slave library, that is, the data is lost.
PS: As early as 11 years ago, Alibaba database innovated to solve this problem by waiting for Slave ACK before commit on the engine layer.
3. Fully synchronous replication
In response to the above problems, MySQL officially introduced Loss-less Semi-Synchronous in 5.7.2. After calling binlog sync, wait for Slave ACK before commit on the engine layer. In this way, the transaction will only be committed after confirming that the Slave has received the transaction events.
rpl_semi_sync_master_wait_point=WAIT_AFTER_SYNC
In the after_sync mode, the problem of data inconsistency caused by the after_commit mode is solved, because the main library does not commit the transaction. But there will be a problem. When the main library is binlog flushed and binlog is synchronized to the standby library, an abort occurs before binlog sync, then it is obvious that this transaction was not successfully submitted on the main library (because binlog was not synced before abort, The transaction will be rolled back after the main library is restored), but because the slave library has received these Binlogs and executed successfully, it is equivalent to extra data on the slave library, which may cause "data inconsistency".
In addition, in the MySQL semi-synchronous replication architecture, when the main database is waiting for the standby database ack, if the timeout will degenerate to asynchronous, it may also cause "data inconsistency".
3. Remarks & Solutions (The above solution ideas can satisfy 99.8% of the company’s business scenarios)
1. Through the analysis and configuration of the above two points, we found that MySQL's own Repliaction can no longer satisfy the desire of our savvy classmates (the back-end programmers will be too careful in thinking), what should we do? In order to ensure the absolute consistency of the master-slave data, let me provide two ideas below (a bit tired today, just ideas, please listen to the next time for specific solutions).
2. A data correction platform developed by Alibaba Cloud itself.
3. PXC data strong consistency solution and supports multiple masters and multiple slaves. The disadvantage is that you need to apply to the boss for machines with little difference in performance to do clusters.
Author: Lonely stick stick
link: https: //www.jianshu.com/p/328ad87bde5e
Source: Jane books
are copyrighted by the author. For commercial reprints, please contact the author for authorization, and for non-commercial reprints, please indicate the source.