Mysql master-slave basic principle, main form and master-slave synchronization delay principle (read and write separation) lead to the problem and solution
First, the difference between the master and slave databases
The slave database (Slave) is a backup of the master database. When the master database (Master) changes, the slave database should be updated. These database software can design the update cycle. This is a means to improve information security. The master-slave database server is not in a geographical location, and the database can be saved when an accident occurs.
(1) Master-slave division of labor
The Master is responsible for the load of write operations, which means that all write operations are performed on the Master, and the read operations are allocated to the Slave. In this way, the reading efficiency can be greatly improved. In general Internet applications, after some data surveys, it is concluded that the ratio of read / write is about 10: 1, which means that a large number of data operations are concentrated on read operations, which is why we have multiple slaves. s reason. But why separate reading and writing? Developers familiar with DB know that write operations involve locks, whether it is row locks, table locks, or block locks, which are relatively things that reduce system execution efficiency. Our separation is to concentrate the write operation on one node, while the read operation is performed on the other N nodes, which effectively improves the read efficiency from another aspect and ensures the high availability of the system.
(2) Basic process
1) Mysql master-slave synchronization means that when the master (master database) changes data, it will be synchronized to the slave (slave database) in real time.
2). Master-slave replication can horizontally expand the load capacity of the database, fault tolerance, high availability, and data backup.
3) Whether it is delete, update, insert, or create functions, stored procedures, are on the master, when the master has operations, the slave will quickly receive these operations, so as to synchronize.
(3) Uses and conditions
1), MySQL master-slave replication purposes
● Real-time disaster recovery, used for failover
● Separation of read and write, provide query services
● Backup, to avoid affecting the business
2), Master-slave deployment necessary conditions:
● Master library is open binlog log (set log-bin parameter)
● The master-slave server-id is different
● The slave server can connect to the master database
Second, the granularity, principle and form of master-slave synchronization:
(1), three main implementation granularities
Detailed master-slave synchronization mainly has three forms: statement, row, mixed
1), statement: will write the SQL statement for database operation to binlog
2), row: will each The data changes are written in binlog.
3) Mixed: statement and row are mixed. Mysql decides when to write binlog in statement format and when to write binlog in row format.
(2), the main realization principle, specific operation, schematic diagram
1) Operation on the master machine:
When the data on the master changes, the event change will be written into the bin-log in order. When the slave is linked to the master, the master machine will start the binlog dump thread for the slave. When the master's binlog changes, the bin-log dump thread will notify the slave and send the corresponding binlog content to the slave.
2). Operate on the slave machine:
When master-slave synchronization is enabled, two threads are created on the slave: I \ O threads. The thread is connected to the master machine, and the binlog dump thread on the master machine will send the contents of the binlog to the I \ O thread. After receiving the binlog content, the I / O thread writes the content to the local relay log; the sql thread. This thread reads the ralay log written by the I / O thread. And according to the relay log. And according to the content of relay log, do corresponding operation to the slave database.
3) The principle diagram of MySQL master-slave replication is as follows:
Generate two threads from the library, one I / O thread and one SQL thread; the
i / o thread requests the binlog of the main library, and writes the obtained binlog log to the relay log (relay log) file; the
main library generates A log dump thread is used to transfer the binlog to the slave library i / o thread;
SQL thread will read the log in the relay log file and parse it into specific operations to achieve consistent master-slave operations and final data consistency;
(2), master-slave form
mysql master-slave replication is flexible
● one master one slave
● master master replication
● one master multi-slave --- expand the performance of system reading, because the read is read from the library;
● multi-master one slave --- 5.7 support
● cascade replication ---
3. Problems, causes and solutions for the delay of master-slave synchronization:
(1) The delay of mysql database synchronization from the database
1) Related parameters:
First execute show slave satus on the server; you can see many synchronized parameters:
Master_Log_File: The name of the master server binary log file currently being read by the I / O thread in SLAVE Read_Master_Log_Pos: The position of the I / O thread in SLAVE that has been read in the current master server binary log Relay_Log_File: The SQL thread is currently The name of the relay log file that is read and executed Relay_Log_Pos: In the current relay log, the location where the SQL thread has read and executed Relay_Master_Log_File: The name of the master server binary log file that contains most recent events executed by the SQL thread Slave_IO_Running : Whether the I / O thread is started and successfully connected to the master server Slave_SQL_Running: Whether the SQL thread is started Seconds_Behind_Master: The time gap between the slave server SQL thread and the slave server I / O thread, in seconds.
Synchronization delay from the slave library occurs ● show slave status shows that the parameter Seconds_Behind_Master is not 0, this value may be large ● show slave status shows that the parameters Relay_Master_Log_File and Master_Log_File show that the bin-log number is very different, indicating that bin-log is in the slave the failure to make timely synchronized, so the recent execution of bin-log and the current IO thread reads bin-log vary considerably large number of mysql-relay-log log data exist from the library catalog ● mysql, it will be after the completion of the synchronization log The system automatically deletes, there are a lot of logs, indicating that the master-slave synchronization delay is very severe
(2) The delay of synchronization of the MySql database from the library
1), MySQL database master-slave synchronization delay principle mysql master-slave synchronization principle: the master library for write operations, write binlog sequentially, from the library single-threaded to the master library sequential read "write operation binlog", get the binlog from the library as it is locally Perform (random write) to ensure that the master and slave data are logically consistent. The master-slave replication of MySQL is a single-threaded operation. The main library generates binlogs for all DDL and DML. The binlogs are written sequentially, so the efficiency is very high. The Slave_IO_Running thread of the slave fetches logs to the main library. Here comes, the Slave_SQL_Running thread of the slave implements the DDL and DML operations of the main library in the slave. The IO operations of DML and DDL are random, not sequential, and the cost is much higher. It may also cause lock contention for other queries on the slave. Since Slave_SQL_Running is also single-threaded, a DDL card master needs to execute for 10 minutes. Then all subsequent DDL will wait for this DDL to finish execution before continuing, which causes a delay. A friend may ask: "The same DDL on the main library also needs to execute 10 points, why is the slave delayed?" The answer is that the master can be concurrent, but the Slave_SQL_Running thread cannot.
2) How is the master-slave synchronization delay in the MySQL database generated? When the TPS concurrency of the main library is high, the number of DDL generated exceeds the range that a slave sql thread can bear, then the delay is generated, and of course there may be a lock wait with the slave's large query statement. The main reason: the database has too much read and write pressure on the business, the CPU calculation load is large, the network card load is heavy, and the hard disk random IO is too high. Secondary reasons: performance impact caused by reading and writing binlog, network transmission delay.
(3), the delay solution of the MySql database synchronization from the library
1) 、 Architecture
1. The implementation of the business persistence layer adopts a sub-database architecture, and the mysql service can be extended in parallel to disperse pressure.
2. Separate reading and writing of a single library, one master and many slaves, master and slave reading, disperse pressure. In this way, the pressure of the secondary storage is higher than that of the primary storage, protecting the primary storage.
3. The infrastructure of the service adds memcache or redis cache layer between the business and mysql. Reduce the read pressure of mysql.
4. MySQL of different businesses is physically placed on different machines to disperse pressure.
5. Use a better hardware device than the main library as a slave summary, mysql pressure is small, the delay will naturally become smaller.
2), hardware
1. Use a good server. For example, 4u is significantly better than 2u, and 2u is better than 1u.
2. Storage uses SSD or disk array or san to improve the performance of random write.
3. The master-slave room is guaranteed to be under the same switch and in a 10 Gigabit environment.
In summary, the hardware is strong, and the delay will naturally become smaller. In short, the solution to reduce latency is to spend money and time.
3), mysql master-slave synchronization acceleration
1. Sync_binlog is set to 0 on the slave side
2. –logs-slave-updates Updates received by the slave server from the master server are not recorded in its binary log.
3. Directly disable the binlog on the slave side
4. On the slave side, if the storage engine used is innodb, innodb_flush_log_at_trx_commit = 2
4), optimization from the perspective of the file system itself
The master end modifies the etime attribute of files in the Linux and Unix file systems. Since the OS will write back the time when the read operation occurs to the disk when reading the file, this is not necessary for database files with frequent read operations. It only increases the burden on the disk system and affects I / O performance. You can organize the operating system to write atime information by setting the mount attribute of the file system. The operation on Linux is: open / etc / fstab, add the noatime parameter / dev / sdb1 / data reiserfs noatime 1 2 and then remount the file system #mount -oremount / data
5). The main library for synchronization parameter adjustment is written, which has high data security. For example, sync_binlog = 1, innodb_flush_log_at_trx_commit = 1, and other settings are needed, and slave does not need such high data security. It can be said that sync_binlog is set to 0 or close binlog, innodb_flushlog can also be set to 0 to improve the efficiency of sql execution
1. sync_binlog = 1 oMySQL provides a sync_binlog parameter to control the database binlog to flash to disk. By default, sync_binlog = 0 means that MySQL does not control the refresh of binlog, and the file system controls the refresh of its cache. The performance at this time is the best, but the risk is also the biggest. Once the system crashes, all binlog information in binlog_cache will be lost.
If sync_binlog> 0, it means that every sync_binlog transaction commits, MySQL calls the file system refresh operation to flush the cache. The safest is sync_binlog = 1, which means that MySQL will brush down the binlog every time a transaction is submitted, which is the most secure but has the highest performance loss setting. In this case, the system may lose the data of one transaction only when the operating system of the host where the database is located is damaged or suddenly loses power. Although binlog is sequential IO, but set sync_binlog = 1, multiple transactions are submitted at the same time, which also greatly affects the performance of MySQL and IO. Although it can be alleviated by the group commit patch, the impact of excessive refresh frequency on IO is also very large.
For systems with high concurrent transactions, the difference between the write performance of the system with "sync_binlog" set to 0 and 1 may be as high as 5 times or more. So the sync_binlog set by many MySQL DBAs is not the safest one, but 2 or 0. This sacrifices certain consistency to achieve higher concurrency and performance. By default, binlog is not synchronized with the hard disk every time it is written. Therefore, if the operating system or machine (not just the MySQL server) crashes, it is possible that the last statement in the binlog is lost. To prevent this, you can use the sync_binlog global variable (1 is the safest value, but also the slowest) to synchronize binlog with the hard disk after every N binlog writes. Even if sync_binlog is set to 1, there may be an inconsistency between the contents of the table and the contents of binlog when a crash occurs.
2. Innodb_flush_log_at_trx_commit (this works) complains that Innodb is 100 times slower than MyISAM? Then you probably forgot to adjust this value. The default value of 1 means that every transaction commit or out-of-transaction instruction requires the log to be flushed to the hard disk, which is time-consuming. Especially when using battery backed up cache. Set to 2 for many applications, especially the transfer from the MyISAM table, it means not written to the hard disk but written to the system cache. The logs will still be flushed to the hard disk every second, so you will not normally lose more than 1-2 seconds of updates. Set to 0 will be faster, but the security is relatively poor, even if MySQL hangs, it may lose the transaction data. The value 2 will only lose data when the entire operating system hangs.
3. The ls (1) command can be used to list atime, ctime and mtime of files.
The access time of atime file changes the create time of the ctime file when reading the file or executes the file. When the file is written, the owner, permissions or link settings are changed when the content of the inode changes. The modified time of the mtime file is changed when the file is written. When the file content changes, ls -lc filename lists the file's ctimels -lu filename lists the file's atimels -l filename lists the file's mtimestat filename lists the atime, mtime, ctimeatime are not necessarily modified after accessing the file because : When using the ext3 file system, if the noatime parameter is used during the mount, then the atime information will not be updated. These three time stamps are all placed in the inode. If mtime and atime are modified, the inode will be changed. Since the inode is changed, the ctime will also be changed. The reason why the noatime is used in the mount option is that I do n’t want the file system to do it. Too many modifications to improve reading performance
(4), MySql database from the library to synchronize other problems and solutions
1) Problems with mysql master-slave replication: ● Data may be lost after the master database is down ● There is only one sql Thread in the slave database, the master database has a large write pressure, and replication is likely to be delayed 2). Solution: ● Semi-synchronous replication --- Solve the problem of data loss ● Parallel replication ---- Solve the problem of delayed replication from the library
3), semi-synchronous replication mysql semi-sync (semi-synchronous replication) semi-synchronous replication: ● 5.5 is integrated into mysql, exists in the form of a plug-in, and needs to be installed separately ● ensure that the binlog is transferred to at least one slave library after the transaction is submitted ● there is no guarantee from The binlog after the library has applied this transaction ● The performance will be reduced to a certain extent, and the response time will be longer ● The network is abnormal or the slave library is down, the card master master library, until the timeout or recovery from the library 4), master slave replication-asynchronous replication principle , Semi-synchronous replication and parallel replication principle comparison
a. The principle of asynchronous replication:
b. The principle of semi-synchronous replication:
The transaction needs to return an accepted one from the library after writing the binlog in the main library, and then put it back to the client; 5.5 is integrated into mysql and exists in the form of a plug-in. It needs to be installed separately to ensure that the binlog is transferred to at least one slave library after the transaction is submitted. The binlog performance of the slave application to complete this transaction has a certain reduction in network abnormalities or slave downtime, card master library, until timeout or recovery from the library
c. Parallel replication mysql parallel replication ● New in community version 5.6 ● Parallel refers to multi-thread apply binlog from the library ● Binary level parallel application of binlog, the same library data changes or serial (version 5.7 parallel replication based on transaction group) setting set global slave_parallel_workers = 10; set the number of sql threads to 10
Principle: From the library multi-threaded apply binlog, a new library-level parallel application of binlog is added in the community 5.6. The same library data changes or serial 5.7 version parallel replication based on transaction group
Reference: https://blog.csdn.net/hao_yunfeng/article/details/82392261