Backing the database to a specified point in time or location is often achieved by using full backup + binlog increment.
In the case of a large amount of data, incremental recovery of binlog has always been a distressing problem.Because the recovery of binlog is very slow and error-prone.
ps All the following boxes can slide left and right
It is recommended to read horizontally
Common binlog incremental recovery methods
First parse into sql file, and then import MySQL
1mysqlbinlog mysql-bin.000001 --start-position=n > /data/add.sql
2mysqlbinlog mysql-bin.000002 ... mysql-bin.n >> /data/add.sql
3mysql -u -p -S < /data/add.sql
Directly pipe to MySQL
1mysqlbinlog mysql-bin.000001 --start-position=n | mysql -u -p -S
2mysqlbinlog mysql-bin.000002 ... mysql-bin.n | mysql -u -p -S
The method of direct pipe entry is not necessarily safe. The manual also specifies:
1If you have more than one binary log to execute on the MySQL server,
2the safe method is to process them all using a single connection to the server.
For more information about this method, you can refer to:
https://dev.mysql.com/doc/refman/5.7/en/point-in-time-recovery.html
However, the principles of these two methods are the same. They are mysqlbinlogparsed into sql and imported into MySQL.
advantage:
-
Convenient operation and simple logic.
No need to close
mysqld.
Disadvantages:
-
It is
ERRORdifficult to "recover from a breakpoint" when it is difficult to locate the position.Problems with special characters or character sets.
max_allowed_packetproblem.Recovery speed is slow.
Because relaylog and binlog are essentially the same thing,
is it possible to use MySQL's own sql_thread to increase binlog?
Recovery via sql_thread
Processing ideas:
1) Reinitialize an instance and restore the full backup file.
2) Find the firstbinlogfileposition, and all the restbinlog.
3)binlogPretend to berelaylogandsql threadrestore through incremental.
Only the core part is introduced here, that is, the process of disguising as a relaylog.
① Change the repository of relay log info to file and generate this file. ( relay_log_info_repositorWritten to the configuration file)
1SET GLOBAL relay_log_info_repository='FILE';
2CHANGE MASTER TO master_host='1', master_password='1', master_user='1', master_log_file='1', master_log_pos=4;
By change orders, to tell MySQL itself as a slave instance, because no use IO_Thread, so host, password, userand so are free to fill in.
And through this step, a relay.infofile is generated .
② Close the instance and disguise the binlog file that needs to be incremented as a relaylog.
1cp mysql-bin.000003 mysql-bin.000004 mysql-bin.000005 mysql-bin.000006 mysql-bin.000007 mysql-bin.000008 mysql-bin.000009 mysql-bin.000010 $relaylogdir
2cd $relaylogdir
3rename mysql-bin. mysql-relay. mysql-bin.0000*
4chown mysql:mysql -R .
binlogMove to it through the cp command $relaylogdir. This variable depends on the option parameters of the instance and is placed datadirunder by default .
Then binlogbatch renamed relaylogand given the corresponding permission, otherwise it will error OS error code 13: Permission denied.
③ Modify the relay.info file and relay-log.index file and
change relay.infothe second and third lines to the first binlog(now relaylog) file name and position:
1/data/mysql57/relaylog/mysql-relay.000003
21276895
The second Relay_log_nameand third lines correspond to the sum Relay_log_pos, which is equivalent to:
1mysqlbinlog mysql-relay.000003 --start-position=1276895 | mysql -u -p -S
This file is modified to tell SQL_Threadwhich one fileand which one to positionstart execution from events.
Modify again relay-log.index, clear the original information, add the following information, in order to tell SQL_Threadwhat else relaylogneeds to be executed.
1/data/mysql57/relaylog/mysql-relay.000003
2/data/mysql57/relaylog/mysql-relay.000004
3/data/mysql57/relaylog/mysql-relay.000005
4/data/mysql57/relaylog/mysql-relay.000006
5/data/mysql57/relaylog/mysql-relay.000007
6/data/mysql57/relaylog/mysql-relay.000008
7/data/mysql57/relaylog/mysql-relay.000009
8/data/mysql57/relaylog/mysql-relay.000010
④ Start the instance, open SQL_Thread:
1START SLAVE sql_thread ;
⑤ Check the copy status:
1mysql> SHOW SLAVE STATUS\G
2*************************** 1. row ***************************
3Slave_IO_State:
4Master_Host: 1
5Master_User: 1
6Master_Port: 3306
7Connect_Retry: 60
8Master_Log_File: 1
9Read_Master_Log_Pos: 4
10Relay_Log_File: mysql-relay.000003 -- 已经执行到的日志名
11Relay_Log_Pos: 11529982 -- 已经执行到日志的位置
12Relay_Master_Log_File: 1
13Slave_IO_Running: No
14Slave_SQL_Running: Yes
15Replicate_Do_DB:
16Replicate_Ignore_DB:
17Replicate_Do_Table:
18Replicate_Ignore_Table:
19Replicate_Wild_Do_Table:
20Replicate_Wild_Ignore_Table:
21Last_Errno: 0
22Last_Error:
23Skip_Counter: 0
24Exec_Master_Log_Pos: 11529982
25Relay_Log_Space: 5347038913
26Until_Condition: None
27Until_Log_File:
28Until_Log_Pos: 0
29Master_SSL_Allowed: No
30Master_SSL_CA_File:
31Master_SSL_CA_Path:
32Master_SSL_Cert:
33Master_SSL_Cipher:
34Master_SSL_Key:
35Seconds_Behind_Master: 274354 -- 若变为0，则表示已经增量完毕
36Master_SSL_Verify_Server_Cert: No
37Last_IO_Errno: 0
38Last_IO_Error:
39Last_SQL_Errno: 0
40Last_SQL_Error:
41Replicate_Ignore_Server_Ids:
42Master_Server_Id: 0
43Master_UUID:
44Master_Info_File: /data/mysql57/master.info
45SQL_Delay: 0
46SQL_Remaining_Delay: NULL
47Slave_SQL_Running_State: Reading event from the relay log
48Master_Retry_Count: 86400
49………………………………
At this point, it can be sql_threadrestored incrementally binlog.
Of course, the above process is only for the specified --start-positionmethod of recovery. For example, if a single-point MySQL instance innodb_force_recovery=6cannot be started, it needs to be restored through the last available full backup + the remaining binlog.
The version used for this test is:MySQL 5.7.16
Effect :
Quickly restore to the specified point, that is, through the complete file + binlogrestore to the last one before the failure position.
For --stop-position
For example, if the wrong SQL is executed at a certain moment, such as truncate and other operations, this method can also be used.
But it --start-positionis slightly different from the specified method:
just START SLAVE sql_threadadd one after it UNTIL RELAY_LOG_FILE = 'log_name', RELAY_LOG_POS = log_pos.
This option is used to control SQL_Threadthe final position of execution, similar to mysqlbinlog mysql-bin.n --stop-position=$log_pos.
Of course, this kind of data back-off operation can also consider flashbacktools with functions.
Performance comparison
For the same set of binlog file increment: the time
through mysqlbinlogparsing + import is 69min.
And SQL_Threadthe execution time passed is 41min.
And the larger the binlog file to be incremented, the more obvious the effect.
to sum up
Advantages :
1) It can be restored by breakpoints, and the progress can be controlled artificially. For example, when you stop slave or encounter an error, you can know where the error is.
2) The performance is relatively good. In the case of a large number of binlogs, the recovery speed can be accelerated.
3) In some versions, MTS may be used to speed up the incremental speed and make the recovery faster.Disadvantages :
1) Need to close mysqld.
2) The manual execution process is more complicated than the mysqlbinlog method.
mysqlbinlog --start-positionIt is relay.infoequivalent to the third line modified : the
purpose is to specify the first one to start execution position.
mysqlbinlog --stop-positionIt SQL_Threadis UNTIL RELAY_LOG_FILE = 'log_name', RELAY_LOG_POS = log_posequivalent to specifying at startup : the
purpose is to specify the last one to end execution position.
The full text is over.
Enjoy MySQL :)
Scan the QR code to follow the author's WeChat public account
Teacher Ye's "MySQL Core Optimization" class has been upgraded to MySQL 8.0, scan the code to start the journey of MySQL 8.0 practice