[Teacher Zhao Qiang] MySQL flashback

MySQL DBA or developers sometimes mistakenly delete or update data by mistake. If it is an online environment and has a large impact, it needs to be able to roll back quickly. The traditional recovery method is to use the backup to rebuild the instance, and then use the binlog after removing the error sql to recover the data. This method is time-consuming and laborious, even requires downtime for maintenance, and is not suitable for fast rollbacks. There are also teams that use LVM snapshots to shorten the recovery time, but the disadvantage of snapshots is that they will affect the performance of mysql. MySQL flashback (flashback) uses binlog to roll back directly, which can quickly recover without downtime.

Principle of flashback

MySQL binlog records all changes in the MySQL server since binlog was enabled in the form of events, which can help reproduce all changes between them. MySQL introduces binlog for two main purposes: one is for master-slave replication; the other is that binlog needs to be re-applied after some backup and restore operations. There are three optional binlog formats, each with advantages and disadvantages:

• statement: Based on the SQL statement mode, the binlog data volume is small, but some statements and functions may cause data inconsistency or even errors during the copy process;
• row: Based on the row mode, it records the complete changes of the row. Very safe, but binlog will be much larger than the other two modes;
• mixed: mixed mode, choose statement or row mode according to the sentence;

To use binlog flashback, you need to set the binlog format to row. Use the following statement to view the current binlog mode.

show global variables like "%binlog_format%";

Flashback combat

In a real flashback scenario, the most important thing is to quickly filter out the SQL that really needs to be rolled back. We use the open source tool binlog2sql for actual combat exercises. binlog2sql is produced by the Meituan-Dianping DBA team (Shanghai), and has been quickly rolled back several times in the online environment.

① Install binlog2sql tool

首先安装Python工具管理表pip
yum -y install epel-release
yum -y install python-pip

安装binlog2sql
git clone https://github.com/danfengcao/binlog2sql.git && cd binlog2sql
pip install -r requirements.txt

② Flashback case actual combat

1. We use the previous employee table data to create a separate database

create database testflashback;
use testflashback;
source /root/tools/scott.sql

2. Misoperation, execute the following affairs.

start transaction;
delete from emp where sal>3000;
update emp set sal=6000;
delete from emp where job='CLERK';
commit;

3. View the current binlog file

show master logs;

4. The latest binlog file is mysql-binlog.000001. Our goal is to filter out the SQL that needs to be rolled back. Since the misoperation person only knows the approximate misoperation time, we first filter it based on time. Only need to parse the emp table of the testflashback library. (Note: If there are multiple SQL misoperations, the generated binlog may be distributed in multiple files, and multiple files need to be parsed)

python binlog2sql/binlog2sql.py -uroot -pWelcome_1 \
--start-file='mysql-binlog.000001' > /root/tools/raw.sql

The above statement will train all binlog logs of the emp table. If you can determine the approximate time range, you can use the parameters --start-datetime and --stop-datetime to filter. E.g:

--start-datetime='2016-12-26 11:44:00' --stop-datetime='2016-12-26 11:50:00'

The binlog processed by parsing is as follows:

5. According to the location information, we determined that the misoperation sql came from the same transaction, and the accurate location is between 14956-16791 (binlog2sql will output the same start position for the same transaction). Then filter according to the location, use the -B option to generate rollback sql, and check whether the rollback sql is correct. (Note: In real scenarios, the generated rollback SQL often needs to be further filtered. Combined with grep, editor, etc.)

python binlog2sql/binlog2sql.py -uroot -pWelcome_1 --start-file='mysql-binlog.000001' \
--start-position=14956 --stop-position=16791 -B > /root/tools/rollback.sql

The following is the generated flashback statement:

6. Confirm with the business party that there is no problem with the rollback SQL, and execute the rollback statement. Log in to mysql and confirm that the rollback is successful.

mysql -uroot -pWelcome_1 < /root/tools/rollback.sql

7. Check if the data is restored