Chapter VII backup and recovery and migration

Operation and maintenance of backup and recovery related duties

• Design a backup strategy
  • Backup cycle
  • Backup
  • Back Up
• Check Backup
  • Backup file size
  • Back Up
• Regular exercise recovery
• Recovery
• Upgrades, migrations

Backup Type

Hot Standby (hot backup)

Backup does not lock table, the table only for transactional engines (for example: InnoDB).
Impact on the smallest businesses

Preparation of cold (cold backup)

Business stop or shut down the database, back up
the greatest impact on business

Preparation temperature (warm backup)

Lock table backup, read-only backup. Block all operational changes
affect the write operation (query only)

Backup Strategy

All equipment

Full database backup.

Incremental

Data Backup changes

Logical backup

The data raised saved as a text file from the library

mysqldump + binlog

Physical backup

Direct copy backup (archiving) data files

XKB_full + xbk_inc + binlog
或者
xtrabackup_full + binlog 

Backup cycle

The amount of data backup cycle design
example: Sun all equipment, week - Zhou incremental backup 6

Backup Tools

mysqldump

Logical backup
advantages:

1. Text format (compression ratio)
2. SQL statements are backed up
3. Highly readable, ease of secondary processing.

Disadvantages:

1. Backup and recovery process is slow
2. When restored, if the target library does not exist, you need to be created manually in advance

Xtrabackup (Percona)

Physical backup
advantages:

1. Backup speed
2. Supports hot standby
3. Incremental backup feature comes

Disadvantages:

1. Backup is a data file
2. Readability is poor, not easy to secondary treatment

Select recommended

数据小于100G             MDP、XBK
数据在100G - 1T        XBK
数据超过TB级别        XBK、MDP

mysqldum logical backup tool

General parameters client

-p  # 密码
-S  # sock文件
-h  # 连接地址
-P  #端口
-u  #用户名

Backup specific basic parameters

Full database backup-A

mysqldump -uroot -p -A >/backup/full.sql

Single library or database backup-B

##  只备份world和test两个库
mysqldump -uroot -p123 -B world test >/backup/db.sql

Single table or table backup

## 备份单张表
mysqldump -uroot -p123 world city >/backup/city.sql

## 备份多张表  wolrd库下:city表 和 country表
mysqldump -uroot -p123 world city country >/backup/tab.sql

Special function parameters

1) Special parameters

-R           ## 存储过程及函数
--triggers　  ## 触发器
-E　         ## 事件

例:
mysqldump -uroot -p -A -R -triggers -E >/backup/full.sql

2）--master-data=2

说明：以注释形势保存备份开始时间点的binlog 状态信息

功能：
    1. 在备份是，自动记录 二进制日志文件名和位置号
    2. 自动锁表
    3. 配合--single-transaction，只对非InnoDB表进行锁表备份，InnoDB表进行热备，实际上是实现快照备份。

--master-data=[0|1|2]
    0 默认值
    1 以change master to命令形式，可以用作主从复制
    2 以注释的形式记录，备份时刻的文件名+postion号

3） --single-transaction

功能：能够对InnoDB存储引擎实现热备。
master-data
1）不加 --single-transaction 启动所有表的温备份，所有表锁定
2）加上 --single-transaction 对innodb进行快照备份,对非innodb表可以实现自动锁表功能

例:
mysqldump -uroot -p -A -R -E --triggers --master-data=2 --single-transaction --set-gtid-purged=OFF >/data/backup/full.sql

4) - set-gtid-purged = OFF (GTID pattern unique parameter)

作用,去除gtid所有信息,在日常备份恢复时可加.
做主从复制应用的时候,不能加此参数.

5）--max-allowed-packet=512M

mysql服务器端和客户端在一次传送数据包的过程中最大允许的数据包大小

A complete backup statement

 mysqldump -uroot -p123 -A --master-data=2 --single-transaction -R -E --triggers --set-gtid-purged=OFF --max-allowed-packet=256M > /backup/full.sql

Enterprise Recovery Case (MDP)

Context:

Small sites running system, mysql-5.7.20 database, the amount of data 50G.
23:00 pm every day, scheduled tasks call mysqldump to perform a full backup script

mysqldump -uroot -p123456 -A -E -R --triggers --master-data=2 --single-transaction --set-gtid-purged=OFF --max-allowed-packet=256M |gzip > /backup/full_$(date +%F).sql.gz

Failure time

The end of failure Walkthrough: Analog Wednesday 10:00 accidentally deleted the database, and recovery.

Recovery ideas

1) stopping service failure, maintenance pages linked to
2) preparing test library, library full recovery
3) Data taken from the start until all equipment binlog Fault Time
4) Data availability and integrity test
5) fault data export, import to production
6) to withdraw the maintenance page, start a business

Failure drill

Analog fault site

(1) Analog data
mysql mysql[(none)]>create database mdp charset utf8mb4; mysql[(none)]>use mdp; mysql[mdp]>create table t1(id int) engine=innodb charset=utf8mb4; mysql[mdp]>insert into t1 values(11),(22),(33); mysql[mdp]>commit; mysql[mdp]>select * from t1;; +------+ | id | +------+ | 11 | | 22 | | 33 | +------+ 3 rows in set (0.00 sec)

(2) Preparation of whole analog 23:00

mysqldump -uroot -p123456 -A -E -R --triggers --master-data=2 --single-transaction --set-gtid-purged=OFF --max-allowed-packet=256M |gzip >/backup/full_$(date +%F).sql.gz 

ls -l /backup/

-rw-r--r-- 1 root root 794255 Aug 17 21:51 full_2019-08-17.sql.gz

(3) Analog backup data after change

mysql[(none)]>use mdp;
mysql[mdp]>insert into t1 values(111),(222),(333);
mysql[mdp]>commit;
mysql[mdp]>select * from t1;
+------+
| id   |
+------+
|   11 |
|   22 |
|   33 |
|  111 |
|  222 |
|  333 |
+------+
6 rows in set (0.00 sec)

(4) analog data corruption

Entire libraries from accidental deletion

mysql[mdp]>drop database mdp;

Failure to prepare recovery

(1) failure to stop business, hang Maintenance Page
(2) Prepare to back up

Full library preparation:

Preparation full backup data is compressed, the first backup data decompression where

cd /backup
gunzip full_2019-08-17.sql.gz

binlog preparation:

Gets to binlog of pos pos starting position and end position to export data.

# 获取使用binlog日志和pos起始号 数据在22行
sed  -n 22p /backup/full_2019-08-17.sql 

-- CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=1096;

# 获取到pos 结束号（找到drop数据命令那行的pos列）
mysql -uroot -p123456 -e 'show binlog events in "mysql-bin.000001" limit 7,10';

# 此时已经获取到了 使用的binlog文件和 pos的起始号和结束号
# 截取binlog记录的数据导出
mysqlbinlog  --skip-gtids  --start-position=1096 --stop-position=1357 /data/binlog/mysql-bin.000001 >/backup/binlog.sql

Data recovery

# 临时关闭binlog日志
mysql[(none)]>set sql_log_bin=0;
# 全备数据恢复
mysql[(none)]>source /backup/full_2019-08-17.sql
mysql[mdp]>select * from mdp.t1;
+------+
| id   |
+------+
|   11 |
|   22 |
|   33 |
+------+
# binlog 数据恢复
mysql[mdp]>source /backup/binlog.sql
mysql[mdp]>select * from mdp.t1;
+------+
| id   |
+------+
|   11 |
|   22 |
|   33 |
|  111 |
|  222 |
|  333 |
+------+

XBK (percona-Xtrabackup) - physical backup tool

XBK installation

Installation dependencies

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum -y install perl perl-devel libaio libaio-devel perl-Time-HiRes perl-DBD-MySQL libev

Download and install software

https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-2.4.4/binary/redhat/6/x86_64/percona-xtrabackup-24-2.4.4-1.el6.x86_64.rpm

wget https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-2.4.12/binary/redhat/7/x86_64/percona-xtrabackup-24-2.4.12-1.el7.x86_64.rpm
yum -y install percona-xtrabackup-24-2.4.4-1.el7.x86_64.rpm

Introduction

Physical backup tools, data similar to CP

Backup details

非InnoDB：
​   自动锁表备份，会有短暂的全局锁（例如MyISAM）
InnoDB：
​    立即进行CKPT，将当前所有已提交事务的脏页，立即刷写到磁盘
​    拷贝所有InnoDB的数据文件，系统数据文件也进行拷贝
​    将备份过程中产生的reodo截取并备份走

innobackupex backup application

Profile Settings

[client]
socket=/tmp/mysql.sock

Perform a full backup command

 innobackupex --user=root --password=123  --no-timestamp  /backup/xbk/full

Analog damage

 pkill mysqld
 \rm -rf  /data/3306/data/*

Data recovery preparation (backup process)

innobackupex --apply-log /backup/xbk/full/

Data recovery

--copy-back (依赖于my.cnf [mysqld]的配置)

innobackupex --copy-back   /backup/xbk/full/ 
chown -R mysql.mysql /data/*

--apply-log Parameter Description (Interview emphasis)

ACSR mimics the process of InnoDB engine of
the backup set of data and log LSN tied the
use of roll-front redo carried out
using the undo rollback

Backup directory generated file information notes

xtrabackup_binlog_info          -- 存储的是binlog截取的起始点信息(position,gtid)
xtrabackup_checkpoints         -- 备份的乐行和lsn相关信息
xtrabackup_info               -- 详细备份信息 

xtrabackup_checkpoints file content

# 备份类型
backup_type = full-prepared

# 整个备份包含的最起始的LSN
from_lsn = 0

# CKPT后数据页面(ibd)的LSN
to_lsn = 244445125

# 备份结束时，redo的LSN,（5.7版本默认产生9个LNS）
last_lsn = 244445134
compact = 0
recover_binlog_info = 0

XBK incremental backups (increamental)

Incremental backup Introduction

Features: full backup and incremental backup based on last backup
advantages: saving disk space and time
limitations: dependent on the full backup and the last backup, XKB tools need to be incorporated into the full incremental backup in order to recover.

XBK increment (incremental) backup simulation

Note:

Xtrabackup enterprise-level incremental restore real
background:
a large site, mysql database, the amount of data 500G, updated daily volume 20M-30M
backup strategy:
xtrabackup, 23:00 full backup every Sunday, Monday to Saturday 23:00 conducted incremental backups.
Failure scenarios:
Wednesday 14:00 occur accidentally deleted the database table operations.
How to recover?

Analog data

mysql[(none)]>create database test charset utf8mb4;
mysql[(none)]>use test;
mysql[test]>create table t1(id int) engine=innodb charset=utf8mb4;
mysql[test]>insert into t1 values(1),(2),(3);
mysql[test]>commit;

Analog bountiful Sunday

innobackupex --user=root --password=123456 --no-timestamp /backup/xbk/full

Monday's simulation data changes

mysql[(none)]>use test;
mysql[test]>insert into t1 values(11),(22),(33);
mysql[test]>commit;

Analog incremental backups Monday night

innobackupex --user=root --password=123456 --no-timestamp --incremental --incremental-basedir=/backup/xbk/full /backup/xbk/inc1

--incremental                          打开增量备份开关
--incremental-basedir=/backup/xbk/full 设定增量备份的基备份(一般是上一天)

Analog data changes during the day Tuesday

mysql[(none)]>use test;
mysql[test]>insert into t1 values(111),(222),(333);
mysql[test]>commit;

Analog incremental backups on Tuesday night

innobackupex --user=root --password=123456 --no-timestamp --incremental --incremental-basedir=/backup/xbk/inc1 /backup/xbk/inc2

Analog data changes during the day Wednesday

mysql[(none)]>use test;
mysql[test]>insert into t1 values(1111),(2222),(3333);
mysql[test]>commit;

Analog data corruption

pkill mysqld 
rm -rf /data/3306/data/*

Recovery ideas

1) test libraries, Maintenance Page
2) backup process: merging, not quasi-
3) intercept binary log
4) Data Recovery

Began to recover drill - handle the backup

# 处理原始全备
innobackupex --apply-log --redo-only /backup/xbk/full/

# 合并周一数据并处理
innobackupex --apply-log  --incremental-dir=/backup/xbk/inc1 /backup/xbk/full

# 合并周二数据并处理
innobackupex --apply-log  --incremental-dir=/backup/xbk/inc2 /backup/xbk/full

# 处理合并后全备数据
innobackupex --apply-log /backup/xbk/full/

Restore the backup to start the database

# 将备份数据拷贝到数据目录
cp -a /backup/xbk/full/* /data/3306/data/ 

# 修改权限
chown -R mysql.mysql  /data/3306/data/

# 启动数据库
/etc/init.d/mysqld start

Interception binlog

# 查看使用binlog日志和 pos起始位置
cat /backup/xbk/inc2/xtrabackup_binlog_info 
    mysql-bin.000004    1339

# 获取pos结束位置(drop)
mysql[test]>show binlog events in 'mysql-bin.000004';

# 导出binlog数据
mysqlbinlog --skip-gtids --start-position=1339 --stop-position=1600 /data/binlog/mysql-bin.000004 >/backup/bin.sql

binlog data recovery

mysql[test]>set sql_log_bin=0;
mysql[test]>source /backup/bin.sql
mysql[test]>select * from t1;
+------+
| id   |
+------+
|    1 |
|    2 |
|    3 |
|   11 |
|   22 |
|   33 |
|  111 |
|  222 |
|  333 |
| 1111 |
| 2222 |
| 3333 |

Efficiency of the recovered data (small extension)

The entire amount of data larger libraries, but rarely damaged data **
eg: 500G amount of data, the data is corrupted 10M

XBK: table space migration

MDP:
manual analysis
1, the table structure

sed -e'/./{H;$!d;}' -e 'x;/CREATE TABLE city/!d;q'  /tmp/full.sql>createtable.sql

2, get INSERT INTO statement used to recover data

# grep -i 'INSERT INTO `city`'  /tmp/full.sqll >data.sql

3. Get back up a single repository

 sed -n '/^-- Current Database: `world`/,/^-- Current Database: `/p' all.sql >world.sql

Small expansion: Flashback table data (binlog flashback)

Data corruption:
physical: disk, raid, FS, ibd
logic: drop, alter, delete, update

Description: The recording binlog row automatically log inverse conversion operation, to realize the function of flashback.

mariadb default support
binlog2sql

Small expansion: migration

The Oracle, SQL Server -----> MySQL
the Oracle ---- OGG ---> MySQL
MySQL version low --XBK, MDP -> MySQL version of the high
cloud on migration, DTS