MySQL backup data method

1 What to back up?

Under normal circumstances, the data we need to back up is divided into the following types

• data

• Binary log, InnoDB transaction log

• Code (stored procedures, stored functions, triggers, event schedulers)

• Server configuration file

2 General way of backing up data

• Hot backup

• Warm backup

• Cold backup

  Hot backup refers to when the database is backed up,  the read and write operations of the database are not affected. 
  Warm backup refers to when the database is backed up,  the read operation of the database can be performed, but the write operation cannot be performed. 
  Cold backup refers to the time when the database is backed up. When performing a backup, the  database cannot be read or written, that is, the database must be offline

  MySQLDifferent ways of backing up in the process also need to consider whether the storage engine supports

• MyISAM 

   Hot spare×

   Warm preparation√

   Cold standby √

• InnoDB

   Hot standby√

   Warm preparation√

   Cold standby √

  When the data is backed up, after the running state of the MySQLdatabase, you also need to consider the way of backing up the data in the database.

  Physical backup is generally through tar, cpwaiting for commands to directly package and copy the data files of the database to achieve the effect of 
  backup.Logical backup is generally to export data from the database through a specific tool and save it as a backup (logical backup will lose data accuracy)

  • Physical backup

  • Logical backup

3 Scenario backup strategy

1 Directly copy database files with cp, tar: If the amount of data is small, copy the database files directly

FLUSH TABLES WITH READ LOCK;   
 #1 选择数据库，向所有表施加读锁

 mkdir /backup                
 #2  创建文件夹存放备份数据库文件

 cp -a /var/lib/mysql/* /backup 
 # 3  保留权限的拷贝源数据文件

 ls /backup   
 #4  查看目录下的文件

employees  ibdata1  ib_logfile0  ib_logfile1  mysql  mysql.sock  test（对应文件）


模拟数据丢失宕机

rm -rf /var/lib/mysql/*    
#删除数据库的所有文件
service mysqld restart  
#重启MySQL, 如果是编译安装的应该不能启动, 如果rpm安装则会重新初始化数据库


rm -rf /var/lib/mysql/*    #这一步可以不做
cp -a /backup/* /var/lib/mysql/    #将备份的数据文件拷贝回去
service mysqld restart  #重启MySQL


#重新连接数据并查看
mysql> SHOW DATABASES;    #数据库已恢复

2 mysqldump+copy BIN LOGS: The amount of data is average, first use mysqldump to fully backup the database, and then back up BINARY LOG regularly to achieve the effect of incremental backup

 

mysqldumpIt is a client-side logical backup tool. It can generate a SQL statement that reproduces the original database and table creation. It can support all storage engines. For InnoDB, it supports hot backup   official documentation  ( https://dev.mysql.com/doc/ refman/5.7/en/mysqldump.html )

 mysql -uroot -p -e 'SHOW MASTER STATUS'  
#查看当前二进制文件的状态, 并记录下position的数字
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 |      106 |              |                  |
+------------------+----------+--------------+------------------+

 1 mysqldump --all-databases --lock-all-tables  > backup.sql   
#备份数据库到backup.sql文件中

mysql> CREATE DATABASE TEST1;   #创建一个数据库

mysql> SHOW MASTER STATUS;   #记下现在的position
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 |      191 |              |                  |
+------------------+----------+--------------+------------------+


2  cp /var/lib/mysql/mysql-bin.000003 /root  #备份二进制文件
   service mysqld stop   #停止MySQL
   rm -rf /var/lib/mysql/*   #删除所有的数据文件
   service mysqld start    
  #启动MySQL, 如果是编译安装的应该不能启动(需重新初始化), 如果rpm安装则会重新初始化数据库


3  SET sql_log_bin=OFF;   #暂时先将二进制日志关闭  


4  source backup.sql  #恢复数据，所需时间根据数据库时间大小而定

5  SET sql_log_bin=ON; 开启二进制日志

6  SHOW DATABASES;   #数据库恢复, 但是缺少TEST1


7  mysqlbinlog --start-position=106 --stop-position=191 mysql-bin.000003 | mysql 
   employees 
   #通过二进制日志增量恢复数据

8  SHOW DATABASES;    #查看数据是否还原！


基本语法格式

shell> mysqldump [options] db_name [tbl_name ...]    恢复需要手动CRATE DATABASES
shell> mysqldump [options] --databases db_name ...   恢复不需要手动创建数据库
shell> mysqldump [options] --all-databases           恢复不需要手动创建数据库


其他选项:
     -E, --events: 备份事件调度器
     -R, --routines: 备份存储过程和存储函数
     --triggers: 备份表的触发器; --skip-triggers 
     --master-date[=value]  
         1: 记录为CHANGE MASTER TO 语句、语句不被注释
         2: 记录为注释的CHANGE MASTER TO语句
         基于二进制还原只能全库还原

     --flush-logs: 日志滚动
         锁定表完成后执行日志滚动

 3     lvm2 snapshot + copy BIN LOGS: the amount of data is average, but does not excessively affect business operations, lvm2the snapshot used to back up the data files, and then regularly back up the BINARY LOG to achieve the effect of incremental backup

LVMSnapshot is simply to save the metadata of all files in the source partition of the snapshot at a point in time. If the source file has not changed, then the corresponding file accessing the snapshot volume directly points to the source file of the source partition. If the source file changes, then The corresponding files in the snapshot volume will not change. Snapshot volumes are mainly used to assist in backing up files. Click to view detailed introduction

添加硬盘; 这里我们直接实现SCSI硬盘的热插拔, 首先在虚拟机中添加一块硬盘, 不重启

 ls /dev/sd*   #只有以下几块硬盘, 但是我们不重启可以让系统识别新添加的硬盘
/dev/sda  /dev/sda1  /dev/sda2

echo '- - -' > /sys/class/scsi_host/host0/scan 
 echo '- - -' > /sys/class/scsi_host/host1/scan 
echo '- - -' > /sys/class/scsi_host/host2/scan 

# ls /dev/sd*    #看！sdb识别出来了
/dev/sda  /dev/sda1  /dev/sda2  /dev/sdb


# fdisk /dev/sdb   #分区
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xd353d192.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-2610, default 1): 
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-2610, default 2610): +15G

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 8e
Changed system type of partition 1 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!



##创建逻辑卷
[root@node1 ~]# pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created
[root@node1 ~]# vgcreate myvg /dev/sdb1 
  Volume group "myvg" successfully created
[root@node1 ~]# lvcreate -n mydata -L 5G myvg 
  Logical volume "mydata" created.

[root@node1 ~]# mkfs.ext4 /dev/mapper/myvg-mydata   #格式化
[root@node1 ~]# mkdir /lvm_data
[root@node1 ~]# mount /dev/mapper/myvg-mydata /lvm_data  #挂载到/lvm_data


[root@node1 ~]# vim /etc/my.cnf    #修改mysql配置文件的datadir如下

datadir=/lvm_data

[root@node1 ~]# service mysqld restart  #重启MySQL

####重新导入employees数据库########略过####



mysql> FLUSH TABLES WITH READ LOCK;     #锁定所有表

[root@node1 lvm_data]# lvcreate -L 1G -n mydata-snap -p r -s /dev/mapper/myvg-mydata   #


创建快照卷
  Logical volume "mydata-snap" created.

mysql> UNLOCK TABLES;  #解锁所有表

 mkdir /lvm_snap  #创建文件夹
 mount /dev/myvg/mydata-snap /lvm_snap/  #挂载snap
mount: block device /dev/mapper/myvg-mydata--snap is write-protected, mounting read-only

 lvm_data]# cd /lvm_snap/     ---- ls

 tar cf /tmp/mysqlback.tar *  #打包文件到/tmp/mysqlback.tar

[root@node1 ~]# umount /lvm_snap/  #卸载snap
[root@node1 ~]# lvremove myvg mydata-snap  #删除snap


恢复数据
 rm -rf /lvm_data/*
 service mysqld start    #启动MySQL, 如果是编译安装的应该不能启动(需重新初始化), 如果rpm安装则会重新初始化数据库


[root@node1 ~]# cd /lvm_data/
[root@node1 lvm_data]# rm -rf * #删除所有文件
[root@node1 lvm_data]# tar xf /tmp/mysqlback.tar     #解压备份数据库到此文件夹 
[root@node1 lvm_data]# ls  #查看当前的文件
employees  ibdata1  ib_logfile0  ib_logfile1  mysql  mysql-bin.000001  mysql-bin.000002  mysql-bin.000003  mysql-bin.index  test

4 xtrabackup: The amount of data is very large, but does not affect the business operation too much. After using it xtrabackupfor full backup, use it regularly xtrabackupfor incremental backup or differential backup

In order to better demonstrate, mariadb-5.5the version we are using this time , using xtrabackupInnoDB can exert its maximum effect, and each table of InnoDB must use a separate table space, we need to add it in the configuration file  innodb_file_per_table = ON to enable

我们这里通过wget percona官方的rpm包进行安装
 wget https://www.percona.com/downloads/XtraBackup/Percona-XtraBackup-2.3.4/binary/redhat/6/x86_64/percona-xtrabackup-2.3.4-1.el6.x86_64.rpm   
[root@node1 ~]# yum localinstall percona-xtrabackup-2.3.4-1.el6.x86_64.rpm   #需要EPEL源

XtrabackupIt is perconaa mysqldatabase backup tool provided by it . According to the official introduction, this is also the only open source tool in the world that can perform hot backup of innodb and xtradb databases. Features:

1. The backup process is fast and reliable;

2. The backup process will not interrupt the ongoing transaction;

3. Can save disk space and traffic based on compression and other functions;

4. Automatically realize backup inspection;

5. Fast recovery speed;

xtrabackupThe front-end configuration tool we use here innobackupexto achieve a complete backup of the database

When using innobackupexbackup, it will call to xtrabackupback up all InnoDB tables, copy all related files (.frm) about table structure definition, and related files of MyISAM , MERGE , CSV and ARCHIVE tables, and also backup trigger and database configuration file information. Related files, these files will be saved to a directory named after time.

reference

https://www.cnblogs.com/gered/p/10410978.html 

https://www.cnblogs.com/fengzhongzhuzu/p/9101782.html  

[root@node1 ~]# mkdir /extrabackup  #创建备份目录
[root@node1 ~]# innobackupex --user=root /extrabackup/ #备份数据

[root@node1 ~]# ls /extrabackup/  #看到备份目录
2016-04-27_07-30-48 


#完整备份
[root@node1 ~]# innobackupex --apply-log /extrabackup/2016-04-27_07-30-48/ 
 #指定备份文件的目录

#一般情况下下面三行结尾代表成功*****************
InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number 369661462
160427 07:40:11 completed OK!

 cd /extrabackup/2016-04-27_07-30-48/
[root@node1 2016-04-27_07-30-48]# ls -hl  #查看备份文件
total 31M
-rw-r----- 1 root root  386 Apr 27 07:30 backup-my.cnf
drwx------ 2 root root 4.0K Apr 27 07:30 employees
-rw-r----- 1 root root  18M Apr 27 07:40 ibdata1
-rw-r--r-- 1 root root 5.0M Apr 27 07:40 ib_logfile0
-rw-r--r-- 1 root root 5.0M Apr 27 07:40 ib_logfile1
drwx------ 2 root root 4.0K Apr 27 07:30 mysql
drwx------ 2 root root 4.0K Apr 27 07:30 performance_schema
drwx------ 2 root root 4.0K Apr 27 07:30 test
-rw-r----- 1 root root   27 Apr 27 07:30 xtrabackup_binlog_info
-rw-r--r-- 1 root root   29 Apr 27 07:40 xtrabackup_binlog_pos_innodb
-rw-r----- 1 root root  117 Apr 27 07:40 xtrabackup_checkpoints
-rw-r----- 1 root root  470 Apr 27 07:30 xtrabackup_info
-rw-r----- 1 root root 2.0M Apr 27 07:40 xtrabackup_logfile



[root@node1 ~]# rm -rf /data/*   #删除数据文件

***不用启动数据库也可以还原*************

[root@node1 ~]# innobackupex --copy-back /extrabackup/2016-04-27_07-30-48/   
#恢复数据, 记清使用方法

#########我们这里是编译安装的mariadb所以需要做一些操作##########
[root@node1 data]# killall mysqld

[root@node1 ~]# chown -R mysql:mysql ./* 
[root@node1 ~]# ll /data/      #数据恢复
total 28704
-rw-rw---- 1 mysql mysql    16384 Apr 27 07:43 aria_log.00000001
-rw-rw---- 1 mysql mysql       52 Apr 27 07:43 aria_log_control
-rw-rw---- 1 mysql mysql 18874368 Apr 27 07:43 ibdata1
-rw-rw---- 1 mysql mysql  5242880 Apr 27 07:43 ib_logfile0
-rw-rw---- 1 mysql mysql  5242880 Apr 27 07:43 ib_logfile1
-rw-rw---- 1 mysql mysql      264 Apr 27 07:43 mysql-bin.000001
-rw-rw---- 1 mysql mysql       19 Apr 27 07:43 mysql-bin.index
-rw-r----- 1 mysql mysql     2166 Apr 27 07:43 node1.anyisalin.com.err


重启数据库查看数据


增量备份
#########创建连两个数据库以供测试#####################
MariaDB [(none)]> CREATE DATABASE TEST1;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> CREATE DATABASE TEST2;
Query OK, 1 row affected (0.00 sec)

[root@node1 ~]# innobackupex --incremental /extrabackup/ --incremental-basedir=/extrabackup/2016-04-27_07-30-48/ 

[root@node1 ~]# ls /extrabackup/2016-04-27_07-57-22/ #查看备份文件
total 96
-rw-r----- 1 root root   386 Apr 27 07:57 backup-my.cnf
drwx------ 2 root root  4096 Apr 27 07:57 employees
-rw-r----- 1 root root 49152 Apr 27 07:57 ibdata1.delta
-rw-r----- 1 root root    44 Apr 27 07:57 ibdata1.meta
drwx------ 2 root root  4096 Apr 27 07:57 mysql
drwx------ 2 root root  4096 Apr 27 07:57 performance_schema
drwx------ 2 root root  4096 Apr 27 07:57 test
drwx------ 2 root root  4096 Apr 27 07:57 TEST1
drwx------ 2 root root  4096 Apr 27 07:57 TEST2
-rw-r----- 1 root root    21 Apr 27 07:57 xtrabackup_binlog_info
-rw-r----- 1 root root   123 Apr 27 07:57 xtrabackup_checkpoints
-rw-r----- 1 root root   530 Apr 27 07:57 xtrabackup_info
-rw-r----- 1 root root  2560 Apr 27 07:57 xtrabackup_logfile


BASEDIR指的是完全备份所在的目录，此命令执行结束后，innobackupex命令会在/extrabackup目录中创建一个新的以时间命名的目录以存放所有的增量备份数据。另外，在执行过增量备份之后再一次进行增量备份时，其--incremental-basedir应该指向上一次的增量备份所在的目录。

需要注意的是，增量备份仅能应用于InnoDB或XtraDB表，对于MyISAM表而言，执行增量备份时其实进行的是完全备份。

[root@node1 ~]# innobackupex --apply-log --redo-only /extrabackup/2016-04-27_07-30-48/
[root@node1 ~]# innobackupex --apply-log --redo-only /extrabackup/2016-04-27_07-30-48/ --incremental-dir=/extrabackup/2016-04-27_07-5
7-22/

恢复数据

[root@node1 ~]# rm -rf /data/*   #删除数据

[root@node1 ~]# innobackupex --copy-back /extrabackup/2016-04-27_07-30-48/     #整理增量备份之后可以直接通过全量备份还原

[root@node1 ~]# chown -R mysql.mysql /data/
[root@node1 ~]# ls /data/ -l
total 28732
-rw-rw---- 1 mysql mysql     8192 Apr 27 08:05 aria_log.00000001
-rw-rw---- 1 mysql mysql       52 Apr 27 08:05 aria_log_control
drwx------ 2 mysql mysql     4096 Apr 27 08:05 employees
-rw-r----- 1 mysql mysql 18874368 Apr 27 08:05 ibdata1
-rw-r----- 1 mysql mysql  5242880 Apr 27 08:05 ib_logfile0
-rw-r----- 1 mysql mysql  5242880 Apr 27 08:05 ib_logfile1
drwx------ 2 mysql mysql     4096 Apr 27 08:05 mysql
-rw-rw---- 1 mysql mysql      245 Apr 27 08:05 mysql-bin.000001
-rw-rw---- 1 mysql mysql       19 Apr 27 08:05 mysql-bin.index
-rw-r----- 1 mysql mysql     1812 Apr 27 08:05 node1.anyisalin.com.err
-rw-rw---- 1 mysql mysql        5 Apr 27 08:05 node1.anyisalin.com.pid
drwx------ 2 mysql mysql     4096 Apr 27 08:05 performance_schema
drwx------ 2 mysql mysql     4096 Apr 27 08:05 test
drwx------ 2 mysql mysql     4096 Apr 27 08:05 TEST1
drwx------ 2 mysql mysql     4096 Apr 27 08:05 TEST2
-rw-r----- 1 mysql mysql       29 Apr 27 08:05 xtrabackup_binlog_pos_innodb
-rw-r----- 1 mysql mysql      530 Apr 27 08:05 xtrabackup_info



#关于xtrabackup还有很多强大的功能没有叙述、有兴趣可以去看官方文档