| Backup and restoration of large amounts of data is always a difficult point. When MYSQL exceeds 10G, it will be slower to use mysqldump to export. Here is a powerful open source tool Xtrabackup. |
Introduction to Xtrabackup
Xtrabackup is a mysql database backup tool provided by percona. According to the official introduction, this is the only open source tool in the world that can perform hot backup for innodb and xtradb databases. It is a good substitute for the commercial backup tool InnoDB Hotbackup. Features:
(1) The backup process is fast and reliable; (2) The backup process will not interrupt the ongoing transaction; (3) It can save disk space and traffic based on functions such as compression; (4) Automatically realize backup inspection; (5) Fast recovery speed;
Xtrabackup installation
The latest version of its software is available at http://www.percona.com/software/percona-xtrabackup/. This article is based on the CentOS 6.6 system, so just download the corresponding version of the rpm package and install it.
[root@localhost xtrabackup]# yum -y install perl perl-devel libaio libaio-devel perl-Time-HiRes perl-DBD-MySQL //Install dependency packages
[root@localhost xtrabackup]# rpm -ivh percona-xtrabackup-2.2.4-5004.el6.x86_64.rpm warning: percona-xtrabackup-2.2.4-5004.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID cd2efd2a: NOKEY Preparing... ########################################### [100%] 1:percona-xtrabackup ########################################### [100%]
Implementation of Xtrabackup backup
1. Full backup
# innobackupex --user=DBUSER --password=DBUSERPASS /path/to/BACKUP-DIR/
If you want to use a user with minimal privileges for backup, you can create such a user based on the following command :
mysql> CREATE USER ’feiyu'@’localhost’ IDENTIFIED BY ’s3cret’; mysql> REVOKE ALL PRIVILEGES, GRANT OPTION FROM ’feiyu’; mysql> GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO ’feiyu’@’localhost’; mysql> FLUSH PRIVILEGES;
When using innobakupex backup, it will call xtrabackup to back up all InnoDB tables, copy all related files (.frm) related to table structure definition, and related files of MyISAM, MERGE, CSV and ARCHIVE tables, and also backup triggers and files related to database configuration information. These files will be saved to a directory named by time .
While backing up, innobackupex will also create the following files in the backup directory:
(1) xtrabackup_checkpoints——Backup type (such as full or incremental), backup status (such as whether it has been prepared) and LSN (log sequence number) range information;
Each InnoDB page (usually 16k in size) contains a log sequence number, or LSN. LSN is the system version number of the entire database system, and the LSN associated with each page can indicate how the page has changed recently.
(2) xtrabackup_binlog_info——The binary log file currently being used by the mysql server and the location of the binary log event until the moment of backup.
(3) xtrabackup_binlog_pos_innodb - binary log files and the current position of binary log files for InnoDB or XtraDB tables.
(4) xtrabackup_binary - the executable file of xtrabackup used in backup;
(5) backup-my.cnf —— the configuration option information used by the backup command;
When using innobackupex for backup, you can also use the --no-timestamp option to prevent the command from automatically creating a time-named directory; in this way, the innobackupex command will create a BACKUP-DIR directory to store backup data.
2. Prepare a full backup
Generally, after the backup is completed, the data cannot be used for recovery operations, because the backed up data may contain transactions that have not been committed or transactions that have been committed but have not been synchronized to the data file. Therefore, at this point the datafile is still processing an inconsistent state. The main function of "preparation" is to make the data files in a consistent state by rolling back uncommitted transactions and synchronizing committed transactions to data files.
The --apply-log option of the innobakupex command can be used to achieve the above functions. Such as the following command:
# innobackupex --apply-log /path/to/BACKUP-DIR
If executed correctly, the last few lines of information usually output are as follows:
xtrabackup: starting shutdown with innodb_fast_shutdown = 1 120407 9:01:36 InnoDB: Starting shutdown... 120407 9:01:40 InnoDB: Shutdown completed; log sequence number 92036620 120407 09:01:40 innobackupex: completed OK!
In the process of implementing "preparation", innobackupex can usually also use the --use-memory option to specify the size of the memory it can use, and the default is usually 100M. If enough memory is available, you can allocate more memory to the prepare process to speed up its completion.
3. Restore data from a full backup
The --copy-back option of the innobackupex command is used to perform a recovery operation, which performs the recovery process by copying all data-related files to the mysql server DATADIR directory. innobackupex obtains information about the DATADIR directory through backup-my.cnf.
# innobackupex --copy-back /path/to/BACKUP-DIR
When executed correctly, the last few lines of its output usually look like this:
innobackupex: Starting to copy InnoDB log files innobackupex: in '/backup/2012-04-07_08-17-03' innobackupex: back to original InnoDB log directory '/mydata/data' innobackupex: Finished copying back files. 120407 09:36:10 innobackupex: completed OK!
Please make sure that "innobackupex: completed OK!" appears in the last line of the above information.
When the data is restored to the DATADIR directory, it is also necessary to ensure that the owner and group of all data files are correct users, such as mysql, otherwise, the owner and group of the data files need to be modified before starting mysqld. like:
# chown -R mysql:mysql /mydata/data/
4. Use innobackupex for incremental backup
Each InnoDB page will contain an LSN information. Whenever the related data changes, the LSN of the related page will automatically increase. This is the basis on which InnoDB tables can be incrementally backed up, that is, innobackupex is implemented by backing up pages that have changed since the last full backup.
To achieve the first incremental backup, you can use the following command:
# innobackupex --incremental /backup --incremental-basedir=BASEDIR
(If you want to perform incremental backup and full backup, basedir refers to the directory of the full backup, and if you want to perform incremental backup on the last incremental backup, specify the
directory of the last incremental backup)
Among them, BASEDIR refers to the directory where the full backup is located. After the execution of this command, the innobackupex command will create a new directory named after time in the /backup directory to store all incremental backup data. In addition, when performing an incremental backup after performing an incremental backup, its --incremental-basedir should point to the directory where the last incremental backup is located.
It should be noted that incremental backups can only be applied to InnoDB or XtraDB tables. For MyISAM tables, incremental backups are actually full backups.
There are a few differences between "prepare" an incremental backup and a full backup, in particular:
(1) It is necessary to "replay" the committed transactions on each backup (including full and incremental backups). After "replay", all backup data will be merged into the full backup.
(2) "Roll back" uncommitted transactions based on all backups.
(The following is to merge the incremental backup to the full backup, and then only specify the full backup when restoring)
So, the operation becomes:
# innobackupex --apply-log --redo-only BASE-DIR
Then execute:
# innobackupex --apply-log --redo-only BASE-DIR --incremental-dir=INCREMENTAL-DIR-1
And then the second increment:
# innobackupex --apply-log --redo-only BASE-DIR --incremental-dir=INCREMENTAL-DIR-2
Among them, BASE-DIR refers to the directory where the full backup is located, while INCREMENTAL-DIR-1 refers to the directory of the first incremental backup, INCREMENTAL-DIR-2 refers to the directory of the second incremental backup, and so on, that is, if there are multiple incremental backups, the above operation must be performed each time;
5. "Stream" and "Backup Compression" functions of Xtrabackup
Xtrabackup supports the "stream" function for the backup data files, that is, the backup data can be transmitted to the tar program through STDOUT for archiving, instead of being directly saved to a backup directory by default. To use this feature, just use the --stream option. like:
# innobackupex --stream=tar /backup | gzip > /backup/`date +%F_%H-%M-%S`.tar.gz
It is even possible to back up data to another server with a command like:
# innobackupex --stream=tar /backup | ssh [email protected] "cat - > /backups/`date +%F_%H-%M-%S`.tar"
Additionally, when performing a local backup, it is also possible to use the --parallel option to perform a parallel copy of multiple files. This option is used to specify the number of threads to start when copying. Of course, to take advantage of the convenience of this function during actual backup, you also need to enable the innodb_file_per_table option or the shared table space is stored in multiple ibdata files through the innodb_data_file_path option. Replication of multiple files for a database cannot take advantage of this feature. Its simple usage is as follows:
# innobackupex --parallel /path/to/backup
At the same time, the data files backed up by innobackupex can also be stored to a remote host, which can be achieved by using the --remote-host option:
# innobackupex [email protected] /path/IN/REMOTE/HOST/to/backup
6. Import or export a single table
By default, InnoDB tables cannot be migrated between MySQL servers by directly copying table files, even with the innodb_file_per_table option. This function can be realized by using the Xtrabackup tool. However, at this time, the mysql server that needs to "export" the table has enabled the innodb_file_per_table option (strictly speaking, the mysql server has enabled the innodb_file_per_table option before the table to be "exported" is created), and the server that "imports" the table has enabled the innodb_file_per_table and innodb_expand_import options at the same time.
(1) "Export" table
The export table is performed in the prepare phase of the backup. Therefore, once the full backup is completed, a table can be exported through the –export option during the prepare process:
# innobackupex --apply-log --export /path/to/backup
This command will create a file ending with .exp for the table space of each innodb table, and these files ending with .exp can be used to import to other servers.
(2) "Import" table
To import an innodb table from another server on the mysql server, you need to create a table on the current server with the same structure as the original table, and then you can import the table:
mysql> CREATE TABLE mytable (...) ENGINE=InnoDB;
Then delete the tablespace for this table:
mysql> ALTER TABLE mydatabase.mytable DISCARD TABLESPACE;
Next, copy the mytable.ibd and mytable.exp files of the mytable table from the server that "exported" the table to the data directory of the current server, and then "import" it using the following command:
mysql> ALTER TABLE mydatabase.mytable IMPORT TABLESPACE;
7. Use Xtrabackup to perform partial backup of the database
Xtrabackup can also realize partial backup, that is, only back up one or some specified databases or one or some tables in a certain database. But to use this function, the innodb_file_per_table option must be enabled, that is, each table is saved as a separate file. At the same time, it does not support the –stream option, that is, it does not support the transmission of data to other programs for processing through pipelines.
In addition, restoring a partial backup is different from restoring a full data backup, that is, you cannot simply copy a prepared partial backup back to the data directory using the --copy-back option, but restore it through the direction of the import table. Of course, in some cases, some backups can also be restored directly through --copy-back, but most of the data restored in this way will cause data inconsistency, so this method is not recommended anyway.
(1) Create a partial backup
There are three ways to create partial backups: regular expressions (--include), enumerating table files (--tables-file), and listing databases to backup (--databases).
(a) Use –include
When using –include, it is required to specify the full name of the table to be backed up, which is in the form of databasename.tablename, such as:
# innobackupex --include='^feiyu[.]tb1' /path/to/backup
(b) Use --tables-file
The parameter of this option needs to be a file name, and each line in this file contains the full name of a table to be backed up; such as:
# echo -e 'feiyu.tb1\nmageedu.tb2' > /tmp/tables.txt # innobackupex --tables-file=/tmp/tables.txt /path/to/backup
(c) Use –databases
The parameter accepted by this option is the data name. If you want to specify multiple databases, they need to be separated by spaces; at the same time, when specifying a certain database, you can also specify only one of the tables. In addition, this option can also accept a file as a parameter, and each line in the file is an object to be backed up. like:
# innobackupex --databases="feiyu testdb" /path/to/backup
(2)
The process of preparing a partial backup is similar to the process of exporting a table, using the –export option:
# innobackupex --apply-log --export /pat/to/partial/backup
During the execution of this command, innobackupex will call the xtrabackup command to remove missing tables from the data dictionary, so many warning messages about the "table does not exist" category will be displayed. At the same time, information about creating .exp files for tables existing in the backup file will also be displayed.
(3) Restoring a partial backup
The process of restoring a partial backup is the same as that of importing a table. Of course, you can also directly copy the backup in the prepared state to the data directory for restoration. Do not require the data directory to be in a consistent state at this time.
The following actually demonstrates its complete backup process:
←#14#root@localhost /tmp/full-backup →innobackupex --user=root /tmp/full-backup/ #full backup InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved. 。。。。。。。。。。 xtrabackup: The latest check point (for incremental): '2987626' xtrabackup: Stopping log copying thread. .>> log scanned up to (2987626) xtrabackup: Creating suspend file '/tmp/full-backup/2015-06-25_05-58-26/xtrabackup_log_copied' with pid '7858' xtrabackup: Transaction log of lsn (2987626) to (2987626) was copied. 150625 05:58:30 innobackupex: All tables unlocked innobackupex: Backup created in directory '/tmp/full-backup/2015-06-25_05-58-26' innobackupex: MySQL binlog position: filename 'mysql-bin.000001', position 2383 150625 05:58:30 innobackupex: Connection to database server closed 150625 05:58:30 innobackupex: completed OK!
mysql> insert into tutors(tname) values('stu00011');#Insert data in the database
Query OK, 1 row affected (0.03 sec)
mysql> insert into tutors(tname) values('stu00012');
Query OK, 1 row affected (0.00 sec)
←#246#root@localhost /tmp →innobackupex --incremental /tmp/full-backup/ --incremental-basedir=/tmp/full-backup/2015-06-25_05-58-26/ #Make incremental backup InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved. 。。。。。。。。。。。 xtrabackup: Creating suspend file '/tmp/full-backup/2015-06-25_06-00-48/xtrabackup_log_copied' with pid '8663' xtrabackup: Transaction log of lsn (2988209) to (2988209) was copied. 150625 06:00:53 innobackupex: All tables unlocked innobackupex: Backup created in directory '/tmp/full-backup/2015-06-25_06-00-48' innobackupex: MySQL binlog position: filename 'mysql-bin.000001', position 2924 150625 06:00:53 innobackupex: Connection to database server closed 150625 06:00:53 innobackupex: completed OK!
mysql> insert into tutors(tname) values('stu00014'); #insert data again
Query OK, 1 row affected (0.02 sec)
mysql> insert into tutors(tname) values('stu00015');
Query OK, 1 row affected (0.00 sec)
←#247#root@localhost /tmp →innobackupex --incremental /tmp/full-backup/ --incremental-basedir=/tmp/full-backup/2015-06-25_06-00-48 #do incremental backup again InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved 。。。。。。。。。 xtrabackup: Creating suspend file '/tmp/full-backup/2015-06-25_06-02-41/xtrabackup_log_copied' with pid '9259' xtrabackup: Transaction log of lsn (2988781) to (2988781) was copied. 150625 06:02:45 innobackupex: All tables unlocked innobackupex: Backup created in directory '/tmp/full-backup/2015-06-25_06-02-41' innobackupex: MySQL binlog position: filename 'mysql-bin.000001', position 3465 150625 06:02:46 innobackupex: Connection to database server closed 150625 06:02:46 innobackupex: completed OK!
←#266#root@localhost /tmp/full-backup/2015-06-25_05-58-26 →cat xtrabackup_checkpoints #Check whether the log sequence number is consistent backup_type = log-applied from_lsn = 0 to_lsn = 2987626 last_lsn = 2987626 compact = 0 ←#267#root@localhost /tmp/full-backup/2015-06-25_05-58-26 →cd ../2015-06-25_06-00-48/ ←#268#root@localhost /tmp/full-backup/2015-06-25_06-00-48 →cat xtrabackup_checkpoints backup_type = incremental from_lsn = 2987626 to_lsn = 2988209 last_lsn = 2988209 compact = 0 ←#269#root@localhost /tmp/full-backup/2015-06-25_06-00-48 →cd ../2015-06-25_06-02-41/ ←#270#root@localhost /tmp/full-backup/2015-06-25_06-02-41 →cat xtrabackup_checkpoints backup_type = incremental from_lsn = 2988209 to_lsn = 2988781 last_lsn = 2988781 compact = 0
←#248#root@localhost /tmp →innobackupex --apply-log --redo-only /tmp/full-backu2015-06-25_05-58-26/ #Prepare InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved. [notice (again)] If you use binary log and don't use any hack of group commit, the binary log position seems to be: InnoDB: Last MySQL binlog file position 0 2241, file name ./mysql-bin.000001 xtrabackup: starting shutdown with innodb_fast_shutdown = 1 InnoDB: Starting shutdown... InnoDB: Shutdown completed; log sequence number 2987626 150625 06:04:03 innobackupex: completed OK! ←#249#root@localhost /tmp →innobackupex --apply-log --redo-only /tmp/full-backu2015-06-25_05-58-26/ --incremental-dir=/tmp/full-backup/2015-06-25_06-00-48/ #Merge the first incremental backup file 。。。。。。。。。。 innobackupex: Copying '/tmp/full-backup/2015-06-25_06-00-48/management/admin.frm' to '/tmp/full-backup/2015-06-25_05-58-26/management/admin.frm' 150625 06:05:28 innobackupex: completed OK! ←#251#root@localhost /tmp →innobackupex --apply-log --redo-only /tmp/full-backu2015-06-25_05-58-26/ --incremental-dir=/tmp/full-backup/2015-06-25_06-02-41/ #Merge the second incremental backup file 。。。。。。。。。。。。。。。。 innobackupex: Copying '/tmp/full-backup/2015-06-25_06-02-41/management/classinfo.frm' to '/tmp/full-backup/2015-06-25_05-58-26/management/classinfo.frm' innobackupex: Copying '/tmp/full-backup/2015-06-25_06-02-41/management/admin.frm' to '/tmp/full-backup/2015-06-25_05-58-26/management/admin.frm' 150625 06:07:10 innobackupex: completed OK! ←#258#root@localhost ~ →rm -rf /mydata/data1/* #Delete data file directory ←#259#root@localhost ~ →innobackupex --copy-back /tmp/full-backup/2015-06-25_05-58-26/ #恢复 。。。。。。。。。。。。。。。。 innobackupex: Starting to copy InnoDB log files innobackupex: in '/tmp/full-backup/2015-06-25_05-58-26' innobackupex: back to original InnoDB log directory '/mydata/data1' innobackupex: Finished copying back files. 150625 06:12:29 innobackupex: completed OK! ←#276#root@localhost /mydata/data1 →chown -R mysql.mysql ./* #Modify owner and group