Detailed explanation of mysqldump

Ⅰ. Simple use and attention of mysqldump

1.1 Basic parameters

Only back up innodb, you can't use a few parameters, just remember the following, the others are useless

-A 备份所有的database
-B 备份哪几个数据库
-R 备份存储过程(-- routines)
-E 备份定时任务(-- events)
-d 只备份表结构
-w 备份过滤数据
-t 只备份数据
-q 直接读数据,绕过缓冲池,默认已加
--triggers 备份触发器
--master-data=2 在备份文件中以注释的形式记录备份开始时binlog的position,默认值是1,不注释

tips:
①mysqldump无法备份视图
②--set-gtid-purged=OFF 如果实例开了gtid最好加上这个参数,不然备份时候会报warning,且备份出来的数据恢复到其他版本的实例上会报错：A partial dump from a server that has GTIDs is not allowed.
③--dump-slave,该参数可以用作在从库做备份获取主库的位置点,来做一个新从库,避免在主库做备份影响业务,带该参数备份时,从上sql线程会被kill,备份完再拉起

Common usage:

mysqldump --single-transaction -B test a > backup.sql    备份test库和a库
mysqldump --single-transaction test a > backup.sql       备份test库下的a表
mysqldump --single-transaction test a -w "c=12"> backup.sql

1.2 Other parameters

--lock-tables(-l)
在备份中依次锁住所有表,一般用于myisam备份,备份时数据库只能提供读操作,以此来保证数据一致性,该参数和--single-transaction是互斥的,所以实例中既存在myisam又存在innodb则,只能使用该参数

--lock-all-tables(-x)
比上面的参数力度更大,备份时将整个实例锁住

1.3 Key points

--single-transaction
必须加（一个事务中导出数据,确保产生一致性的备份数据）

Add the following configuration to my.cnf

[mysqldump]
single-transaction
master-data=2

Ⅱ, mysqldump implementation principle analysis

2.1 Open glog and take a look (^__^)

session 1:
(root@localhost) [(none)]> truncate mysql.general_log;
Query OK, 0 rows affected (0.02 sec)

(root@localhost) [(none)]> set global general_log = 1;
Query OK, 0 rows affected (0.00 sec)

(root@localhost) [(none)]> set global log_output = 'table';
Query OK, 0 rows affected (0.00 sec)

session 2:
[root@VM_0_5_centos src]# mysqldump --single-transaction --master-data=2 -B dump_test > /tmp/back.sql

session 1:
(root@localhost) [(none)]> set global general_log = 0;
Query OK, 0 rows affected (0.00 sec)

(root@localhost) [(none)]> set global log_output = 'file';
Query OK, 0 rows affected (0.00 sec)

(root@localhost) [(none)]> select thread_id,left(argument,64) from mysql.general_log;

(root@localhost) [(none)]> select argument from mysql.general_log where thread_id=239 order by event_time;

FLUSH /*!40101 LOCAL */ TABLES
# 先把表刷一把,减少ftwrl锁的等待时间
FLUSH TABLES WITH READ LOCK
# 把当前整个实例锁成只读
SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
# 设置备份线程事务隔离级别为rr
START TRANSACTION /*!40100 WITH CONSISTENT SNAPSHOT */
# 开启事务
SHOW VARIABLES LIKE 'gtid\_mode'                                                                    
SHOW MASTER STATUS
# 获得当前二进制日志位置
UNLOCK TABLES
# 释放实例级别的只读锁
SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IN (SELECT DISTINCT LOGFILE_GROUP_NAME FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA IN ('dump_test'))) GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME
SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA IN ('dump_test')) ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME
SHOW VARIABLES LIKE 'ndbinfo\_version'
dump_test
SHOW CREATE DATABASE IF NOT EXISTS `dump_test`
SAVEPOINT sp
# 使用savepoint sp,便于回滚,用于快速释放metadata数据共享锁
show tables                                                                                        
show table status like 'dump\_inno'
SET SQL_QUOTE_SHOW_CREATE=1
SET SESSION character_set_results = 'binary'
show create table `dump_inno`
SET SESSION character_set_results = 'utf8'
show fields from `dump_inno`
show fields from `dump_inno`
SELECT /*!40001 SQL_NO_CACHE */ * FROM `dump_inno`
SET SESSION character_set_results = 'binary'
use `dump_test`
select @@collation_database
SHOW TRIGGERS LIKE 'dump\_inno'
SET SESSION character_set_results = 'utf8'
# 以上部分备份数据
ROLLBACK TO SAVEPOINT sp                                                                            
RELEASE SAVEPOINT sp
# 回到savepoint sp,释放metadata的锁
# 每取一张表的数据,就rollback to savepoint sp（一个savepoint就够了）

root@localhost on  using Socket
/*!40100 SET @@SQL_MODE='' */
/*!40103 SET TIME_ZONE='+00:00' */

2.2 mysqldump process summary

-	operate	Parse
step1	flush tables/flush tables with read lock	Lock the instance as read-only
step2	SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ/START TRANSACTION	Turn on transactions with rr isolation level
step3	SHOW MASTER STATUS/UNLOCK TABLES	Get the binary log location and release the global read lock
step4	SAVEPOINT sp	set rewind point
step5	select xxx	backup data
step6	ROLLBACK TO SAVEPOINT sp/RELEASE SAVEPOINT sp	End a table backup and go back to sp

2.3 Details of key points

start transaction with consistent snapshot

Location problem: The transaction isolation level is rr, the transaction is started, and a read_view is created immediately, so the data backed up by mysqldump is the data at the beginning of the backup instead of the data at the end of the backup (backed up for 30 minutes, the entire process instance is always readable and writable. , the data before 30min is backed up instead of the data after 30min)

Execute start transaction at the same time (instead of waiting until the first sql is executed) to establish a snapshot of consistent read with this transaction

--single-transaction

All databases are read in a transaction, and the transaction isolation level is such as rr, so the read data is consistent

Consistent backup: The entire backup starts from the start transaction, backs up all tables, and the data of all tables is in a transaction, which is exported by select

savepoint savepoint (added in 4.1 or 5.0)

Savepoint is rarely used, and the most used one is when backing up a table. After a table is backed up, it will be rolled back to the corresponding savepoint. At this time, the metadata locks on the table corresponding to the backup are released. At this time, the table can be used as ddl. operate.

Otherwise, in a transaction, holding a metadata lock and doing ddl (for example, other threads want to create an index on a table here), even if the backup is completed, it cannot be done, and it can only be moved after all backups are completed.

When there is no savepoint, the read-only lock is held for the entire transaction time, not the time of the table backup

Ⅲ. Routine operation

① Backup and compress

mysqldump --single-transaction --master-data=1 --triggers -R -E -B sbtest | pv | gzip -c > sbtest.backup.tgz

Compressed backup recovery

gunzip < sbtest.backup.tgz | mysql

② Backup and compress to a remote server

mysqldump --single-transaction --master-data=1 --triggers -R -E -B sbtest | gzip -c | ssh root@test-3 'cat > /tmp/sbtest.sql.gz'

Backup verification, to be considered separately

③Use of backup files

mysql < xxx.sql;

tips:

Backups take up a lot of bandwidth, and a scheduling algorithm is required to ensure that only one machine in the same cluster does backups at the same time, or does not do backups every day to stagger the backup time