I. Description of the problem
Received a written warning from the failure of the library, so the on-line viewing, found mainly caused by the backup, but many online MySQL, and other instances no problem, this is the only reported instance, there must be other reasons, keep looking.
First, we look at the log backup at 20:14:16, when the program will idb backup files are backed up, and then began to prepare the backup file frm performed first flush no_write_to_binlog tables, then execute flush tables with read lock, it did not problem,
But from one second start, the library will block live from DDL and DML operations. As the backup log too long, here I only part of the interception important tips to them.
20:14:20 when prompted: Finished backing up non-InnoDB tables and files, that is, from the beginning of 20:14:16 seconds frm file backup, backup to the end, with only four seconds, this in line with expectations, no problem.
But then it is flush no_write_to_binlog engine logs run about 33 minutes, and then it performs table unlock. This problem is out in 33 minutes here, locked for too long, the alarm is triggered.
Second, the investigation process
1, parameter settings see log_file found log_file setting became 8G, and set up 12 documents.
> show variables like '%innodb_log_file%';
+---------------------------+------------+
| Variable_name | Value |
+---------------------------+------------+
| innodb_log_file_size | 8589934592 |
| innodb_log_files_in_group | 12 |
+---------------------------+------------+
2 rows in set (0.00 sec)
8.0G ib_logfile0
8.0G ib_logfile1
8.0G ib_logfile10
8.0G ib_logfile11
8.0G ib_logfile2
8.0G ib_logfile3
8.0G ib_logfile4
8.0G ib_logfile5
8.0G ib_logfile6
8.0G ib_logfile7
8.0G ib_logfile8
8.0G ib_logfile9
2, but the tone of this parameter for a reason, and that is innobackupex needs, if the arguments piecemeal, as follows (need to restart the instance), again there will be a backup error:
Amended to 256M, a total of four documents
show variables like '%innodb_log_file%';
+---------------------------+-----------+
| Variable_name | Value |
+---------------------------+-----------+
| innodb_log_file_size | 268435456 |
| innodb_log_files_in_group | 4 |
+---------------------------+-----------+
After the small file transfer, backup again, the error occurs is:
xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.
3880 xtrabackup: Error: xtrabackup_copy_logfile() failed.
On this issue, the following article has described me this instance data size is 2.2T, by compressing the size of the backup after the close 1.2T, In addition, we are using a remote backup, file copy speed to do, so back up spending nearly 12 hours, which also led to ib_logfile small files can not be transferred, otherwise the backup is unsuccessful.
3, see a post said that the broadband network problems behind and I test it, use this command to test the backup is successful, innobackupex --defaults-file = / data / mysql / 3306 / my.cnf --user = xxxx --password = xxxx --stream = tar / data / tmp> / dev / null
Scan is completed log immediately (seconds to complete the same), it does not lead to delays. Is time to start the backup: 09: 45: 12, the start time of the lock table is: 11: 05: 50 seconds, unlock time: 11: 05: 51 is only about one second lock, the end of backup time is 11:06:49, the entire backup took only about 81 minutes.
Reference article
- 链接1：https://bugs.launchpad.net/percona-xtrabackup/+bug/1050355
- 链接2：https://bugs.launchpad.net/percona-xtrabackup/+bug/1206309
- 链接3：https://blog.csdn.net/caoshulin1989/article/details/73332379
Third, the current risks and improve
1, this example during the backup, will lead to a read-only instance of 33 minutes, can only provide access, though from the library, but 33 minutes from the library belongs to the delay of the state, does not allow for delays from the library business, is not appropriate . At present this is not operational from the library, all businesses have access to the main library;
2, during the backup lock table, appear delayed from the library, if this time the main library hardware failure, MHA automatic failover can not be completed, due to delays from the library and a prompt to switch fails (switching logic to be perfect)
3, for the monitor can not write, write for this case due to failure, not for disaster level of alarm.