Mysql backup and recovery, and log management

1. Mysql log management

The default storage location of MySQL logs is /usr/local/mysql/data

1.1. Log classification

• Transaction log:
  The write type of transaction log transaction log is "append", so its operation is "sequential IO"; it is also commonly called: write-ahead log write ahead logging transaction log file: ib_logfile0, ib_logfile1

• error logerror log

• general loggeneral log

• slow query log slow query log

• binary logbinary log

• Relay log relay log, in the master-slave replication architecture, the slave server is used to save events read from the binary log of the master server

1.1.1. Error log

用来记录当MySQL启动、停止或运行时发生的错误信息，默认已开启

vim /etc/my.cnf
log-error=/usr/local/mysql/data/mysql_error.log	

1.1.2, General query log

用来记录MySQL的所有连接和语句，默认是关闭的

vim /etc/my.cnf
general_log=ON
general_log_file=/usr/local/mysql/data/mysql_general.log

1.1.3. Binary log

用来记录所有更新了数据或者已经潜在更新了数据的语句，记录了数据的更改，可用于数据恢复，默认已开启

vim /etc/my.cnf
#也可以 log_bin=mysql-bin
log-bin=mysql-bin				

1.1.4, Slow query log

用来记录所有执行时间超过long_query_time秒的语句，可以找到哪些查询语句执行时间长，以便于优化，默认是关闭的

vim /etc/my.cnf
slow_query_log=ON
slow_query_log_file=/usr/local/mysql/data/mysql_slow_query.log
long_query_time=5	

1.1.5, configuration log

1. #修改my.cnf配置文件
 
#错误日志
log-error=/usr/local/mysql/data/mysql_error.log	
#通用查询日志
general_log=ON
general_log_file=/usr/local/mysql/data/mysql_general.log
#二进制日志
log-bin=mysql-bin	
#慢查询日志
slow_query_log=ON
slow_query_log_file=/usr/local/mysql/data/mysql_slow_query.log
long_query_time=5
 
2. #重新mysql服务
systemctl restart mysqld.service

Modify configuration file

and restart service

1.2. Log query

#登入mysql
mysql -u root -p[密码]
 
#查看通用查询日志是否开启
show variables like 'general%';	
 
#查看二进制日志是否开启
show variables like 'log_bin%';									
#查看慢查询日功能是否开启
show variables like '%slow%';								
#查看慢查询时间设置
show variables like 'long_query_time';						
#在数据库中设置开启慢查询的方法
set global slow_query_log=ON;									

2. Backup and recovery

2.1. The necessity of data backup

• The main purpose of backup is disaster recovery

• In a production environment, data security is critical

• Any loss of data can have serious consequences

2.2. Reasons for data loss

• program error

• Human operating error.

• Operation error

• disk failure

• Disasters (such as fires, earthquakes) and theft

2.3. Classification of database backup

2.3.1. Physical backup

对数据库操作系统的物理文件(如数据文件、日志文件等)的备份。

Physical backup methods:

• Cold backup (offline backup): is performed when the database is closed;
• Hot backup (online backup): the database is running and relies on the log file of the database;
• Warm backup: Backup operation is performed while the database table is locked (not writable but readable).

2.3.2. Logical backup

对数据库的表或者对象进行备份。

2.4. Database backup strategy

• Full backup : The entire data set, the complete database is backed up every time.

• Differential backup : only backs up data that has changed since the most recent full backup. The backup is slow and the restore is simple.

• Incremental backup : only backs up data that has changed since the last full backup or incremental backup (if an incremental backup exists). The backup is fast and the restore is complicated.

1) Full backup : Make a complete backup of the database every time

- 是对整个数据库、数据库结构和文件结构的备份

- 保存的是备份完成时刻的数据库

- 是差异备份与增量备份的基础

优点：备份与恢复操作简单方便 缺点：数据存在大量的重复、占用大量的备份空间及备份与恢复时间长 

2) Differential backup : Back up files that have been modified since the last full backup

3) Incremental backup : Only files that have been modified since the last full backup or incremental backup will be backed up.

2.5, MySQL full backup

The database of the InnoDB storage engine is stored in three files on the disk:

• db.opt (table properties file)

• Table name.frm (table structure file)

• Table name.ibd (table data file)

2.5.1. Physical cold backup and recovery

• The database is closed during backup and the database files are packaged directly.

• Backup is fast and recovery is also the easiest

1. #关闭mysql，备份data目录
systemctl stop mysqld
yum -y install xz
cd /usr/local/mysql
 
#压缩备份data目录
tar Jcvf /opt/mysql_all_$(date +%F).tar.xz data/
 
2. #登录mysql，删除NBA库 
systemctl start mysqld.service 
mysql -u root -p123123
 
 
3. #解压之前备份的数据库data目录，不用删除原目录，会自动替换
cd /opt
ls
cd /usr/local/mysql
tar Jxvf /opt/mysql_all_2021-11-28.tar.xz -C ./
 
4. #重启服务查看被删除的库 

Create a library and create a table in the library.

Close mysql and back up the data directory.


Log in to mysql and delete the NBA library. Unzip

the previously backed up database data directory. There is no need to delete the original directory, it will be automatically replaced.

2.5.2. Special backup tools

• mysqldump commonly used logical backup tool

• mysqlhotcopy only has backup MyISAM and ARCHIVE tables

Full backup of one or more complete libraries (including all tables within them)

#导出的就是数据库脚本文件
mysqldump -u root -p[密码] --databases 库名1 [库名2] … > /备份路径/备份文件名.sql	

Example: Backing up single and multiple libraries

mysqldump -uroot -p --databases NBA > /opt/mysql_bak/nba.sql
mysqldump -uroot -p --databases  NBA info1 > /opt/mysql_bak/nba_info1.sql

Complete backup of all libraries in the MySQL server

mysqldump -u root -p[密码] --all-databases > /备份路径/备份文件名.sql

Example: Back up all libraries

mysqldump -uroot -p --all-databases > /opt/mysql_bak/all.sql

Full backup of some tables in the specified library

mysqldump -u root -p[密码] 库名 [表名1] [表名2] … > /备份路径/备份文件名.sql

Example: Back up two tables in the NBA database

mysqldump -uroot -p NBA  [-d]  info1 info2 > /opt/mysql_bak/school_info1_info2.sql

• Use the "-d" option to save only the table structure of the database

• If the "-d" option is not used, the table data will also be backed up.

View backup files

 cat school_info1_info2.sql |grep -v '^/'|grep -v '^$'|grep -v '^-'

2.5.3. MySQL full backup (use without interaction)

systemctl start mysqld

mysql -u root -p -e 'drop database NBA;
#“-e”选项，用于指定连接 MySQL 后执行的命令，命令执行完后自动退出
mysql -u root -p -e 'show databases;'
 
#恢复school数据库
mysql -u root -p < /opt/mysql_bak/school.sql		
mysql -u root -p -e 'show databases;'

Restore data table

当备份文件中只包含表的备份，而不包含创建的库的语句时，执行导入操作时必须指定库名，且目标库必须存在。

mysqldump -uroot -p school  info1 info2 > /opt/mysql_bak/school_info1_info2.sql
 
1. #删除school中的info1表 
mysql -u root -p -e 'drop table NBA.lilade;'
mysql -u root -p -e 'show tables from school;'
 
2. #恢复school库中的表 
mysql -u root -p NBA < /opt/mysql_bak/school_info1_info2.sql 
mysql -u root -p -e 'show tables from NBA;'

Delete the info table in NBAl

and restore the table in NBA library

2.5.4, MySQL incremental backup and recovery

vim /etc/my.cnf
[mysqld]
log-bin=mysql-bin
#指定二进制日志(binlog)的记录格式为 MIXED
binlog_format = MIXED				
server-id = 1

Binary log (binlog) has 3 different recording formats:

• STATEMENT (based on SQL statement)

• ROW (row-based)

• MIXED (mixed mode), the default format is STATEMENT

#只要重启就会自动生成二进制文件
systemctl start mysqld
ls -l /usr/local/mysql/data/mysql-bin.*

Incremental backup operations can be performed every day to generate new binary log files (for example, mysql-bin.000002)

mysqladmin -u root -p flush-logs

Insert new data to simulate additions or changes to data

insert into  NBA.info  values (6, 'cj', 33, '男', '打篮球');
insert into  NBA.info  values (7, 'lu', 33, '男', '打篮球');

Generate a new binary log file again (e.g. mysql-bin.000007)

mysqladmin -u root -p flush-logs

View the contents of the binary log file

cp /usr/local/mysql/data/mysql-bin.000007 /opt/
mysqlbinlog --no-defaults --base64-output=decode-rows -v /opt/mysql-bin.000007
#--base64-output=decode-rows：使用64位编码机制去解码并按行读取
#-v：显示详细内容

2.5.5, MySQL incremental recovery

1. general recovery

 将所有备份的二进制日志内容全部恢复

Simulate recovery steps for lost changed data

use nba;
delete from info where id=6;
delete from info where id=7;
 
mysqlbinlog --no-defaults /usr/local/mysql/data/mysql-bin.000007 | mysql -u root -p

Deleting data

Recovery via binary logs


Simulating recovery steps for losing all data

use NBA;
drop table info;
 
mysql -u root -p NBA < /opt/mysql_bak/NBA_info_info2.sql
mysqlbinlog --no-defaults /usr/local/mysql/data/mysql-bin.000007 | mysql -u root -p

Delete the info1 table in the school library

First restore the entire table, and then restore the binary log file

based on location recovery

• The database may have both incorrect and correct operations at a certain point in time.

• Wrong operations can be skipped based on precise location

• A node before the error node, where the last correct operation stopped.
  

# 刷新生成新的二进制日志文件
mysqladmin -u root -p flush-logs 	
# 进入到data目录
cd /usr/local/mysql/data	
#查看二进制日志文件
mysqlbinlog --no-defaults --base64-output=decode-rows -v mysql-bin.000012

Insert data into the database and view it

use school
insert into info values(8,'小斌','男','唱歌');

mysqlbinlog --no-defaults --base64-output=decode-rows -v mysql-bin.0000008	
# at 1207										#断点
#211130  9:28:59 								#时间
insert into info values(8,'小斌','男','唱歌')     #插入的数据
insert into info values(8,'小斌','男','唱歌')

Insert another piece of data

use school
insert into info values(9,'晴','女','画画');
 
# at 22236
#211130 10:36:38
insert into info values(9,'晴','女','画画')

The test is to restore the data before the ID is 609. The data of "the next one" will not be restored.

删除数据之前必须先刷新一下日志文件

mysqladmin -u root -p flush-logs 

Restore data

mysqlbinlog --no-defaults --stop-position='22236' mysql-bin.000012 | mysql -u root -p

Only the "last" data is recovered, and the "previous multiple" data is skipped.

删除表中所有数据

time based recovery

测试恢复到ID为609之前的数据， 不恢复“后面一条”的数据

Only the "last" data is recovered, and the "previous multiple" data is skipped.

mysqlbinlog --no-defaults --start-datetime='2021-11-30 10:36:38' /usr/local/mysql/data/mysql-bin.000012 |mysql -uroot -p123123