MySQL database log management, backup and recovery (graphic details!)

MySQL database log management, backup and recovery

Preparation: first create a database and table

mysql -u root -p

create database SCHOOL;
use SCHOOL;
create table class (id int(10) not null,name varchar(10) not null,cardid varchar(18),phone varchar(11),address varchar(50));

desc class;

insert into class values (1,'zhangsan','12','111111','nanjing');
insert into class values (4,'lisi','123','444444','suzhou');
insert into class values (2,'wangwu','1234','222222','beijing');
insert into class values (5,'zhaoliu','12345','555555','nanjing');
insert into class values (3,'qianqi','123456','333333','shanghai');

select * from class;

Insert data and view table structure

One, MySQL log management

The default storage location of MySQL logs is /usr/local/mysql/data

1.1 MySQL log classification

1. Error log
2. General query log
3. Binary log
4. Slow query log

1.2 MySQL log is turned on

• By permanently repairing the MySQL configuration file

vim /etc/my.cnf
[mysqld]
##错误日志，用来记录当MySQL启动、停止或运行时发生的错误信息，默认已开启
#指定错误日志的保存位置和文件名
log-error=/usr/local/mysql/data/mysql_error.log

##通用查询日志，用来记录MySQL的所有连接和语句，默认是关闭的
general_log=ON
general_log_file=/usr/local/mysql/data/mysql_general.log

##二进制日志(binlog)，用来记录所有更新了数据或者已经潜在更新了数据的语句，记录了数据的更改，可用于数据恢复，默认已开启
log-bin=mysql-bin
#也可以 log_bin=mysql-bin

##慢查询日志，用来记录所有执行时间超过long_query_time秒的语句，可以找到哪些查询语句执行时间长，以便于优化，默认是关闭的
slow_query_log=ON
slow_query_log_file=/usr/local/mysql/data/mysql_slow_query.log
long_query_time=5  #设置超过5秒执行的语句被记录，缺省时为10秒

systemctl restart mysqld.service 

1.3 Enter the database to check whether the corresponding log is enabled

1) Check whether the general query log is enabled

mysql -u root -p
show variables like 'general%';

2) Check whether the binary log is turned on

show variables like 'log_bin%';

3) View related functions of slow query log

(1) Check whether the slow query log function is enabled

show variables like '%slow%';

(2) View the slow query time setting

show variables like 'long_query_time';

(3) Set the method to start slow query in the database-

It has been configured before and will not be demonstrated anymore

set global slow_query_log=ON;

Two, MySQL full backup and recovery

2.1 The importance of data backup

• The main purpose of backup is disaster recovery
• In a production environment, data security is of paramount importance
• Any loss of data may have serious consequences

2.2 Causes of data loss

• Program error
• Human error
• Operation error
• Disk failure
• Disasters (such as fires, earthquakes) and theft

2.3 Classification of database backup

1) Classification from the perspective of physics and logic

(1) Physical backup

• Backup of physical files (such as data files, log files, etc.) of the database operating system
• Methods of physical backup
  • Cold backup (offline backup): It is performed when the database is closed
  • Hot backup (online backup): The database is running and depends on the log file of the database
  • Warm backup: the database is backed up when the table is locked (not writable but readable)

(2) Logical backup

• Backup of database logic components (such as database objects such as tables)

2) Classification from the perspective of database backup strategy

(1) Full backup

• Make a complete backup of the database every time

(2) Differential backup

• Back up files that have been modified since the last full backup

(3) Incremental backup

• Only files modified after the last full backup or incremental backup will be backed up

2.4 Common backup methods

1) Physical cold standby

• The database is closed during backup, and the database files are directly packaged
• The backup speed is fast, and the recovery is also the easiest

2) Dedicated backup tool mydump or mysqlhotcopy

• Commonly used logical backup tools for mysqldump
• mysqlhotcopy only has backup MyISAM and ARCHIVE tables

3) Enable binary log for incremental backup

• For incremental backups, the binary log needs to be refreshed

4) Third-party tool backup

• Free MySQL hot backup software Percona XtraBackup

2.5 MySQL full backup

1) Concept

• It is a backup of the entire database, database structure and file structure
• What is saved is the database at the time the backup is completed
• It is the basis of differential backup and incremental backup

2) Advantages and disadvantages of full backup

• advantage:

  Simple and convenient operation of backup and recovery

• Disadvantages:

  There is a lot of duplication in data

  Takes up a lot of backup space

  Long backup and restore time

3) Database full backup classification

(1) Physical cold backup and recovery

• Shut down the MySQL database
• Use the tar command to directly package the database folder
• Simply replace the existing MySQL directory

(2) mysqldump backup and recovery

• MySQL's own backup tool can easily realize the backup of MySQL
• You can export specified libraries and tables as SQL scripts
• Use the command mysql to import the backup data

3. Basic commands for cold database backup and recovery and full backup and recovery

3.1 Physical cold backup and recovery

systemctl stop mysqld
yum -y install xz					#xz是一个压缩工具
#压缩备份
tar Jcvf /opt/mysql_all_$(date +%F).tar.xz /usr/local/mysql/data/
#解压恢复
tar Jxvf /opt/mysql_all_2021-02-05.tar.xz -C /usr/local/mysql/data

systemctl start mysqld

3.2 mysqldump backup and recovery

1) Full backup of one or more complete libraries (including all tables)

#导出的备份文件就是数据库脚本文件
mysqldump -u root -p[密码] --databases 库名1 [库名2] … > /备份路径/备份文件名.sql
例：
mysqldump -u root -p --databases school > /opt/school.sql
mysqldump -u root -p --databases mysql school > /opt/mysql-school.sql

2) Full backup of all libraries in the MySQL server

mysqldump -u root -p[密码] --all-databases > /备份路径/备份文件名.sql
例：
mysqldump -u root -p --all-databases > /opt/all.sql

3) Full backup of some tables in the specified library

mysqldump -u root -p[密码] [-d] 库名 [表名1] [表名2] … > /备份路径/备份文件名.sql
#使用“ -d ”选项，说明只保存数据库的表结构
#不使用“ -d ”选项，说明表数据也进行备份
例：
mysqldump -u root -p school class > /opt/school_class.sql

4) View the backup file

cat /opt/备份的文件 |grep -v "^--" | grep -v "^/" | grep -v "^$"

例：
cat /opt/school_class.sql |grep -v "^--" | grep -v "^/" | grep -v "^$"

3.3 MySQL complete recovery

1) Restore the database

#“-e”选项，用于指定连接 MySQL 后执行的命令，命令执行完后自动退出
mysql -u root -p -e 'drop database school;'
mysql -u root -p -e 'show databases;'

mysql -u root -p < /opt/school.sql
mysql -u root -p -e 'show databases;'

2) Restore the data table

Note: When the backup file contains only the backup of the table and does not contain the statement of the created library, the library name must be specified when the import operation is performed, and the target library must exist.

mysql -u root -p -e 'drop table school.class;'
mysql -u root -p -e 'show tables from school;'

mysql -u root -p school < /opt/school_class.sql
mysql -u root -p -e 'show tables from school;'

Four, MySQL incremental backup and recovery method

4.1 MySQL incremental backup

1) Turn on the binary log function

vim /etc/my.cnf
[mysqld]
log-bin=mysql-bin
server-id = 1
binlog_format = MIXED				#指定二进制日志(binlog)的记录格式为 MIXED


#二进制日志(binlog)有3种不同的记录格式：STATEMENT(基于SQL语句)、ROW(基于行)、MIXED(混合模式)，默认格式是STATEMENT

systemctl restart mysqld.service
ls -l /usr/local/mysql/data/mysql-bin.*

2) A full backup of the database or table can be performed every week

#手动执行备份
mysqldump -u root -p school class > /opt/school_class_$(date +%F).sql
mysqldump -u root -p --all-databases > /opt/allmysql_$(date +%F).sql

#使用crontab -e 计划性任务来执行；每周1凌晨2点对表class和所有的库进行备份
0 2 * * 1 mysqldump -u root -p school class > /opt/school_class_$(date +%F).sql
0 2 * * 1 mysqldump -u root -p --all-databases > /opt/allmysql_$(date +%F).sql

Note: This kind of timing task can be executed in combination with crontab -e scheduled task

3) Incremental backup operations can be performed every day to generate new binary log files

mysqladmin -u root -p flush-logs

4) Insert new data to simulate the increase or change of data

mysql -u root -p
use school;
insert into class values ('6','qqq','223366','666666','nanjing');
insert into class values ('7','www','666555','777777','changzhou');

5) Generate a new binary log file again

mysqladmin -u root -p flush-logs
#之前的步骤4的数据库操作会保存到mysql-bin.000002文件中，之后数据库数据再发生变化则保存在mysql-bin.000003文件中

6) View the contents of the binary log file

cp /usr/local/mysql/data/mysql-bin.000002 /opt/
mysqlbinlog --no-defaults --base64-output=decode-rows -v /opt/mysql-bin.000002
#--base64-output=decode-rows：使用64位编码机制去解码并按行读取
#-v：显示详细内容

4.2 MySQL incremental recovery

1) General recovery

(1) Simulate the recovery steps of lost and changed data

mysql -u root -p
use school;
delete from class where id=6;
delete from class where id=7;
select * from class;
quit

mysqlbinlog --no-defaults /opt/mysql-bin.000002 | mysql -u root -p
mysql -u root -p -e "select * from school.class;"

(2) Simulate the recovery steps of all lost data

mysql -u root -p
use school;
drop table class;
show tables;
quit

mysql -uroot -p school < /opt/school_class_2021-02-24.sql
mysqlbinlog --no-defaults /opt/mysql-bin.000002 | mysql -uroot -p
mysql -u root -p -e "select * from school.class;"

2) Breakpoint recovery

mysqlbinlog --no-defaults --base64-output=decode-rows -v /opt/mysql-bin.000002

#部分二进制文件的内容
......
BEGIN
/*!*/;
##-------------解释：at xxx 表示位置点------------------------------------------------
# at 302
##--------------解释：开头210224 15:45:53表示时间，其他的现在用不到-----------------------------------
#210224 15:45:53 server id 1  end_log_pos 449 CRC32 0xe972def7 	Query	thread_id=6	exec_time=0	error_code=0
##--------------解释：这里是执行的操作语句---------------------
use `school`/*!*/;        <-------------use school;使用数据库
SET TIMESTAMP=1612597553/*!*/; <------------建立时间戳
insert into class values ('6','qqq','223366','666666','nanjing') <-------向表中插入数据
/*!*/;
##---------------------------------------------------------------
# at 449
#210224 15:45:53 server id 1  end_log_pos 480 CRC32 0x5efde826 	Xid = 446
COMMIT/*!*/;
# at 480
#210224 15:45:54 server id 1  end_log_pos 545 CRC32 0x11768895 	Anonymous_GTID	last_committed=1	sequence_number=2	rbr_only=no
SET @@SESSION.GTID_NEXT= 'ANONYMOUS'/*!*/;
# at 545
#210224 15:45:54 server id 1  end_log_pos 628 CRC32 0x778ea5fa 	Query	thread_id=6	exec_time=0	error_code=0
SET TIMESTAMP=1612597554/*!*/;
##-------------------------------插入第二个数据--------------------------
BEGIN
/*!*/;
# at 628
#210206 15:45:54 server id 1  end_log_pos 775 CRC32 0x66e3bb53 	Query	thread_id=6	exec_time=0	error_code=0
SET TIMESTAMP=1612597554/*!*/;
insert into class values ('7','www','666555','777777','changzhou')
/*!*/;
# at 775
#210224 15:45:54 server id 1  end_log_pos 806 CRC32 0x7b972395 	Xid = 447
COMMIT/*!*/;
# at 806
#210224 15:48:52 server id 1  end_log_pos 853 CRC32 0x0d77c456 	Rotate to mysql-bin.000003  pos: 4
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
.......

(1) Restore based on location

Only restore to the data before the position "628", that is, do not restore the data of "id=7"

#模拟数据丢失
mysql -uroot -p123456 school < /opt/school_class_2021-02-24.sql
mysql -uroot -p123456 -e "select * from school.class;"
#到位置点628停止恢复数据
mysqlbinlog --no-defaults --stop-position='628' /opt/mysql-bin.000002 | mysql -uroot -p123456
#查看class表的数据
mysql -uroot -p123456 -e "select * from school.class;"

Restore only the data of "id=7", skip the data of "id=6"

#模拟数据丢失
mysql -uroot -p123456 school < /opt/school_class_2021-02-24.sql
mysql -uroot -p123456 -e "select * from school.class;"
#从位置点628开始恢复数据
mysqlbinlog --no-defaults --start-position='628' /opt/mysql-bin.000002 | mysql -uroot -p123456
#查看class表的数据
mysql -uroot -p123456 -e "select * from school.class;"

(2) Recovery based on point in time

Only restore the data before 210224 15:45:54, that is, do not restore the data with "id=7"

#模拟数据丢失
mysql -uroot -p123456 school < /opt/school_class_2021-02-24.sql
mysql -uroot -p123456 -e "select * from school.class;"
#到2021-02-24 15:45:54截止恢复数据
mysqlbinlog --no-defaults --stop-datetime='2021-02-24 15:45:54' /opt/mysql-bin.000002 | mysql -uroot -p123456
#查看class表的数据
mysql -uroot -p123456 -e "select * from school.class;"

Only restore the data of "id=7", skip the data restoration of "id=6"

#模拟数据丢失
mysql -uroot -p123456 school < /opt/school_class_2021-02-24.sql
mysql -uroot -p123456 -e "select * from school.class;"
#从2021-02-24 15:45:54开始恢复数据
mysqlbinlog --no-defaults--start-datetime='2021-02-24 15:45:54' /opt/mysql-bin.000002 | mysql -uroot -p123456
#查看class表的数据
mysql -uroot -p123456 -e "select * from school.class;"

Summary:
If you restore from a breakpoint to restore all the data before a certain SQL statement, stop at the location node or time point of the statement.
If you restore a certain SQ statement and all the data after it, start from the location node or time point of the statement

mysqlbinlog --no-defaults --start-position='449' --stop-position='806' /opt/mysql-bin.000002 | mysql -uroot -p #恢复位置从449到806之间的数据