Article Directory

MySQL log management, backup and recovery
- - Ready to work
MySQL log management
- One, MySQL common log types and opening
- Two, view the log status
MySQL backup and restore
Four, MySQL complete backup and recovery
- MySQL full backup
- Full backup and recovery
Five, MySQL incremental backup and recovery
- MySQL incremental backup
- MySQL incremental backup and recovery
at 302
at 430
at 461
at 526
at 609
at 742

MySQL log management, backup and recovery

Ready to work

Install MySQL database

Shell script one-click deployment-source code compile and install MySQL

MySQL log management

MySQL's default log storage location is /usr/local/mysql/data

There are two ways to open the log: through the configuration file or
through the command. Modifying the open log through the command is temporary, and it will be closed after the service is closed or restarted.

One, MySQL common log types and opening

vim /etc/my.cnf
[mysqld]
......

1. Error log
Used to record error messages that occur when MySQL is started, stopped, or running. It is enabled by default

指定日志的保存位置和文件名
log-error=/usr/local/mysql/data/mysql_error.log

2. General query log
Used to record all connections and statements of MySQL, it is closed by default

general_log=ON
general_log_file=/usr/local/mysql/data/mysql_general.log

3. Binary log (binlog)
Used to record all error messages sent when MySQL is started, stopped or running, it is closed by default

log-bin=mysql-bin				
或
log_bin=mysql-bin

4. Slow query log
It is used to record all statements whose execution time exceeds long_query_time seconds. You can find out which query statements have a long execution time for optimization. The default is off

slow_query_log=ON
slow_query_log_file=/usr/local/mysql/data/mysql_slow_query.log
long_query_time=5	   		

systemctl restart mysqld

Insert picture description here

Two, view the log status

1. Check whether the general query log is enabled

mysql -u root -p
show variables like 'general%';

2. Check whether the binary log is turned on

show variables like 'log_bin%';

3. Check whether the slow query day function is enabled

show variables like '%slow%';

View slow query time settings

show variables like 'long_query_time';

Set the method to start slow query in the database

set global slow_query_log=ON;
该方法重启服务失效

Insert picture description here

MySQL backup and restore

1. The importance of data backup

The main purpose of backup is disaster recovery.
In a production environment, data security is of paramount importance.
Any loss of data may have serious consequences

Reasons for data loss
1. Program error
2. Human operation error
3. Operation error
4. Disk failure
5. Disaster (fire, earthquake, theft, etc.)

Two, the classification of database backup

1. From a physical and logical perspective, backup can be divided into

1. Physical backup : backup of physical files (such as data files, log files, etc.) of the database operating system

Physical backup methods:
1. Cold backup (offline backup): is performed when the database is closed
2. Hot backup (online backup): the database is running and depends on the log file of the database
3. Warm backup: the database locks the table Backup operation in the state of (not writable but readable)

2. Logical backup : backup of database logical components (such as tables and other database objects)

2. From the perspective of database backup strategy, backup can be divided into

1. Full backup : make a complete backup of the database every time

A full backup is a backup of the entire database, database structure and file structure.
What is saved is the database at the time the backup was completed.
It is the basis of differential backup and incremental backup.
Equivalent to the cornerstone.

2. Differential backup : backup files that have been modified since the last full backup

3. Incremental backup : only the modified files after the last full backup or incremental backup will be backed up

Three, common backup methods

1. Physical cold standby

The database is closed during the backup, and the database files are directly packaged. The
backup speed is fast and the simplest recovery is to
close the MySQL database.
Use the tar command to directly package the database folder and
directly replace the existing MySQL directory.

2. The dedicated backup tool mydump or mysqlhotcopy

mysqldump Commonly used logical backup tool
MySQL comes with a backup tool that can back up MySQL.
You can export specified libraries and tables as SQL scripts.
Use the command mysql to import the backup data

mysqlhotcopy only has backup myisam and archive tables

3. Start the binary log for incremental backup

For incremental backup, you need to refresh the binary log

4. Third-party tool backup

Free MySQL hot backup software Percona XtraBackup

Four, MySQL complete backup and recovery

lab environment

Host	operating system	IP address	Required tools/software/installation package
MySQL	CentOS7	192.168.184.10	mysql-boost-5.7.20.tar.gz

mysql -u root -p
create database SCHOOL;
use SCHOOL;
create table if not exists CLASS1 (
id int(4) not null auto_increment,
name varchar(10) not null,
sex char(10) not null,
hobby varchar(50),
primary key (id));

insert into CLASS1 values(1,'user1','male','running');
insert into CLASS1 values(2,'user2','female','singing');

set password = password('123123');

Insert picture description here

MySQL full backup

The database of the InnoDB storage engine is stored on the disk as three files: db.opt (table attribute file), table name .frm (table structure file), table name .ibd (table data file).

1. Physical cold backup and recovery

systemctl stop mysqld
yum -y install xz

Compressed backup

tar Jcvf /opt/mysql_all_$(date +%F).tar.xz /usr/local/mysql/data/

Unzip and restore

tar Jxvf /opt/mysql_all_2020-11-22.tar.xz -C /usr/local/mysql/data

systemctl restart mysql

2. mysqldump backup and recovery
(1) Full backup of one or more complete libraries (including all tables)

mysqldump -u root -p[password] --databases library name 1 [library name 2]…> /backup path/backup file name.sql #exported is the database script file

Example:

mysqldump -uroot -p123123 --databases SCHOOL > /opt/SCHOOL.sql
mysqldump -uroot -p123123 --databases mysql SCHOOL > /opt/mysql-SCHOOL.sql

(2) Fully backup all the libraries in the MySQL server

mysqldump -u root -p[password] --all-databases> /backup path/backup file name.sql

Example:

mysqldump -u root -p123123 --all-databases > /opt/all.sql

Insert picture description here

(3) Full backup of some tables in the specified library

mysqldump -u root -p[password] library name [table name 1] [table name 2]…> /backup path/backup file name.sql

Example:

mysqldump -uroot -p123123 SCHOOL CLASS1 > 	/opt/SCHOOL_CLASS1.sql
#使用“-d”选项，说明只保存数据库的表结构
#不使用“-d”选项，说明表数据也进行备份

(4) View the backup file

grep -v "^--" /opt/SCHOOL_CLASS1.sql | grep -v "^/" | grep -v "^$"

Insert picture description here

Full backup and recovery

1. Restore the database

mysql -uroot -p123123 -e 'drop database SCHOOL;'

#"-E" option, used to specify the command to be executed after connecting to MySQL, it will automatically exit after the command is executed

mysql -uroot -p123123 -e 'SHOW DATABASES;'

mysql -uroot -p123123 < /opt/SCHOOL.sql
mysql -uroot -p123123 -e 'SHOW DATABASES;'

Insert picture description here

2. Restore the data table
When the backup file only contains the backup of the table and does not contain the statement of the created library, the library name must be specified when the import operation is performed, and the target library must exist.

mysql -uroot -p123123 -e 'drop table SCHOOL.CLASS1;'
mysql -uroot -p123123 -e 'show tables from SCHOOL;'

mysql -uroot -p123123 SCHOOL < /opt/SCHOOL_CLASS1.sql
mysql -uroot -p123123 -e 'show tables from SCHOOL;'

Insert picture description here

Five, MySQL incremental backup and recovery

MySQL incremental backup

1. Turn on the binary log function

vim /etc/my.cnf
[mysqld]
log-bin=mysql-bin
binlog_format = MIXED				
server-id = 1

# Binlog (binlog) has 3 different record formats: STATEMENT (based on SQL statements), ROW (based on rows), MIXED (mixed mode), the default format is STATEMENT

systemctl restart mysqld
ls -l /usr/local/mysql/data/mysql-bin.*

Insert picture description here

2. Complete backup of database or table every week

mysqldump -uroot -p123123 SCHOOL CLASS1 > /opt/SCHOOL_CLASS1_$(date +%F).sql
mysqldump -uroot -p123123 --all-databases SCHOOL > /opt/SCHOOL_$(date +%F).sql

Insert picture description here

3. Incremental backup operations can be performed every day to generate new binary log files (for example, mysql-bin.000002)

mysqladmin -uroot -p123123 flush-logs

Insert picture description here

4. Insert new data to simulate the increase or change of data

mysql -uroot -p123123
use SCHOOL;
insert into CLASS1 values(3,'user3','male','game');
insert into CLASS1 values(4,'user4','female','reading');

Insert picture description here

5. Generate a new binary log file again (for example mysql-bin.000003)

mysqladmin -uroot -p123123 flush-logs

#The database operation of the previous step 4 will be saved in the mysql-bin.000002 file, and then the database data will be changed again in the mysql-bin.000003 file

6. View the contents of the binary log file

cp /usr/local/mysql/data/mysql-bin.000002 /opt/
mysqlbinlog --no-defaults --base64-output=decode-rows -v /opt/mysql-bin.000002

#- -base64-output=decode-rows: Use the 64-bit encoding mechanism to decode and read in rows
#-v: Display detailed content

Insert picture description here

BEGIN
/!/;
#at 302
#210129 0:39:12 server id 1 end_log_pos 430 CRC32 0x2c164d0a Query thread_id=10 time=0 error_code=0
use SCHOOL/!/;
SET TIMESTAMP=1611851952/!/;
insert into CLASS1 values(3,‘user3’,‘male’,‘game’)
/!/;
#at 430
#210129 0:39:12 server id 1 end_log_pos 461 CRC32 0x225bb461 Xid = 76
COMMIT/!/;
#at 461
#210129 0:39:13 server id 1 end_log_pos 526 CRC32 0xe5abe22c Anonymous_GTID last_comd=1 sequence_number=2 rbr_only=no
SET @@SESSION.GTID_NEXT= ‘ANONYMOUS’/!/;
#at 526
#210129 0:39:13 server id 1 end_log_pos 609 CRC32 0x2cfb793b Query thread_id=10 time=0 error_code=0
SET TIMESTAMP=1611851953/!/;
BEGIN
/!/;
#at 609
#210129 0:39:13 server id 1 end_log_pos 742 CRC32 0x7ea13a1a Query thread_id=10 time=0 error_code=0
SET TIMESTAMP=1611851953/!/;
insert into CLASS1 values(4,‘user4’,‘female’,‘reading’)
/!/;
#at 742
#210129 0:39:13 server id 1 end_log_pos 773 CRC32 0x11b21cd0 Xid = 77
COMMIT/!/;

MySQL incremental backup and recovery

1. General recovery
(1) Simulate the recovery steps of lost and changed data

mysql -uroot -p123123
use SCHOOL;
delete from CLASS1 where id=3;
delete from CLASS1 where id=4;
select * from CLASS1;
quit

mysqlbinlog --no-defaults /opt/mysql-bin.000002 | mysql -uroot -p123123
mysql -uroot -p123123 -e "select * from SCHOOL.CLASS1;"

Insert picture description here

2. Simulate the recovery steps of all lost data (basically the same, pay attention to the log date)

mysql -uroot -p123123
use SCHOOL;
drop table CLASS1;
quit

mysql -uroot -p123123 SCHOOL < /opt/SCHOOL_CLASS1_2021-01-29.sql
mysqlbinlog --no-defaults /opt/mysql-bin.000002 | mysql -uroot -p123123
mysql -uroot -p123123 -e "select * from SCHOOL.CLASS1;"

2. Breakpoint recovery

mysqlbinlog --no-defaults --base64-output=decode-rows -v /opt/mysql-bin.000002

Use the same 64-bit encoding mechanism to decode and read the details of the binary file 000002 line by line as just

BEGIN
/!/;

at 302

#210129 0:39:12 server id 1 end_log_pos 430 CRC32 0x2c164d0a Query thread_id=10 time=0 error_code=0
use SCHOOL/!/;
SET TIMESTAMP=1611851952/!/;
insert into CLASS1 values(3,‘user3’,‘male’,‘game’)
/!/;

at 430

#210129 0:39:12 server id 1 end_log_pos 461 CRC32 0x225bb461 Xid = 76
COMMIT/!/;

at 461

#210129 0:39:13 server id 1 end_log_pos 526 CRC32 0xe5abe22c Anonymous_GTID last_comd=1 sequence_number=2 rbr_only=no
SET @@SESSION.GTID_NEXT= ‘ANONYMOUS’/!/;

at 526

#210129 0:39:13 server id 1 end_log_pos 609 CRC32 0x2cfb793b Query thread_id=10 time=0 error_code=0
SET TIMESTAMP=1611851953/!/;
BEGIN
/!/;

at 609

#210129 0:39:13 server id 1 end_log_pos 742 CRC32 0x7ea13a1a Query thread_id=10 time=0 error_code=0
SET TIMESTAMP=1611851953/!/;
insert into CLASS1 values(4,‘user4’,‘female’,‘reading’)
/!/;

at 742

#210129 0:39:13 server id 1 end_log_pos 773 CRC32 0x11b21cd0 Xid = 77
COMMIT/!/;

(1) Restore based on location
# Only restore the data before the operation ID "609", that is, the data of "user4" is not restored

mysqlbinlog --no-defaults --stop-position='609' /opt/mysql-bin.000002 | mysql -uroot -p

Example:

mysql -uroot -p123123 -e "select * from SCHOOL.CLASS1;"
mysql -uroot -p123123 -e "truncate table SCHOOL.CLASS1;"
mysql -uroot -p123123 -e "select * from SCHOOL.CLASS1;"
mysqlbinlog --no-defaults --stop-position='609' /opt/mysql-bin.000002 | mysql -uroot -p
mysql -uroot -p123123 -e "select * from SCHOOL.CLASS1;"

Insert picture description here

#Recover only the data of "user4", skip the data recovery of "user3", after 609 only the fourth record

Example:

mysql -uroot -p123123 -e "select * from SCHOOL.CLASS1;"
mysqlbinlog --no-defaults --start-position='609' /opt/mysql-bin.000002 | mysql -uroot -p123123
mysql -uroot -p123123 -e "select * from SCHOOL.CLASS1;"

Insert picture description here

(2) Restore based on time point
# Only restore the data before 0:39:13, that is, do not restore the data of "user4"

例：先清空表CLASS1,方便实验

mysql -uroot -p123123 -e "truncate table SCHOOL.CLASS1;"
mysql -uroot -p123123 -e "select * from SCHOOL.CLASS1;"

mysqlbinlog --no-defaults --stop-datetime='2021-01-29 0:39:13' /opt/mysql-bin.000002 |mysql -uroot -p123123
mysql -uroot -p123123 -e "select * from SCHOOL.CLASS1;"

Insert picture description here

#Recover only the data of "user4", skip the data recovery of "user3" (basically the same)

mysqlbinlog --no-defaults --start-datetime='2021-01-29 0:39:13' /opt/mysql-bin.000002 |mysql -uroot -p