MySql Operation and Maintenance ---008: Logs: error logs, binary logs, query logs, slow query logs, master-slave replication: Overview of the precautions, principles, and construction steps for changing the IP of a virtual machine

1. Log

1.1 Error log

The error log is one of the most important logs in MySQL. It records relevant information when mysqld starts and stops, as well as when any serious errors occur while the server is running . When any failure occurs in the database that prevents normal use, it is recommended to check this log first.

The log is enabled by default and is stored in the default directory /var/log/. The default log file name is mysqld.log. Check the log location:

#先登录mysql
mysql -uroot -p1234

#通过此系统变量查看日志文件的位置
show variables like '%log_error%';

#通过tail指令查看文件尾部的50行日志
tail -n 50 /var/log/mysqld.log	

1.2 Binary log

1.2.1 Introduction

The binary log (BINLOG) records all DDL (data definition language: create database...) statements and DML (data manipulation language: add, delete, modify) statements, but does not include data query (SELECT, SHOW) statements.

effect:

• ①. Data recovery in case of disaster;
  • Because the binary log records changes to databases, tables, and data. You only need to execute the statements here again to restore the data.
• ②. MySQL master-slave replication. In MySQL8 version
  • The underlying principle of master-slave replication is based on binary logs. See the next chapter for details.

The default binary log of mysql8.0 version is turned on, and the parameters involved are as follows:

#先登录mysql
mysql -uroot -p1234

#通过此系统变量来查看二进制日志相关的参数配置
show variables like '%log_bin%';

Parameter Description:

• log_bin: on means the binary log is on.

• log_bin_basename: The final generated binary log file is in /var/lib/mysqlthe directory. The file name is binlog, but there may be many log files, and binlog is just its prefix.

  • The base name (prefix) of the binlog log of the current database server. The specific binlog file name needs to be added with a number based on the basename (the number starts from 000001 and increases upward).
  • After the first log file is full or the format of the log is changed, it will open a new file again to write the log.

• log_bin_index: Binlog index file, which records the binlog files associated with the current server.

test:

Enter the /var/lib/mysql directory to see if there are any binary files.

#不登录mysql执行
cd /var/lib/mysql

#可以看到二进制日志文件和索引文件
ll

#查看索引文件：里面就记录了当前mysql数据库关联的日志文件有哪些
cat binlog.index

1.2.2 Format

MySQL server provides multiple formats to record binary logs. The specific formats and characteristics are as follows:

Log format	meaning
STATEMENT	Logging based on SQL statements records SQL statements, and the SQL that modifies the data will be recorded in the log file.
ROW	Row-based logging records data changes in each row. (default)
MIXED	It mixes two formats, STATEMENT and ROW. STATEMENT is used by default. In some special cases, it will automatically switch to ROW for recording.

Example: If we execute an update statement, the number of rows affected by this update statement is 5 rows

• STATEMENT: This update statement is recorded
• ROW: It records the five rows affected by the update statement. What the data content of each row looks like before the change and what it looks like after the change.

#先登录mysql
mysql -uroot -p1234

#通过此系统变量，查看当前mysql的版本中默认的日志格式是那个
show variables like '%binlog_format%';

If we need to configure the format of the binary log, we only need to configure the binlog_format parameter in /etc/my.cnf.

vim /etc/my.cnf 

#在这个文件中添加一行内容
binlog_format=STATEMENT

#重新启动mysql服务
systemctl restart mysqld.service

cd /var/lib/mysql

#可以看到此时重新生成了一个日志文件binlog.000005，原先是1~4。
#因为它的二进制日志格式改了，他不会再往原来的二进制日志文件写入了，而是写到一个新的日志文件中。
ll

Check this system variable again and find that the log format has been changed to STATEMENT.

#先登录mysql
mysql -uroot -p1234

#通过此系统变量，查看当前mysql的版本中默认的日志格式是那个
show variables like '%binlog_format%';

1.2.3 View

Since the log is stored in binary mode and cannot be read directly, it needs to be viewed through the binary log query tool mysqlbinlog. The specific syntax is:

#logfilename：二进制文件名
mysqlbinlog [ 参数选项 ] logfilename


参数选项：

	-d 		指定数据库名称，只列出指定的数据库相关操作。
	-o 		忽略掉日志中的前n行命令。
	-v 		将行事件(数据变更)重构为SQL语句
	-vv 	将行事件(数据变更)重构为SQL语句，并输出注释信息

Test: Next, let’s set up these two log formats and see what the difference between them is.

Case 1: The current log format is row

Step 1: Take the stu table under the db01 database as an example for demonstration.

客户端1：就是登录进mysql执行的命令

mysql -uroot -p1234

#当前的日志格式是row
show variables like '%binlog_format%';

use db01;

#查看db01数据库下面有哪些表
show tables;

#查看stu表下面有哪些数据
select * from stu;

Step 2: Execute the update statement

客户端1：

update stu set age = age +1 where id =1; 

Step 3: Check what is recorded in the binary log table

客户端2：就是没有登录进mysql执行的命令

cd /var/lib/mysql

ll

#因为二进制日志是第一个日志文件写满了之后会开启一个新的日志文件，所以只需要看最后一个日志文件即可。
#二进制文件不能直接查看使用cat显示的是乱码，需要通过mysqlbinlog 指令来查看
#日志里面是以行的格式显示的，所以看不到sql语句，我们还需要使用-v把它重构为sql语句才能看到
#效果：在日志的最后部分可以看到数据执行前后的变化
mysqlbinlog  -v binlog.000004;

It can be seen that the log format is row, and what is recorded is the data change of each row, what it was like before the change, and what it was like after the change.

Case 2: The current log format is STATEMENT

Step 1: Change the log format to STATEMENT. Just configure the binlog_format parameter in /etc/my.cnf.

vim /etc/my.cnf 

#在这个文件中添加一行内容
binlog_format=STATEMENT

#重新启动mysql服务
systemctl restart mysqld.service

cd /var/lib/mysql

#可以看到此时重新生成了一个日志文件binlog.000005，原先是1~4。
#因为它的二进制日志格式改了，他不会再往原来的二进制日志文件写入了，而是写到一个新的日志文件中。
ll

Step 2: Execute the previous update statement again

mysql -uroot -p1234

use db01;

update stu set age = age +1 where id =1; 

Step 3: View the contents of the newly generated binary log table again

#进入到二进制日志文件存放的位置
cd /var/lib/mysql

#可以看到此目录下有这个日志文件
ll

#查看此二进制日志文件
#不需要加-v，因为是STATEMENT它本身记录的就是sql语句
#效果：可以看到此时记录的就是sql语句而不是每一行的数据变化
mysqlbinlog  binlog.000005;

1.2.4 Delete

For relatively busy business systems, the binlog data generated every day is huge. If it is not cleared for a long time, it will occupy a lot of disk space. Logs can be cleared in the following ways:

instruction	meaning
reset master	Delete all binlog logs. After deletion, the log number will restart from binlog.000001
purge master logs to 'binlog.*****'	Delete all logs before the ***** number
purge master logs before 'yyyy-mm-dd hh24:mi:ss'	Delete all logs generated before the time point of "yyyy-mm-dd hh24:mi:ss"

You can also configure the expiration time of the binary log in the mysql configuration file. After setting it, the binary log will be automatically deleted when it expires.

mysql -uroot -p1234

#查看系统变量，在mysql命令行中执行
#单位是秒，默认过期时间为30天，到期之后会自动删除
show variables like '%binlog_expire_logs_seconds%';

test:

Client 1:

mysql -uroot -p1234

#删除000002之前的日志文件,不包含000002
purge master logs to 'binlog.000002';

Client 2:

cd /var/lib/mysql

#可以看到二进制日志文件和索引文件
ll

1.3 Query log

The query log records all operation statements of the client , while the binary log does not contain SQL statements for querying data. By default, query logging is not enabled .

mysql -uroot -p1234

#检查参数查看开关是否开启
#可以看到默认是关闭的以及日志文件所处位置和文件名
show variables like '%general%';

If you need to enable query logs, you can modify the MySQL configuration file /etc/my.cnf file and add the following content:

vim  /etc/my.cnf 

#该选项用来开启查询日志 ， 可选值 ： 0 或者 1 ； 0 代表关闭， 1 代表开启
general_log=1

#设置日志的文件名 ， 如果没有指定， 默认的文件名为 host_name.log
general_log_file=mysql_query.log

#重启mysql服务
systemctl restart mysqld.service

#查看这个目录下是否会生成此日志文件
cd /var/lib/mysql/

ll

After the query log is turned on, the mysql_query.log file will appear in the MySQL data storage directory, that is, the /var/lib/mysql/ directory. All subsequent additions, deletions, modifications and queries of the client will be recorded in this log file. After running for a long time, the log file will be very large. So this log file is not used, we can close it.

Test :

Client 1:

mysql -uroot -p1234

use db01;

#执行查询操作（前提是已经登录）
select * from stu;

#执行更新操作
update stu set age=100 where id=7; 

Client 2:

cd /var/lib/mysql/

#前提是已经进入到了这个目录，并且目录下有这个文件(上面已经配置过了)
#实时刷新此日志文件尾部的内容（tail查看文件尾部，-f表示实时刷新）
tail -f mysql_query.log

You can see that all DDL and DML operations are recorded in the log table.

1.4 Slow query log

The slow query log records the logs of all SQL statements whose execution time exceeds the setting value of the parameter long_query_time and the number of scan records is not less than min_examined_row_limit. It is not enabled by default . long_query_time defaults to 10 seconds, the minimum is 0, and the accuracy can be up to microseconds.

explain:

• The slow query log records SQL statements with relatively low execution efficiency and slow execution speed.
• It was explained before in SQL performance analysis of index.

If you need to enable slow query logs, you need to configure the following parameters in the MySQL configuration file /etc/my.cnf:

#慢查询日志：1代表开启
slow_query_log=1

#执行时间参数：表示执行时间超过2秒就是慢查询日志，此时慢查询日志文件就会记录这条sql.
long_query_time=2

By default, administrative statements are not logged, nor are queries that do not use indexes for lookups . This behavior can be changed using log_slow_admin_statements and log_queries_not_using_indexes, as described below.

explain:

• By vim /etc/my.cnf configuring these two parameters in the configuration file, you can change its default behavior.
• If log_slow_admin_statements =1 is added: it means that when we execute relatively slow management statements, they will also be recorded in the slow query log.
• If log_queries_not_using_indexes = 1 is added: it means that if a certain SQL statement does not use indexes and causes slow execution, it will also be recorded in the slow query log.
• Through slow query logs, you can locate those SQL statements with low execution efficiency, so that you can optimize such SQL statements.

#记录执行较慢的管理语句
log_slow_admin_statements =1

#记录执行较慢的未使用索引的语句
log_queries_not_using_indexes = 1

After all the above parameters are configured, you need to restart the MySQL server to take effect.

test:

Client 1:

mysql -uroot -p1234

#db01数据库下有tb_sku表，存放了1000万条记录
#电脑太卡，所以我没有创建tb_sku表,这里不在演示，只显示最终结果
use db01;

#不会记录
select * from tb_user limit 0,10; -- 这条SQL执行效率比较高, 执行耗时 0.01sec

#前面学习过SQL优化，分页查询越向后效率越低，此时超过2秒，会记录在慢查询日志中
select * from tb_user limit 1000000,10; -- 由于tb_sku表中, 预先存入了1000w的记录, count一次,耗时4.71sec（秒）

Client 2:

#配置慢查询日志
vim /etc/my.cnf 

#配置的内容
slow_query_log=1

long_query_time=2

# 重启Mysql服务器
systemctl restart mysqld

# 进入到此目录，发现会有一个后缀是-slow.log的日志文件
cd /var/lib/mysql/

ll

#实时刷新文件尾部的位置发现：
#记录了什么时间哪一个用户在哪一个主机上执行了什么样的sql语句
tail -f mysql8-slow.log

2. Master-slave replication

2.1 Overview

Master-slave replication refers to transmitting the DDL and DML operations of the master database to the slave database server through binary logs, and then re-executing these logs on the slave database (also called redo), so that the data of the slave database and the master database are maintained. Synchronize.

MySQL supports one master database to replicate to multiple slave databases at the same time. The slave database can also serve as the master database of other slave servers to achieve chain replication.

The advantages of MySQL replication mainly include the following three aspects:

• If there is a problem with the main library, you can quickly switch to the slave library to provide services.
• Realize the separation of reading and writing and reduce the access pressure of the main library.
  • Add, delete, and modify the master database, and query the slave database.
• The backup can be performed in the slave database to avoid affecting the master database service during the backup.
  • When backing up data, a global lock is added to prevent the backup data from being incomplete. At this time, the database is in a read-only state, and other clients can only query and cannot make additions, deletions or modifications.
  • With master-slave replication, you can perform backups in the slave database by simply locking the slave database. The master database can still perform operations such as additions, deletions, and modifications. Querying can still be done after adding a global lock to the slave database, but there may be a certain data delay during the data backup period because the slave database cannot execute the binary log synchronized from the master database during the backup period.
  • Solution: You can use the single-transaction parameter instead of adding a global lock for backup to ensure consistent backup of data ------ See the global lock for details.

2.2 Principle

The core of MySQL master-slave replication is the binary log . The specific process is as follows:

From the picture above, copying is divided into three steps:

1. When the Master database commits a transaction, it will record the data changes in the binary log file Binlog.

2. The slave library reads the binary log file Binlog of the main library and writes it to the slave library 中继日志 Relay Log.

   • The IOthread thread in the slave library: initiates a request to connect to the master database, and then reads the Binlog log in the master database. After reading and returning to the slave library, this thread will write the Binlog log to a log in the slave library itself ( Relay Log).

3. The slave redoes events in the relay log from the database, and the changes will reflect its own data.

   • The SQLthread thread in the slave library: reads the data in the relay log, and then reflects the data changes recorded in the relay log to the data changes in its own database, thereby ensuring the consistency of the master-slave data.

For example: after the main library executes the insert statement, it writes it to the binary log, and then it is read by the IOthread thread and then written to the relay log. Then the SQLthread thread will read the insert statement when reading the relay log. Then, then Come down and execute this insert in the slave library. At this time, the master-slave data is guaranteed to be consistent.

2.3 Setup

2.3.1 Environment preparation

After preparing the two servers, install MySQL on the two servers mentioned above and complete the basic initialization preparations (installation, password configuration, etc.) ( 注意要关闭防火墙). in:

• 192.168.10.200As the main server master

  • Host name: master

• 192.168.10.201As slave

  • Host name: slave
    

注意事项：

• The IP address configured first must be on the same network segment as the domain name resolution configured on the virtual machine. Only the last one can be different.
  
• If the ens33 network card is not displayed after restarting the virtual machine, you need to restart the network service. Of course, an error may be reported when starting the service, and the NetworkManger service needs to be closed.

  • ifconfig exception does not display ens33:
    

  • Ifconfig normally displays ens33:
    

#重启网络服务，可能会报错
service network restart

#如果报错：可能是和 NetworkManager 服务有冲突
#NetworkManager 是一个为系统提供检测和配置功能以便自动连接到网络的程序。包含一个守护程序、一个命令行界面（nmcli）和一个基于 curses 的界面（nmtui）。

#解决：直接关闭 NetworkManger 服务就好了，并且禁止开机启动，之后在重启网络服务
#关闭NetworkManger 服务
service NetworkManager stop

#禁止开机启动
chkconfig NetworkManager off

#此时再次启动网络服务就会成功了
service network restart

Finally, check the running status of the two mysql servers:

systemctl status mysqld

2.3.2 Main library configuration

1. Modify configuration file vim /etc/my.cnf

#mysql 服务ID，保证整个集群环境中唯一，取值范围：1 – 232-1，默认为1
server-id=1

#是否只读,1 代表只读, 0 代表读写
read-only=0

#以下2个不需要配置，表示创建的所有数据库都需要进行同步
#忽略的数据, 指不需要同步的数据库
#binlog-ignore-db=mysql

#指定同步的数据库
#binlog-do-db=db01

2. Restart the MySQL server

#如果没有报错代表配置文件中的配置成功
systemctl restart mysqld

3. 登录mysqlCreate a remote connection account and grant master-slave replication permissions

explain

• 'itcast'@'%': where itcastis the user name, @'% means that this user can access the current server on any host
• password is:Root@123456

#需要先登录mysql
mysql -uroot -p1234

#创建itcast用户，并设置密码，该用户可在任意主机连接该MySQL服务
#作用：在从库当中连接主库时的账号和密码。
CREATE USER 'itcast'@'%' IDENTIFIED WITH mysql_native_password BY 'Root@123456';


#为 'itcast'@'%' 用户分配主从复制权限
GRANT REPLICATION SLAVE ON *.* TO 'itcast'@'%';

4. View binary log coordinates through commands

show master status ;

Field meaning description:

• file: which log file to start pushing the log file from (write to that log file)
• position: From which position to start pushing logs
• binlog_ignore_db: Specify databases that do not need to be synchronized

• 主库配置完后就不要在执行DML增删改以及DDL语句了。

2.3.3 Slave library configuration

1. Modify configuration filevim /etc/my.cnf

#mysql 服务ID，保证整个集群环境中唯一，取值范围：1 – 2^32-1，和主库不一样即可
server-id=2


#是否只读,1 代表只读, 0 代表读写
#从库只需要做查询操作不需要做修改操作，所以设置为1也可以。
#这个选项仅仅代表是普通用户只读，如果这个用户具有超级管理员super的权限，那么他也是可以进行读写的。
read-only=1

#如果想要禁用超级管理员的读写功能，让它也变为只有读的功能，可以设置以下参数
super-read-only=1

2. Restart the MySQL service

systemctl restart mysqld 

3. 登录mysql, set the main library configuration

Now the main library and the slave library have no relationship and are not related, so next we need to set the relevant configuration of the main library in the slave library.

• SOURCE_HOST='192.168.200.200': What is the original host address, which is the ip of the main library
• SOURCE_USER='itcast': Connect to mysql corresponding to this IP address, then what is my username?
• SOURCE_PASSWORD='Root@123456': What is the password?
• SOURCE_LOG_FILE='binlog.000004': Which binary log file to start synchronization from
• SOURCE_LOG_POS=663: Indicates the position in this log file to start synchronization.

mysql -uroot -p1234

CHANGE REPLICATION SOURCE TO SOURCE_HOST='192.168.10.200', SOURCE_USER='itcast',SOURCE_PASSWORD='Root@123456', SOURCE_LOG_FILE='binlog.000012',SOURCE_LOG_POS=663;

The above is the syntax in 8.0.23. If mysql is a version before 8.0.23 , execute the following SQL:

CHANGE MASTER TO MASTER_HOST='192.168.10.201', MASTER_USER='itcast',MASTER_PASSWORD='Root@123456', MASTER_LOG_FILE='binlog.000012',MASTER_LOG_POS=663;

The difference between the two versions is that the parameter names are different: the currently used version 8.0.26 is compatible with the previous syntax, so you can execute either one

parameter name	meaning	Before 8.0.23
SOURCE_HOST	Main library IP address	MASTER_HOST
SOURCE_USER	Username to connect to the main library	MASTER_USER
SOURCE_PASSWORD	Password to connect to the main library	MASTER_PASSWORD
SOURCE_LOG_FILE	binlog log file name	MASTER_LOG_FILE
SOURCE_LOG_POS	binlog log file location	MASTER_LOG_POS

4. Enable synchronization operation

start replica ; #8.0.22之后

start slave ; #8.0.22之前

5. Check the master-slave synchronization status

show replica status ; #8.0.22之后
#表中的数据比较大展示出来的效果比较混乱，可以加上\G把每一列数据转化为每一行显示。
show replica status\G; 

show slave status ; #8.0.22之前

Effect : As long as the following two options are yes, it means that the master-slave replication is normal, IO-Running means whether the group of io threads is running normally, and SQL-Running means whether the group of sql threads are running normally.

Error situation : If it is a cloned virtual machine, the uuid value of mysql is the same. The uuid value of the mysql server of the slave virtual machine must be modified. It cannot be the same as the main library.

solve:

#不需要登录mysql
#修改此文件中的uuid值，随便修改一个字符
vim  /var/lib/mysql/auto.cnf

#重启mysql服务
systemctl restart mysqld 

Query the master-slave synchronization status again, this time both are yes (note that it is executed from the slave database)

#登录进mysql后执行，可以开启多个会话窗口这样就不需要多次登陆了
show replica status\G; 

2.3.4 Testing

First query the database status at this time:

show databases;

1. Create databases and tables on the main database 192.168.10.200, and insert data

create database db02;

use db02;

create table tb_user(

	id int(11) primary key not null auto_increment,
	name varchar(50) not null,
	sex varchar(1)
	
)engine=innodb default charset=utf8mb4;

insert into tb_user(id,name,sex) values(null,'Tom', '1'),(null,'Trigger','0'),(null,'Dawn','1');

2. Query data in the slave database 192.168.10.201 to verify whether the master and slave are synchronized

show datables;

use db02;

show tables;

select * from tb_user;

Notice:

• The master-slave replication we just demonstrated is a master-slave replication from the current position of the binary log backwards. If we want to synchronize the previous data to the slave database, then at this time we can first Export the data to a sql script, and then execute the sql script in the slave library. First ensure that the initial data of the main library and the slave library are consistent, and then synchronize from the current position.

2.4 Summary