The log is mysql an important part of the database, which records various status information during the operation of the database.

mysqlLogs mainly include error logs, query logs, slow query logs, transaction logs, and binary logs.

As development, we need to focus on the binary log ( binlog ) and the transaction log (including redo log and undo log ), which will be described in detail in the following sections of this article.

1、bin log

binlog It is used to record the write operation (excluding query) information performed by the database, and is stored in the disk in binary form. binlog Yes mysqllogical logging, and logging is done by the layer, the database Server using any storage engine will log.mysqlbinlog

Logical log : It can be simply understood that what is recorded is the sql statement.
Physical log : mysql Data is ultimately stored in data pages, and physical logs record data page changes.

binlog It is written by appending. You can max_binlog_size set the size of each file through parameters binlog. When the file size reaches a given value, a new file will be generated to save the log.

1.1, binlog usage scenarios

In practical applications, binlog there are two main usage scenarios, namely master-slave replication and data recovery .

Master-slave replication : Master open at the end binlog , and then binlogsend it to each Slave end, and the Slave end replays binlog to achieve master-slave data consistency.
Data Recovery : Recover data by using mysqlbinlog tools.

1.2. Timing of binlog flashing

For the InnoDB storage engine, the record is only recorded when the transaction is committed, biglog and the record is still in memory at this time, so biglogwhen is it flushed to the disk?

mysql The brush timing controlled by sync_binlog parameters biglog , the value range is 0-N:

0: No mandatory requirement, the system determines when to write to the disk;
1: Write to disk every commit time ;binlog
N: Every N transactions, will be binlog written to disk.

As can be seen from the above, sync_binlog the safest is to set is 1 , which is also MySQL 5.7.7the default value for later versions. However, setting a larger value can improve database performance. Therefore, in actual situations, the value can be appropriately increased, sacrificing certain consistency to obtain better performance.

1.3, binlog log format

binlog The log has three formats, namely STATMENT , , ROW and MIXED.

Before MySQL 5.7.7 , the default format is STATEMENT , MySQL 5.7.7 after, the default is ROW. The log format is binlog-format specified by.

STATMENT: SQL Statement-based replication ( statement-based replication, SBR ), each SQL statement that modifies data will be recorded binlog in .
- Advantages: no need to record the change of each line, reducing the amount of binlog log, saving IO, thus improving performance;
- Disadvantage: In some cases, the master-slave data will be inconsistent, such as executing sysdate(), sleep(), etc.
ROW: Row-based replication ( row-based replication, RBR ), does not record the context information of each SQL statement, only needs to record which data has been modified.
- Advantages: There is no problem that the invocation and triggering of stored procedures, functions, or triggers cannot be copied correctly under certain circumstances;
- Disadvantage: A lot of logs will be generated, especially when `alter table` will make the log skyrocket
MIXED: Based on the mixed copy of STATMENT the ROW two modes ( mixed-based replication, MBR ), the general copy is saved using the mode, and the mode cannot be copied using STATEMENT the mode save binlog for STATEMENT operations that cannot be copied ROW . binlog

2、redo log

2.1. Why do you need redo log

We all know that one of the four major characteristics of a transaction is persistence . Specifically, as long as the transaction is successfully submitted, the changes made to the database will be permanently saved, and it is impossible to return to the original state for any reason .

So how is durabilitymysql guaranteed ?

The easiest way is to flush all the data pages involved in the transaction to disk when the transaction is committed. But doing so will have serious performance problems, mainly in two aspects:

Because the disk interaction isInnodb performed in a unit, and a transaction is likely to modify only a few bytes in a data page, it is a waste of resources to flush the complete data page to the disk at this time!页
A transaction may involve modifying multiple data pages, and these data pages are not physically contiguous, and the write performance using random IO is too poor!

Therefore , it is mysql designed redo log , specifically, to only record what changes the transaction has made to the data page , which can perfectly solve the performance problem (relatively speaking, the file is smaller and it is sequential IO).

2.2, the basic concept of redo log

redo log It consists of two parts: one is the log buffer in memory ( redo log buffer ) and the other is the log file on disk ( redo logfile).

mysql Each time a DML statement is executed, records are written first redo log buffer, and then multiple operation records are written at a time at a later point in time redo log file. This technique of writing the log first and then writing to the disk is a technique that MySQLis often mentioned in this WAL(Write-Ahead Logging) article.

In a computer operating system, user space the buffer data in the user space ( ) generally cannot be directly written to the disk, and must pass through the operating system kernel space ( kernel space ) buffer ( OS Buffer ).

So redo log buffer writing redo logfile is actually writing first OS Buffer and then flushing it to it via a system call fsync() , the redo log fileprocess is as follows:

mysql It supports three redo log buffer writing redo log file timings, which can be innodb_flush_log_at_trx_commit configured by parameters. The meaning of each parameter value is as follows:

2.3, redo log record form

As mentioned earlier, redo log changes to data pages are actually recorded, and it is not necessary to save all such change records. Therefore, redo logthe implementation adopts a fixed-size, cyclic writing method. When writing to the end, it will return to the beginning to cyclically write the log. . As shown below:

At the same time, it is easy for us to know that in innodb, it is redo log necessary to flash the disk and 数据页 also need to flash the disk. redo logThe meaning of existence is mainly to reduce 数据页 the requirements for flashing.**

In the above figure, write pos it represents the (logical serial number) position of redo log the current record, and represents the corresponding ( logical serial number) position after the data page is changed and the record is flushed .LSNcheck pointredo logLSN

write poscheck point The part between to is an redo log empty part for recording new records; check point the part write pos between to is redo log the change record of the data page to be dropped. When write poscatching up check point , it will push check point forward first, vacate the position and record a new log.

At startup innodb , regardless of whether it was a normal shutdown or an abnormal shutdown last time, a recovery operation is always performed. Because redo logthe physical changes of the data pages are recorded, the recovery speed is much faster than the logical log (eg binlog ).

When restarting innodb , the data page in the disk will be checked first LSN , and if the data page is LSN smaller than the log , it will be restored LSN from the beginning.checkpoint

There is also a situation, in checkpoint the process of flushing the disk before the shutdown, and the flushing progress of the data page exceeds the flushing progress of the log page, then there will be more records in the data page LSN than in the LSNlog . The part of the progress will not be redone, because that itself represents something that has already been done and does not need to be redone.

2.4, the difference between redo log and binlog

It can be seen from the difference between binlog and : the log is only used for archiving, and it is incapable of relying only on it.redo logbinlogbinlogcrash-safe

But only redo log is not enough, because it redo log is InnoDBunique, and the records in the log will be overwritten after they are placed on the disk. Therefore, it is necessary to record binlogwith redo logboth to ensure that the data will not be lost when the database is down and restarted.

3、undo log

One of the four characteristics of database transactions is atomicity . Specifically, atomicity refers to a series of operations on the database that either all succeed or fail, and partial success is impossible .

In fact, the bottom layer of atomicityundo log is achieved through. undo logIt mainly records the logical changes of the data, such as a INSERT statement, corresponding to one DELETE , undo log and for each statement, corresponding to the UPDATE opposite , so that when an error occurs, the data state before the transaction can be rolled back.UPDATEundo log

At the same time, it undo log is also MVCCthe key to the implementation of (multi-version concurrency control).

MySQL - detailed explanation of binlog, redo log, undo log