The first layer: handle client connection, authorization authentication, security verification, etc.

The second layer: the server serverlayer, which is responsible for interpreting, analyzing, optimizing, and executing the operation engine for SQL.

The third layer: storage engine, responsible for the storage and extraction of data in MySQL.

2. Execution process of InnoDB transaction

We need to know that the server layer of MySQL does not manage transactions. Transactions are implemented by storage engines, and the storage engines that support transactions in MySQL are the most InnoDBwidely used, so the storage engines mentioned in the follow-up articles are all InnoDBbased.

The following is the execution flow of InnoDB transactions

2. Log classification

One, redo log (redo log)

1. The premise of existence

The redo log belongs to the transaction log of the MySQL storage engine InnoDB

MySQL data is stored on the disk, and disk IO operations are required every time the data is read or written. If the data is concurrent, the performance will be poor. To this end, MySQL provides an optimization method, the introduction of caching Buffer Pool. This cache contains the mapping of some data pages ( page) in the disk, so as to relieve the disk pressure of the database.

When reading data from the database, first read from the cache, if not in the cache, read from the disk and put it into the cache; when writing data to the database, write to the cache first, at this time the data page in the cache Data changes. This data page is called a dirty page . Buffer PoolAfter the data is modified, it will be periodically flushed to the disk according to the set update strategy. This process is called flushing dirty pages .

2. MySQL downtime

If the flushing of dirty pages has not been completed, MySQL may be restarted due to some reasons. At this time, the Buffer Poolmodified data has not been flushed to the disk in time, which will lead to data loss and cannot guarantee the durability of the transaction.

In order to solve this problem redo log, redo Log focuses on redo as its name suggests! It records the modification of each page in the database, not how a certain row or rows have been modified. It can be used to restore the submitted physical data page, and it can only be restored to the position of the last submission.

redo logThe (Write-Ahead Logging) technology is used WAL. The core of this technology is that before modifying the record, the log must be written first, and the log must be written to the disk before the transaction is submitted.

When the data is modified with the redo logBuffer Pool , the InnoDB engine will first write the update record in the redo log. When the data is being modified, when the transaction is committed, fsyncthe redo log is called to be flushed to disk. As for when the updated data files in the cache are flushed to the disk, it is processed asynchronously by the background thread.

Note : At this time, the transaction status of the redo log is that prepareit has not been submitted successfully. It bin logwill not be changed until the log is written to the disk commit, and the transaction will be considered to be submitted.

In this way, it does not matter even if MySQL crashes unexpectedly before flushing dirty pages, as long as the change records in the redo log are parsed and replayed during restart, and the disk is refreshed again.

3. The size of the redo log

Note : The redo log is full. Before erasing, you need to ensure that the data pages corresponding to the records to be erased in memory have been flushed to disk. During the period of erasing old records to make new space, no new update requests can be received, and the performance of MySQL will decline at this moment. Therefore, in the case of a large amount of concurrency, it is very important to reasonably adjust the file size of the redo log.

4、crash-safe

Because of the existence of the redo log, the InnoDB storage engine has the crash-safe capability. If MYSQL is down and restarts, the system will automatically check the redo log, and restore the modified data that has not been written to the disk from the redo log to MySQL.

When MySQL starts, whether it is a normal startup or a shutdown restart, it will always perform a recovery operation. First check the LSN in the data page. If the LSN is smaller than the LSN in the redo log (that is, the location where it is written), it means that the redo log contains unfinished operations on the data page, and then it will start from the latest checkpoint. Start synchronizing data.

Simple understanding, for example: the redo log LSNis 500, and the data page LSNis 300, indicating that some data has not been completely flushed to the disk before restarting, then the system will LSNreplay and flash the records with sequence numbers from 300 to 500 in the redo log.

Two, undo log (rollback log)

1. The premise of existence

undo logIt is also a transaction log belonging to the MySQL storage engine InnoDB.

undo logBelonging to the logical log, as its name mainly plays the role of rollback, it is the key to ensure the atomicity of the transaction. What is recorded is the state before data modification. During the process of data modification, a logical log opposite to the current operation will be recorded in it undo log.

Let's take a chestnut: If you update the name field of the ID=1 record, the original data of the name is Xiaofu, and now change the name to Programmer.

When a transaction executes update X set name = 程序员内点事 where id =1a statement, it will first undo logrecord a record of the opposite logic update X set name = 小富 where id =1, so that when some reasons cause the service exception transaction to fail, the data can be undo logrolled back to the state before the transaction is executed to ensure the integrity of the transaction.

        Some people may ask: A record in the same thing has been modified multiple times, so is it necessary to write the state of the data before modification every time undo log?

        The answer is no! undo logIt is only responsible for recording the original version of the data to be modified before the start of the transaction. When we modify this row of data again, the resulting modification record will be written to it, responsible for redo logcompleting undo logthe rollback, and redo logresponsible for completing the rollforward.

Two, rollback

        An uncommitted transaction, that is, the transaction is not executed commit. However, among the dirty pages modified in this transaction, some dirty blocks may have been flushed. If the database instance crashes and restarts at this time, you need to use rollback to undo the previous part of the dirty blocks that have been flushed from the disk.

3. Qianqiang

        A transaction that has not been fully committed, that is, the transaction has been executed commit, but only part of the data in the dirty pages modified in the transaction has been flushed, and the other part is still buffer poolin the cache. If the database instance is down and restarted at this time, it needs to be completed by rolling forward Incompletely committed transactions. Recover the data from the previous part due to downtime in the memory and flash it to the disk redo login .

Note: When the MySQL database is updated, the undo log, redo log, and bin log are all flushed before dirty pages are flushed, and mutual cooperation ensures that the data submitted by users is not lost to the greatest extent.

Three, bin log (archive log)

1. Basic introduction

Bin log is a logical log stored on disk in binary form at the database server layer (independent of the storage engine). The bin log records all DDL and DML operations of the database (excluding SELECT and SHOW)

Bin log is also called archive log, because it does not write and erase previous records in a loop like redo log, but keeps recording logs. If the size of a single log file exceeds max_binlog_size, a new file will be created to continue writing.

The content format of bin log is actually the reverse logic of executing SQL commands (similar to undo log).

Generally speaking, opening the bin log will set the expiration time for the log file (expire_logs_days variable, the unit is day)

2. Basic code demonstration
# 查看和binary log有关的变量

mysql> show variables like '%log_bin%';
+---------------------------------+------------------------------------------------------------+
| Variable_name                   | Value                                                      |
+---------------------------------+------------------------------------------------------------+
| log_bin                         | ON                                                         |
| log_bin_basename                | D:\Program Files (x86)\MySQL\Data 8.0\Data\IU077-bin       |
| log_bin_index                   | D:\Program Files (x86)\MySQL\Data 8.0\Data\IU077-bin.index |
| log_bin_trust_function_creators | OFF                                                        |
| log_bin_use_v1_row_events       | OFF                                                        |
| sql_log_bin                     | ON                                                         |
+---------------------------------+------------------------------------------------------------+

# 查看已有的binary log

mysql> show binary logs;
+------------------+-----------+-----------+
| Log_name         | File_size | Encrypted |
+------------------+-----------+-----------+
| IU077-bin.000038 |       157 | No        |
| IU077-bin.000039 |      1400 | No        |
| IU077-bin.000040 |       157 | No        |
| IU077-bin.000041 |       333 | No        |
| IU077-bin.000042 |       157 | No        |

# 查看日志过期天数

mysql> show variables like 'expire_logs_days';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| expire_logs_days | 0     |
+------------------+-------+
3. Master-slave synchronization

masterThe user executes DDLand operates in the main library DML, and writes the modified records sequentially bin log;

Connect to the Master from slavethe I/O thread of the library, and request to read positionthe log content at the specified location;

MasterAfter receiving the request from the library slave, positionpush the log content after the specified location, the name of the bin log file of the master library, and the position in the log to the slave library;

After the slave's I/O thread receives the data, it writes the received log content to the relay logend of the file in turn, and records the read main library bin log file name and location positioninto master-infothe file, so that it can be used for the next reading;

After the SQL thread of the slave detects relay logthe update of the content, it reads the log and parses it into an executable SQL statement, thus realizing the data consistency of the master-slave database;

4. Use bin log to restore data

The difference between bin log and redo log:

The levels are different: the redo log is implemented by the InnoDB storage engine, and the bin log is implemented by the MySQL server layer, but any storage engine in the MySQL database will generate a bin log for changes to the database.

The functions are different: redo log is used for collision recovery ( crash recovery), ensuring that MySQL downtime will not affect persistence; bin log is used for point-in-time recovery ( point-in-time recovery), ensuring that the server can recover data and master-slave replication based on point-in-time.

The content is different: the redo log is a physical log, and the content is based on disk pages Page; the content of the bin log is binary, which can be binlog_formatset according to the parameters.

The writing methods are different: the redo log is recorded in a circular writing method; the binlog is recorded in an appended manner. When the file size is greater than a given value, subsequent logs will be recorded in a new file.

The timing of flushing is different: the bin log is written when the transaction is committed; the redo log is written when the transaction starts.

Therefore, the functions of bin log and redo log do not conflict but complement each other. They need to be recorded at the same time to ensure that data will not be lost when the database is restarted due to downtime.

Four, relay log (relay log)

The format of the relay log log file is the same as that of the bin log log file. As can be seen from the above MySQL master-slave replication process, the relay log acts as a transfer function. The slave first reads the binary log data from the master database and writes it locally to the slave database. , and then read and parse the relay log into corresponding SQL commands by the SQL thread .

Five, slow query log (slow query log)

Slow query log ( slow query log): It is used to record query statements whose execution time exceeds the specified time in MySQL, which is often used in the process of SQL optimization. Through the slow query log, we can find out which query statements are inefficient and time-consuming.

For performance considerations, it is generally only enabled when troubleshooting slow SQL and debugging parameters. By default, the slow query log function is disabled. You can use the following command to check whether the slow query log is enabled:
# 慢查询日志开关

mysql> show variables like '%slow_query%';
+---------------------+----------------+
| Variable_name       | Value          |
+---------------------+----------------+
| slow_query_log      | ON             |
| slow_query_log_file | IU077-slow.log |
+---------------------+----------------+

# 超过多少时间才算慢查询语句

mysql> show variables like '%long_query_time%';
+-----------------+-----------+
| Variable_name   | Value     |
+-----------------+-----------+
| long_query_time | 10.000000 |
+-----------------+-----------+

Six, general query log (general query log)
General query log ( ): Used to record allgeneral query log user operations, including when the client connects to the server, all events sent by the client, and other events, such as service startup and shutdown, etc. The server writes the statements to the log file in the order in which they were received.SQLMySQLMySQL

Because the content of the general query log records is too detailed, the volume of the log file will be very large after it is turned on. Therefore, out of performance considerations, the log function is disabled by default, and it is usually necessary to obtain detailed logs when troubleshooting It will only be turned on temporarily.
# general log的开关

mysql> show variables like 'general_log';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| general_log   | OFF   |
+---------------+-------+

# general log的文件

mysql> show variables like '%general_log_file%';
+------------------+-----------+
| Variable_name    | Value     |
+------------------+-----------+
| general_log_file | IU077.log |
+------------------+-----------+

Seven, error log (error log)

The error log mainly records the time when the MySQL server starts and stops each time, as well as diagnostic and error information.

Not all error messages are recorded in the error log, but also include how MySQL starts the InnoDB table space file, how to initialize its own storage engine, initialize the buffer pool, etc., which are also recorded in the error log.
mysql> show variables like '%log_error%';
+----------------------------+----------------------------------------+
| Variable_name              | Value                                  |
+----------------------------+----------------------------------------+
| binlog_error_action        | ABORT_SERVER                           |
| log_error                  | .\IU077.err                            |
| log_error_services         | log_filter_internal; log_sink_internal |
| log_error_suppression_list |                                        |
| log_error_verbosity        | 2                                      |
+----------------------------+----------------------------------------+

References in this article:

http://t.csdn.cn/gHLJG

Introduction to seven types of logs in MySQL database

Table of contents

1. MySQL logical architecture and transaction execution process

First, the logical structure of the MySQL database

2. Execution process of InnoDB transaction

2. Log classification

One, redo log (redo log)

1. The premise of existence

2. MySQL downtime

3. The size of the redo log

4、crash-safe

Two, undo log (rollback log)

1. The premise of existence

Two, rollback

3. Qianqiang

Three, bin log (archive log)

1. Basic introduction

2. Basic code demonstration

3. Master-slave synchronization

4. Use bin log to restore data

Four, relay log (relay log)

Five, slow query log (slow query log)

Six, general query log (general query log)

Seven, error log (error log)

Guess you like