The order in which a SQL statement is executed

1. Query statement

1.1 Overall process

Generally speaking, MySQL can be divided into two parts: the server layer and the storage engine layer.
The Server layer includes connectors, query caches, analyzers, optimizers, executors, etc., covering most of the core service functions of MySQL, as well as all built-in functions (such as date, time, math and encryption functions, etc.), all across storage engines All functions are implemented at this layer, such as stored procedures, triggers, views, etc.

The storage engine layer is responsible for data storage and retrieval. Its architectural model is plug-in, and supports multiple storage engines such as InnoDB, MyISAM, and Memory. The most commonly used storage engine is InnoDB, which has become the default storage engine since MySQL 5.5.5.

1.2 Process Segmentation

Take a query statement as an example

select * from table where id = 10

1.2.1 Connectors

In the first step, you will first connect to this database, and at this time, you will be greeted by the connector. The connector is responsible for establishing a connection with the client, obtaining permissions, maintaining and managing the connection. After completing the classic TCP handshake, the connector will start to authenticate your identity. At this time, the username and password you entered are used.

• If the username or password is incorrect, you will receive an "Access denied for user" error, and the client program will terminate execution.
• If the username and password authentication passes, the connector will check the permissions you have in the permissions table. After that, the permission judgment logic in this connection will depend on the permission read at this time. [This means that after a user successfully establishes a connection, even if you use the administrator account to modify the user's permissions, it will not affect the existing connection permissions. After the modification is completed, only newly created connections will use the new permission settings. 】

After the connection is completed, if you have no follow-up actions, the connection is idle, and you can see it in the show processlist command. The picture in the text is the result of show processlist, and the Command column in it shows the row of "Sleep", which means that there is an idle connection in the system.

If the client does not move for too long, the connector will automatically disconnect it. This time is controlled by the parameter wait_timeout, the default value is 8 hours.
If the client sends a request again after the connection is disconnected, it will receive an error reminder: Lost connection to MySQL server during query. At this time, if you want to continue, you need to reconnect and then execute the request.

In the database, a long connection means that after the connection is successful, if the client continues to request, the same connection will always be used. A short connection means that the connection is disconnected after a few queries are executed each time, and a new one is re-established for the next query.
The process of establishing a connection is usually more complicated, so try to reduce the action of establishing a connection during use, that is, use a long connection as much as possible.

But after all the long connections are used, you may find that sometimes the memory occupied by MySQL increases very quickly. The accumulation of long connections may cause the memory usage to be too large and be forcibly killed by the system (OOM). From the phenomenon, it can be seen that MySQL restarted abnormally.
The following two options can be considered to solve this problem:

1. Regularly disconnect long connections. After using it for a period of time, or the program judges that a large query that takes up memory has been executed, the connection is disconnected, and then the query needs to be reconnected.
2. If you are using MySQL 5.7 or later, you can reinitialize the connection resource by executing mysql_reset_connection after each large operation. This process does not require reconnection and re-authentication, but will restore the connection to the state just created.

1.2.2 Query cache

It's enough to get a general understanding, version 8.0 has been deleted.
After the connection is established, you can execute the select statement. Execution logic will come to the second step: query cache.

After MySQL gets a query request, it will first go to the query cache to see if this statement has been executed before. Previously executed statements and their results may be directly cached in memory in the form of key-value pairs. The key is the query statement, and the value is the query result. If your query can find the key directly in this cache, then the value will be returned directly to the client.

If the statement is not in the query cache, it will continue to the subsequent execution phase. After the execution is completed, the execution result will be stored in the query cache. You can see that if the query hits the cache, MySQL can directly return the result without performing subsequent complex operations, which is very efficient.

But the advantages of using query cache outweigh the disadvantages:
query cache invalidation is very frequent, as long as there is an update to a table, all query caches on this table will be cleared. For databases with high update pressure, the hit rate of the query cache will be very low. Unless your business is to have a static table, it will be updated once in a long time. For example, a system configuration table, then the query on this table is suitable for query caching

1.2.3 Analyzer

If the query cache is not hit, it is time to actually execute the statement. First of all, MySQL needs to know what you want to do, so it needs to parse the SQL statement.

The analyzer will first do " lexical analysis ". What you input is an SQL statement composed of multiple strings and spaces. MySQL needs to identify what the strings in it are and what they represent.

MySQL recognizes it from the keyword "select" you entered, which is a query statement. It also recognizes the string "T" as "table name T" and the string "ID" as "column ID".

After completing these identifications, it is necessary to do " syntax analysis ". According to the result of the lexical analysis, the syntax analyzer will judge whether the SQL statement you input satisfies the MySQL syntax according to the grammar rules.
If your statement is wrong, you will receive an error reminder of "You have an error in your SQL syntax".

1.2.4 Optimizer

After passing the analyzer, MySQL knows what you are going to do. Before starting to execute, it must be processed by the optimizer. The optimizer decides which index to use when there are multiple indexes in the table; or decides the connection order of each table when a statement has multiple table associations (join).

1.2.5 Actuator

MySQL knows what you want to do through the analyzer, and knows what to do through the optimizer, so it enters the executor stage and starts to execute the statement.
When starting to execute, you must first judge whether you have the permission to execute queries on this table. If not, an error of no permission will be returned.
If you have permission, open the table and continue execution. When opening the table, the executor will use the interface provided by the engine according to the engine definition of the table. Assuming the id field is not indexed

1. Call the InnoDB engine interface to fetch the first row of this table, and judge whether the ID value is 10, if not, skip it, and if it is, store this
   row in the result set;
2. Call the engine interface to fetch the "next row" and repeat the same judgment logic until the last row of the table is fetched.
3. The executor returns the record set composed of all rows satisfying the conditions in the above traversal process as a result set to the client.
   At this point, the execution of this statement is complete.

2. Modify the statement

2.1 Overall process

The overall process is actually similar to the query statement, mainly pay attention to two logs, redoLog and binLog.

1. It is the job of the connector to connect to the database before executing the statement.

2. As mentioned earlier, when a table is updated, the query cache related to this table will be invalidated, so this statement will clear all cached results on table T. This is why query caching is generally not recommended.

3. Next, the analyzer will know that this is an update statement through lexical and syntactic analysis. The optimizer decides to use the ID index.

4. Then, the executor is responsible for the specific execution, finds this row, and updates it.

Suppose there is an update statement

update T set c=c+1 where ID=2;

2.2 redoLog

redo log

Role: To ensure the durability of the transaction. To prevent dirty pages from being written to the disk at the time of the failure, when the mysql service is restarted, it will be redone according to the redo log, so as to achieve the feature of transaction persistence.

Content: The log in physical format records the modification information of the physical data page, and its redo log is sequentially written into the physical file of the redo log file.

There is a problem in MySQL. If each update operation needs to be written to the disk, and then the disk must find the corresponding record, and then update it, the IO cost and search cost of the whole process are very high.
The WAL technology often mentioned in MySQL, the full name of WAL is Write-Ahead Logging, its key point is to write the log first, and then write the disk, so as to solve the problem of high cost.

To be more specific, when there is a record that needs to be updated, the InnoDB engine will first write the record to the redo log and update the memory. At this time, the update is complete. At the same time, the InnoDB engine will update this operation record to the disk at an appropriate time, and this update is often done when the system is relatively idle.

InnoDB's redo log has a fixed size. For example, it can be configured as a set of 4 files, each with a size of 1GB. Then this "pink board" can record operations of 4GB in total. Start writing from the beginning, and then return to the beginning and write in a loop at the end.

With the redo log, InnoDB can guarantee that even if the database restarts abnormally, the previously submitted records will not be lost. This capability is called crash-safe.

2.3 binLog

Archive logs (binary logs)

Function: Used for replication. In master-slave replication, the slave library uses the binlog on the master library to replay to achieve master-slave synchronization. For point-in-time-based restores of databases.

Content: The log in logical format can be simply regarded as the sql statement in the executed transaction.

But it is not as simple as the SQL statement, but includes the reverse information of the executed SQL statement (addition, deletion and modification), which means that delete corresponds to delete itself and its reverse insert; update corresponds to the before and after execution of update Version information; insert corresponds to delete and insert itself.

Binlog has three modes: Statement (SQL statement-based replication), Row (row-based replication) and Mixed (mixed mode)

Looking at MySQL as a whole, there are actually two parts: one is the server layer, which mainly does things at the MySQL functional level; the other is the engine layer, which is responsible for storage-related specific matters. The redo log we talked about above is a log unique to the InnoDB engine , and the server layer also has its own log, called binlog (archived log).

2.4 Comparison of two logs

1. Redolog is specific to the InnoDB engine; binlog is implemented by the MySQL server layer and can be used by all engines.
2. Redolog is a physical log, which records "what changes have been made on a certain data page"; binlog is a logical log, which records the original logic of this statement, such as "add 1 to the c field of the row ID=2".
3. Redolog is written in a loop, and the space will always be used up; binlog can be written additionally. "Append write" means that after the binlog file is written to a certain size, it will switch to the next one, and will not overwrite the previous log.

2.5 Two-phase commit

The writing of redo log is split into two steps: prepare and commit, which is "two-phase commit".

2.5.1 Why is there a two-phase commit?

Simply put, both redo log and binlog can be used to represent the commit status of a transaction, and two-phase commit is to keep these two states logically consistent.

Let's think about a question, how to restore the database to the state of any second within half a month?

As we said before, binlog will record all logical operations, and it adopts the form of "append writing". If your DBA promises that it can be restored within half a month, then all the binlogs of the last half month will be saved in the backup system, and the system will regularly back up the entire database. The "regular" here depends on the importance of the system, which can be once a day or once a week.

When you need to restore to a specified second, for example, at 2 o'clock in the afternoon one day, you find that a table was accidentally deleted at 12 o'clock in the noon, and you need to retrieve the data, then you can do this:

• First, find the latest full backup. If you are lucky, it may be a backup from last night, and restore it to the temporary library from this backup;
• Then, starting from the time point of the backup, take out the backup binlog in sequence, and replay it to the moment before the table was accidentally deleted at noon.

In this way, your temporary database is the same as the online database before the accidental deletion, and then you can take the table data out of the temporary database and restore it to the online database as needed.

Since redo log and binlog are two independent logics, if two-phase commit is not used, either redo log is written first and then binlog is written, or the reverse order is adopted. Let's see what's wrong with these two methods.

Still use the previous update statement as an example. Assume that the value of field c in the current row with ID=2 is 0, and then suppose that during the execution of the update statement, after the first log is written, a crash occurs before the second log is written, what will happen?

• First write redo log and then write binlog. Assume that the MySQL process restarts abnormally when the redo log is written and the binlog is not yet written. As we said before, after the redo log is written, even if the system crashes, the data can still be recovered, so the value of c in this line after recovery is 1. But because the binlog crashed before it finished writing, this statement is not recorded in the binlog at this time. Therefore, when the log is backed up later, this statement does not exist in the saved binlog. Then you will find that if you need to use this binlog to restore the temporary library, because the binlog of this statement is lost, the temporary library will be updated less this time, and the value of c in the restored row is 0, which is different from the value of the original library different.

• Write binlog first and then redo log. If the crash occurs after the binlog is written, since the redo log has not been written yet, the transaction is invalid after the crash recovery, so the value of c in this line is 0. But the log "change c from 0 to 1" has been recorded in the binlog. Therefore, when binlog is used to restore later, one more transaction will appear, and the value of c in the restored line is 1, which is different from the value of the original database.

It can be seen that if the "two-phase commit" is not used, the state of the database may be inconsistent with the state of the library restored with its log.