MySQL Basics (22) Logical Architecture

1. Logical architecture analysis

1.1 Layer 1: Connection layer

Before the system (client) accesses MySQLthe server, the first thing it does is establish TCPa connection.

After the three-way handshake establishes the connection successfully, MySQLthe server TCPperforms identity authentication and authority acquisition on the transferred account password.

• If the username or password is incorrect, you will receive an Access denied for user error and the client program will end execution.
• If the user name and password authentication is passed, the permissions owned by the account and the connection will be found from the permission table. Subsequent permission judgment logic will depend on the permissions read at this time.

TCPAfter the connection receives the request, it must be allocated to a thread dedicated to interacting with this client. So there will be a thread pool to carry out the subsequent processes. Each connection obtains a thread from the thread pool, eliminating the overhead of creating and destroying threads.

1.2 Layer 2: Service Layer

• SQL Interface: SQL interface

  • Receive the user's SQL command and return the results that the user needs to query. For example, SELECT ... FROM is to call SQL Interface
  • MySQL supports multiple SQL language interfaces such as DML (data manipulation language), DDL (data definition language), stored procedures, views, triggers, and custom functions.

• Parser: parser

  • Perform syntax analysis and semantic analysis on SQL statements in the parser. Decompose the SQL statement into a data structure and pass this structure to subsequent steps. The subsequent delivery and processing of SQL statements is based on this structure. If an error is encountered during the decomposition, it means that the SQL statement is unreasonable.
  • When the SQL command is passed to the parser, it will be verified and parsed by the parser, and created 语法树, and the query syntax tree will be enriched according to the data dictionary 验证该客户端是否具有执行该查询的权限. After creating the syntax tree, MySQL will also optimize the syntax of the SQL query and rewrite the query.

• Optimizer: query optimizer

  • After the SQL statement is parsed and before the query, the query optimizer is used to determine the execution path of the SQL statement and generate a 执行计划.
  • This execution plan indicates whether the query should 使用哪些索引be performed (full table retrieval or index retrieval), and the order of connections between tables. Finally, the method provided by the storage engine will be called according to the steps in the execution plan to actually execute the query and return the query results. to users.
  • It uses the " 选取-投影-连接" strategy for querying. For example:

  SELECT id,name FROM student WHERE gender = '女';
  

  This SELECT query is first performed based on the WHERE statement 选取, instead of querying all tables and then performing gender filtering. This SELECT query first performs attributes based on id and name 投影, instead of extracting all attributes and then filtering, and 连接combines these two query conditions to generate the final query result.

• Caches & Buffers: Query caching component

  • MySQL maintains some caches and buffers internally. For example, Query Cache is used to cache the execution results of a SELECT statement. If the corresponding query results can be found in it, then there is no need to go through the entire process of query parsing, optimization, and execution. Instead, the query cache is used to cache the execution results of a SELECT statement. The results are fed back to the client.
  • This caching mechanism is composed of a series of small caches. For example, table cache, record cache, key cache, permission cache, etc.
  • This query cache can be used in 不同客户端之间共享.
  • Starting with MySQL 5.7.20, the query cache is deprecated and is available in MySQL 8.0中删除.

1.3 Layer 3: Engine layer

The plug-in storage engine layer (Storage Engines) is truly responsible for the storage and retrieval of data in MySQL, and performs operations on the underlying data maintained at the physical server level . The service layer communicates with the storage engine through APIs.

1.4 Summary

Simplified into a three-layer structure:

1. Connection layer: The client and server establish a connection, and the client sends SQL to the server;

2. SQL layer (service layer): performs query processing on SQL statements; it has nothing to do with the storage method of database files;

3. Storage engine layer: Deals with database files and is responsible for data storage and reading.

2. SQL execution process

2.1 SQL execution process in MySQL

MySQL query process:

1. Query cache : If the Server finds this SQL statement in the query cache, it will directly return the result to the client; if not, it will enter the parser stage. It should be noted that because query caching is often inefficient, this feature was abandoned after MySQL 8.0.

Query caching caches query results in advance so that you can get the results directly without executing them next time. It should be noted that the query cache in MySQL does not cache the query plan, but the corresponding results of the query. This means that the query matches 鲁棒性大大降低only 相同的查询操作才会命中查询缓存. Any difference in characters between the two query requests (for example: spaces, comments, case) will cause the cache to miss. Therefore, MySQL's query cache hit rate is not high.

At the same time, if the query request contains certain system functions, user-defined variables and functions, and some system tables, such as tables in the mysql, information_schema, and performance_schema databases, then the request will not be cached.

In addition, since it is a cache, there it is 缓存失效的时候. MySQL's cache system will monitor each table involved. As long as the structure or data of the table is modified, such as using the INSERT, UPDATE, DELETE, TRUNCATE TABLE, ALTER TABLE, DROP TABLEor DROP DATABASEstatements on the table, all cached queries using the table will become Invalidate and remove from cache! For 更新压力大的数据库, the query cache hit rate will be very low.

2. Parser : Perform syntactic analysis and semantic analysis of SQL statements in the parser.

The analyzer does " 词法分析" first. What you input is an SQL statement composed of multiple strings and spaces. MySQL needs to identify what the strings in it are and what they represent. MySQL recognizes from the "select" keyword you entered that this is a query statement. It also needs to recognize the string "T" as "table name T" and the string "ID" as "column ID".

Next, do " 语法分析". According to the results of lexical analysis, the syntax analyzer (such as Bison) will determine whether the SQL statement you entered is valid according to the grammatical rules 满足 MySQL 语法. If the SQL statement is correct, a syntax tree will be generated.

3. Optimizer : The optimizer will determine the execution path of the SQL statement, such as whether it is based on 全表检索, or based on, 索引检索etc. In the query optimizer, it can be divided into 逻辑查询optimization phase and 物理查询optimization phase.

4. Executor : Before execution, it is necessary to determine whether the user is 具备权限. If not, a permission error will be returned. If you have permission, execute the SQL query and return the results. In versions below MySQL 8.0, if the query cache is set, the query results will be cached.

The flow of SQL statements in MySQL is: SQL statement → query cache → parser → optimizer → executor.

3. Database buffer pool (buffer pool)

InnoDBThe storage engine manages storage space in units of pages. The addition, deletion, modification and query operations we perform are essentially accessing pages (including reading pages, writing pages, creating new pages, etc.). Disk I/O takes a lot of time, and operations in memory are much more efficient. In order to make the data in the data table or index available to us at any time, the DBMS will apply for it before actually accessing the page 占用内存来作为数据缓冲池. Pages on disk Buffer Poolcan be accessed only after they are cached in memory.

The advantage of this is that disk activity can be minimized, thus 减少与磁盘直接进行 I/O 的时间. You know, this strategy is crucial to improving the query performance of SQL statements. If the indexed data is in the buffer pool, the cost of access will be much reduced.

3.1 Buffer pool vs query cache

1. Buffer Pool

From the figure, you can see that the InnoDB buffer pool includes data pages, index pages, insertion buffers, lock information, adaptive Hash and data dictionary information, etc.

Caching principles:

" 位置 * 频次"This principle can help us optimize I/O access efficiency.

First of all, location determines efficiency. The buffer pool is provided so that data can be directly accessed in memory.

Secondly, frequency determines the order of priority. Because the size of the buffer pool is limited, for example, the disk has 200G, but the memory is only 16G, and the buffer pool size is only 1G, it is impossible to load all the data into the buffer pool. At this time, priority order is involved 优先对使用频次高的热数据进行加载.

2. Query cache

The query cache is set up in advance 查询结果缓存so that the results can be obtained directly without execution next time. It should be noted that the query cache in MySQL does not cache the query plan, but the corresponding results of the query. Because the hit conditions are strict, and as long as the data table changes, the query cache will become invalid, so the hit rate is low.

3.2 How to read data from the buffer pool

The buffer pool manager will try to save frequently used data. When the database reads a page, it will first determine whether the page is in the buffer pool. If it exists, it will be read directly. If it does not exist, it will be read through memory or The disk stores the page in the buffer pool and then reads it.

3.3 View/set the size of the buffer pool

View buffer pool size

show variables like 'innodb_buffer_pool_size';

Set buffer pool size

set global innodb_buffer_pool_size = 268435456;

or

[server] 
innodb_buffer_pool_size = 268435456

3.4 Multiple Buffer Pool instances

[server] 
innodb_buffer_pool_instances = 2

How to check the number of buffer pools

show variables like 'innodb_buffer_pool_instances';

Buffer PoolThe actual memory space occupied by each instance

innodb_buffer_pool_size/innodb_buffer_pool_instances