The following content is a summary of knowledge points based on "MySQL Performance Tuning and Architecture Design", interested friends can read this book, after all, learn and compare systems according to the book. Able to grasp the context of knowledge as a whole.
1. MySQL logic module composition
MySQL can be regarded as a two-tier architecture. The first layer is called SQL Layer. The main function of this part is to complete all the preparations before the mysql database system processes the underlying data, including permission judgment, sql analysis, and execution plan. Optimization, query cache processing, etc.; The second layer is the storage engine layer (Storage Engine Layer), this layer is the realization of the database system data access operation, which is completed by a variety of storage engines. 
It looks simple in structure, but each layer contains many small modules.
The modules included in the SQL Layer layer are:
- Initialization module
The initialization module is to perform various initialization operations on the entire system when the mysql server is started, such as the initialization of various buffers and cache structures and the application of memory space, the initialization setting of various system variables, and various storage engines. Initial settings, etc.
- Core API
The core API module is mainly to provide optimized implementation of some low-level operation functions that require very efficient, including the realization of various low-level data structures, the realization of special algorithms, character channeling, digital processing, etc., small file I/O, formatted output , And the most important part of memory management. All source codes of the core API module are concentrated in the mysys and strings folder.
- Network interaction module
The underlying network interaction module abstracts the interface api used by the underlying network interaction, and realizes the reception and transmission of the underlying network data to facilitate the invocation of other modules. And part of the maintenance of this program. All the source code is in the vio folder.
- Client & Server interactive protocol module
Any C/S structured software system will definitely have its own unique information interaction protocol, and MySQL is no exception. The Client & Server interaction protocol module part of MySQL realizes all the protocols in the interaction process between the client and MySQL. Of course, these protocols are built on the existing OS and network protocols. Such as TCP/IP, Unix Socket.
- User module
The functions implemented by the user module mainly include the user's login connection authority control and user authorization management. Just like the gate guard of MySQL, decide whether to "open the door" to visitors
-
Access control module
What if visitors can do whatever they want when they enter the door? For safety reasons, it must not be so arbitrary. At this time, the access control module needs to monitor every action of the guest in real time, and give different guests different permissions. The function realized by the access control module is to control the user's access to data according to the authorization information of each user in the user module and various constraints unique to the database itself. The user module and the access control module are combined to form the function of permission security management of the entire MySQL database system.
- Connection management, connection thread, thread management module
The connection management module is responsible for monitoring various requests to MySQL Server, accepting connection requests, and forwarding all connection requests to the thread management module. Each client request connected to MySQL Server will be assigned (or created) a connection thread for its separate service . The main job of the connection thread is to be responsible for the communication between MySQL Server and the client, and to accept the client's command requests. Pass the result information on the Server side. The thread management module is responsible for managing and maintaining these connection threads. Also includes thread creation, thread cache, etc.
-
Query parsing and forwarding module
In MySQL, we are accustomed to calling all the commands sent from the client to the server as a query. In MySQL Server, after the connection thread receives a query from the client, it will directly pass the query to the special class responsible for classifying various queries. Forward to the corresponding processing module, this module is the Query parsing and forwarding module. The main job is to perform semantic and grammatical analysis of query statements, then classify them according to different operation types, and then make targeted forwarding.
- Query Cache module
The Query Cache module is a very important module in MySQL. Its main function is to cache the returned result set of the query request of the select class submitted by the client to MySQL in the memory, and make a correspondence with a hash value of the query. After any data changes occur in the base table of the data fetched by the Query, MySQL will automatically invalidate the Cache of the query. In an application system with a very high ratio of reads and writes, the Query cache can significantly improve performance. Of course, his consumption of memory is also huge.
- Query optimizer module
Query optimizer is to optimize the query requested by the client. According to the query statement requested by the client, and some statistical information in the database, it analyzes on the basis of a series of algorithms to obtain an optimal strategy and tell the subsequent program How to get the result of this query statement.
-
Table Change Management Module
The table change management module is mainly responsible for completing some DML and DDL queries, such as: update, delete, insert, create table, alter table and other statements.
- Table maintenance module
Table status check, error repair, optimization and analysis are all things that the table maintenance module needs to do.
-
System State Management Module
The system status management module is responsible for returning various status data to the user when the client requests the system status, such as the show status commands and show variables commands commonly used by DBA. The results obtained are all returned by this module.
-
Table manager
This module is confused with the table change and table maintenance module above from the name, but its function is completely different from the table change and maintenance module. Everyone knows that every MySQL table has a table definition file, which is still a *.frm file. The main job of the table manager is to maintain these files and a cache. The main content of the cache is the structure information of each table. In addition, he also maintains table-level lock management.
-
Logging module
Mainly responsible for the log recording of the logic layer of the entire system level, including error log, binary log, show query log, etc.
-
Copy module
The replication module can be divided into two parts: the Master module and the Slave module. The Master module is mainly responsible for reading the binary log of the Master side in the Replication environment and interacting with the I/O thread of the Slave side. The Slave module does a little more than the Master module. It is mainly reflected in two threads in the system. One is responsible for requesting and receiving the binary log from the Master and writing to the I/O thread in the local relay log. The other is responsible for reading the relevant log files from the relay log, and then parsing them into the SQL thread that can be executed correctly on the Slave side and get exactly the same result as the Master side and then handed over to the SQL thread executed by the Slave.
-
Storage Engine Interface Module
The storage engine interface module is the most distinctive point of the MySQL database. Currently, among various database products, basically only MySQL can realize the plug-in management of its underlying data storage engine. This module is actually just an abstract class, but it is precisely because of its successful high-level abstraction of various database processing that it has achieved the characteristics of today's MySQL pluggable storage engine.
2. Work coordination of each module
Here comes the point. We use an example to illustrate how each module of the mysql system loves each other to complete a query that we think is very simple.
We start mysql, establish a connection with the client, request a query, get the returned result, and finally exit. Such a whole process for analysis.
The first step: After we execute the command to start the mysql system, the mysql initialization module reads the system parameters and command line parameters from the system configuration file, and initializes the entire system according to the parameters, such as applying for and assigning buffers, and initializing global variables , And various structures. At the same time, each storage engine is also started to perform their own initialization work. When the entire system is initialized, the connection management module will take over. The connection management module will start the listener program for processing client connection requests, including tcp/ip network monitoring, and unix sockets. At this time, mysql serve is basically started. , Ready to accept client requests.
Step 2: When the connection management module listens to the client's connection request (with the help of the relevant functions of the network interaction module), the two parties use the protocol defined by the Client & Server interaction protocol module to "greet" a few words, and the connection management module will connect The request is forwarded to the thread management module to request a connection thread.
Step 3: The thread management module then transfers the control to the connection thread module and tells the connection thread module that now my connection request has come over, and a connection needs to be established. Please handle it quickly. After receiving the connection request, the connection thread module will first check whether there is an idle connection thread cached in the current connection thread pool. If there is, it will take out one to connect with the client request, if there is no idle connection thread, then establish one The new connection thread requests a connection with the client. Of course, the connection thread module does not immediately take out a connection thread to connect with the client after receiving the connection request. Instead, it first checks the authorization by calling the user module. Only after the client request passes the authorization check, it will The client request is connected to the connection thread responsible for the request.
In MySQL, client requests are divided into two types, one is query, which needs to call Parser, which is the parsing of the Query parsing and forwarding module, to execute the request; the other is command, which can be executed without calling Parser. Request. If our initial configuration enables the function of Full Query Logging, the Query parsing and forwarding module will call the logging module to count the request into the log. Regardless of whether it is a Query-type request or a command-type request, it will be recorded into the log, so for performance reasons, the function of Full Query Logging is generally rarely turned on.
Step 4: When the client request and the connection thread "exchange secret codes (intercommunication protocol)" are connected, the connection thread starts to process various commands (or queries) sent by the client request and accept related requests. He forwards the received query statements to the Query parsing and forwarding module. The Query parser first performs basic semantic and grammatical analysis on the Query. Then, depending on the command type, some will be processed directly, and some will be distributed to other modules for processing.
If it is a Query type request, the control will be transferred to the Query parser. The Query parser first analyzes whether it is a select type query. If it is, it calls the query cache module to let it check whether the query is in the query cache. already exists. If so, directly return the data in the cache to the connection thread module. Then the data is output to the client through the thread connected with the client. If it is not a query type that can be cached, or if there is no query data in the cache, the query will continue to be passed back to the Query parser, and the Query parser will process it accordingly, and then distribute it to the relevant processing module through the Query Distributor.
Step 5: If the result of parsing by the parser is a select statement that is not cached, the control is transferred to the Optimizer, which is the Query optimizer module. If it is a DML or DDL statement, it will be transferred to the table change management module. If it is a query for updating statistics, detecting, repairing, and sorting, it will be handed over to the table maintenance module for processing, the copy related query will be transferred to the replication module for corresponding processing, and the query requesting status will be handed over to the status collection report Module. In fact, the table change management module is responsible for different DML and DDL by insert processor, delete processor, update processor, create processor, and alter processor according to the corresponding processing request.
Step 6: After each module receives the request from the Query parsing and distribution module, it will first check whether the connected user has the permission to access the control target table and target field through the access control module. If so, it will call the table management module Request the corresponding table and obtain the corresponding lock. The table management module will first see whether the table already exists in the table cache, if it has been opened, it will directly perform lock-related processing. If it is not in the cache, you need to open the table file to obtain the lock, and then hand over the opened table Table change management module.
Step 7: After the table change management module "gets" the opened table, it will determine the storage engine type and other related information of the table based on the related meta information of the table. According to the storage engine type of the table, submit a request to the storage engine interface module, call the corresponding storage engine implementation module, and perform corresponding processing.
However, for the table change management module, what is visible is only a series of "standard" interfaces provided by the storage engine interface module. The specific implementation of the underlying storage engine implementation module is transparent to the table change management module. He only needs to call the corresponding interface and specify the table type, and the interface module will call the correct storage engine according to the table type to perform corresponding processing.
Step 8: When a query or a command is processed (success or failure), the control will be returned to the connection thread module. If the processing is successful, the processing result (which may be a Result set, or an indicator of success or failure) is fed back to the client through the connection thread. If an error occurs during processing, the corresponding error message will also be sent to the client, and then the connection thread module will perform the corresponding clean-up work, and continue to wait for subsequent requests, repeat the above-mentioned process, or complete the client disconnect Connection request.
Step 9: If in the above process, the relevant module causes the data in the database to change, and MySQL calls the bin-log function, the corresponding processing module will also call the log processing module to update the corresponding change statement The form of the event is recorded in the binary log file specified by the relevant parameter.
In the content processing process of each of the above modules, the respective core computing processing functions are highly dependent on the entire MySQL API module, such as memory management, file I/O, number and string processing, etc.
The whole process above is shown in Figure 2-2: