This article introduces the process of sending MySQL query results to the client through the question of "whether large queries will run out of memory". The knowledge points involved include: MySQL Server query result sending process (send while reading), MySQL Thread state Sending to client&Sending data, the impact of a large amount of data query on innoDB bufferPool, memory elimination algorithm LRU and innoDB improved LRU;
Question: Will a large query "run out" of the database host's memory?
For example: My host memory is only 100G , and now I want to perform a full table scan on a 200G large table, will the memory of the database host be used up?
——The answer is no ! For example, when logically backing up, isn't it just scanning the entire database? Therefore, it seems that it should be no problem to do a full table scan on a large table; but, what is the process like?
The general idea may already exist, that is, the idea of batch processing, multiple queries are loaded into the memory and then sent to the MySQL Client, let's see how the Server layer handles it;
Sending process of MySQL Server query results
Suppose we want to perform a full table scan on a 200G InnoDB table and save the scan results on the client , using a command like this:
mysql -h$host -P$port -u$user -p$pwd -e "select * from db1.t" > $target_fileInnoDB data is stored on the primary key index, so the full table scan actually directly scans the primary key index of table t; this query statement has no other judgment conditions, so each row found can be directly placed in the " result set ", and then returned to the client; so, where does this " result set " exist?
——Actually, the server does not need to save a complete result set; the process of fetching and sending data is as follows:
1. Obtain a line and write it into net_buffer; repeatedly acquire lines until net_buffer is full, and call the network interface to send it;
2. If the sending is successful, clear the net_buffer, then continue to fetch the next line and write it into net_buffer;
3. If sending If the function returns EAGAIN or WSAEWOULDBLOCK, it means that the local network stack (socket send buffer) is full and waits until the network stack is writable again before continuing to send;
From this process, you can see:
(1) MySQL query results are sent in batches. The size of a batch is the size of net_buffer (default 16K). Instead of checking all the results at once, they are all packaged and sent at one time. Instead, they are checked and sent in batches . sent ;
(2) Whether the socket send buffer is full will also affect the return of MySQL results; this means that if the client receives slowly , it will cause MySQL Server to enter the sending block because the results cannot be sent out , resulting in longer execution time of this transaction ;
That is to say, MySQL is "publish while reading", this concept is very important! This means that if the client receives slowly, the MySQL server will not be able to send the results, and the execution time of this transaction will be longer ;
MySQL thread status Sending to client&Sending data
Sending to client
Simulate a situation: deliberately let the client not read the content in the socket receive buffer, and then show processlist on the server to view the results;
It can be seen that the value of State is always "Sending to client", which means that the server-side network stack is full; suppose there is a business logic that is more complicated, and if the logic to be processed after each row of data is read is very slow, it will cause the client to It will take a long time for the terminal to fetch the next row of data, and the above situation may occur ;
Therefore, for normal online business, if a query does not return many results, it is recommended that you use the mysql_store_result interface (which is also the default method used by the MySQL client) to directly save the query results to the local memory; but If the amount of query data for an unconventional business is very large, such as executing a large query and causing the client to occupy nearly 20G of memory, the client's local cache cannot be stored at one time. In this case, the mysql_use_result interface needs to be used instead ;
Sending data
A state that looks very similar to "Sending to client" is "Sending data"; in fact, it has little to do with the MyAQL client receiving data;
In fact, the state change of a query statement looks like this:
- After the MySQL query statement enters the execution phase, first set the status to "Sending data";
- Then, send the column-related information (meta data) of the execution result to the client;
- Continue to execute the flow of the statement;
- After the execution is complete, set the state to an empty string;
In other words, "Sending data" does not necessarily mean "sending data", but may be at any stage in the executor process ; for example, you can construct a lock waiting scene, and you can see the status of Sending data;
That is to say, "Sending to client" will be displayed only when a thread is in the state of " waiting for the client to receive the result "; and if it is displayed as "Sending data", it means " executing ";
From the above analysis, we can see that for the server layer, the query results are sent to the client in batches ; therefore, scanning the entire table and returning a large amount of data from the query will not blow up the memory ;
Buffer Pool speeds up queries
The data pages of the memory are managed in BufferPool. In WAL, BufferPool plays a role in speeding up updates (write redolog first instead of random storage, reducing disk IO random writing pressure); in fact, BufferPool has a more important The role is to speed up the query ;
There may be doubts - if you read data after WAL, do you have to read the disk, or do you have to update the data from the redolog before you can return? In fact, it is not necessary. At this time, the latest data is in the memory (provided that the data in the memory is still there), and the previous data is still on the disk. The query can directly return the "dirty page" in the memory ;
The acceleration effect of Buffer Pool on queries depends on an important indicator: memory hit rate ;
You can check the current BP hit rate of a system in the result of show engine innodb status; in general, for an online system with stable service, if the response time meets the requirements, the memory hit rate must be above 99%;
For example, the hit rate in the picture is 99.1%;
If all the data pages required by the query can be obtained directly from the memory, that is the best, and the corresponding hit rate is 100%; but this is difficult to achieve in actual production, because the memory size is generally smaller than the total data size Yes; if a Buffer Pool is full and a data page needs to be read from the disk, an old data page must be eliminated ;
Supplement: redo log and change buffer
The two concepts of redo log and change buffer are indeed easy to confuse. Here we will add the timing of the change buffer: the change buffer mechanism will not always be applied. Only when the data page to be operated is not currently in memory, it needs to be read from the disk and loaded first. The change buffer is only useful for data pages. Its purpose is to avoid the need to read the data from the disk into the memory before updating, thereby reducing the pressure of random IO reads; when the data is really needed to be read, it will trigger the IO read And complete a merge in the memory, generate dirty pages, and record the update to the redo log (note that there is no need to flush the disk immediately, but a scheduled task is responsible for flushing the disk); for details, please refer to my previous article: "MySQL Practical Combat 45 Lectures " "——the last section of Study Notes 09 "Ordinary Index and Unique Index, Change Buffer and Redo Log" ;
Memory elimination algorithm LRU and innoDB improved LRU
InnoDB memory management uses the Least Recently Used (LRU) algorithm. The core of this algorithm is to eliminate the longest unused data;
Classical LRU Algorithm Model
The following figure is a basic model of a traditional LRU algorithm;
(1) Access data page P3, P3 exists in the cache, so P3 is moved to the front of the linked list;
(2) Access data page Px, Px does not exist in the cache, so a new data page needs to be applied for in the Buffer Pool Space for Px, but since the memory is full, it is necessary to clear the memory of the data page Pm at the end of the linked list, store the content of Px, and then move to the head of the linked list;
From the effect point of view, the data page Pm that has not been accessed for the longest time is eliminated ;
InnoDB's improvement of LRU under full table scan
This algorithm looks fine at first glance, but if you consider a full table scan, will there be any problems? ——Yes , the data in the memory is constantly being updated, and the overall memory hit rate becomes lower ;
For example, according to this algorithm, we want to scan a 200G table. If we scan according to this algorithm, all the data in the current BufferPool will be eliminated and stored in the content of the data page accessed during the scanning process. After accessing once, it is quickly eliminated by LRU due to insufficient memory space ;
This has a great impact on a library that is doing business services; you will see that the memory hit rate of BufferPool drops sharply, the disk pressure increases, and the response of SQL statements slows down;
Therefore, InnoDB cannot directly use this LRU algorithm; in fact, InnoDB has improved the LRU algorithm;
In the implementation of InnoDB, the entire LRU linked list is divided into young area and old area according to the ratio of 5:3 ;
(1) For the young area, when accessing an existing Page in the memory, the update rule is the same as the traditional LRU algorithm, and put it at the head of the linked list; (2) When
accessing a new data page that does not exist in the current linked list, it is still It is to eliminate the data page Pm at the end of the linked list, but the newly inserted data page Px is placed at LRU_old;
(3) The data page in the old area must make the following judgment every time it is accessed:
a. If the data page exists in the LRU list for more than 1000ms, move it to the head of the list; b
. If the data page exists in the LRU list for less than 1000ms, the position remains unchanged;
The time of 1000ms is controlled by the parameter innodb_old_blocks_time; its default value is 1000, in milliseconds;
This strategy seems to be tailored to handle operations like full table scans;
Still take the historical data table of 200G scanned just now as an example to see the operation logic of the improved LRU algorithm:
- During the scanning process, the data pages that need to be newly inserted will be placed in the old area first;
- There are multiple records in a data page, so this data page will be accessed multiple times; but due to sequential scanning, the speed of scanning data on the same Page is very fast, the data page is accessed for the first time and the last time it is accessed The access interval will not exceed 1000ms, so this data page will still be kept in the old area;
- Continue to scan subsequent data, the previous data page will not be accessed again, so there is no chance to move to the head of the linked list (that is, the young area), and will soon be eliminated by LRU due to limited cache space;
It can be seen that the biggest improvement of this strategy is to redefine the meaning of "latest use" : in addition to the traditional characteristics of the number of visits, it also requires a slightly longer time to be accessed, so as to prevent this kind of full table scan. The entire cache The situation of brushing off ;
The biggest benefit of the improved algorithm is that in the process of scanning this large table, although BufferPool is also used, it has no effect on the young area at all, thus ensuring the query hit rate of BufferPool in response to normal business;
Next article: To be determined
References for this chapter: 33 | If I check so much data, will the database memory explode?