text

As a back-end development, the most commonly used daily operations of the database are write operations and read operations. We will talk about read operations below. In this category, we mainly look at why SQL slows down during write operations.

brush dirty pages

The definition of a dirty page is as follows: when the memory data page is inconsistent with the disk data page, the memory data page is called a dirty page.

Then why do dirty pages appear, and how can brushing dirty pages cause SQL to slow down? Then we need to see what the process of writing is like.

For a SQL write operation, the execution process involves several situations such as writing logs, memory, and synchronizing disks.

A log file is mentioned here, that is, redo log, which is located at the storage engine layer and is used to store physical logs. During the write operation, the storage engine (Innodb is discussed here) will write the record to the redo log and update the cache, so that the update operation is completed. Subsequent operations The storage engine will synchronize the operation records to the disk at an appropriate time.

Seeing this, you may have a question. Isn’t the redo log a log file? The log file is stored on the disk. Isn’t it very slow when writing?

In fact, the process of writing redo logs is to write to the disk sequentially. The sequential writing of disks reduces the time for seeking, and the speed is much faster than random writing (similar to the storage principle of Kafka), so the writing speed of redo logs is very fast.

Ok, let's go back to the original question, why dirty pages occur, and why dirty pages slow down SQL. If you think about it, the size of the redo log is fixed, and it is written in a loop. In a high-concurrency scenario, the redo log is quickly filled, but the data is too late to be synchronized to the disk. At this time, dirty pages will be generated and subsequent write operations will be blocked. SQL execution will naturally be slower.

Lock

Another situation where SQL is slow during write operations is that locks may be encountered, which is easy to understand. For example, you share a room with someone else, there is only one bathroom, and you both want to go at the same time, but the other person is a little bit ahead of you. Then at this time you can only go in after the other party comes out.

Corresponding to Mysql, when the row to be changed by a certain SQL is just locked, then the follow-up operation can only be performed after the lock is released.

But there is another extreme situation, your roommate has been occupying the bathroom, so what should you do at this time, you can't pee your pants, how embarrassing. Corresponding to Mysql, there is a deadlock or lock waiting situation. How to deal with it at this time?

Mysql provides a way to view the current lock situation:

By executing the statements in the graph on the command line, you can view the current transaction status. Here are some important parameters in the query results:

If the current transaction waits too long or a deadlock occurs, you can release the current lock by " kill thread ID ".

The thread ID here refers to the trx_mysql_thread_id parameter in the table.

read operation

After talking about the write operation, you may be relatively more familiar with the read operation. The problem of slow SQL leading to slow read operations is often involved in work.

slow query

Before talking about the reasons for the slow read operation, let's take a look at how to locate the slow SQL. There is something called slow query log in Mysql , which is used to record SQL statements that exceed a specified time. It is disabled by default, and the slow query log can only be enabled through manual configuration for positioning.

The specific configuration method is as follows:

Check the status of the current slow query log:

Enable slow query log (temporary):

Note that the slow query log is only temporarily enabled here, and it will become invalid after restarting mysql. It can be configured in my.cnf to make it permanent.

reason for existence

Now that we know how to check the slow execution of SQL, let's go on to see why the slow query is caused by the read operation.

(1) Missing index

One of the reasons for the slow SQL query is that the index may not be hit. There are already a lot on the Internet about why the use of the index can make the query faster and the precautions when using it, so I won’t go into details here.

(2) Dirty page problem

The other is the dirty page brushing situation we mentioned above, but it is different from the write operation in that dirty pages are brushed during reading.

Are you a little confused, don't worry, listen to me:

In order to avoid increasing the IO overhead when accessing the disk every time when reading and writing data, the Innodb storage engine improves the reading and writing speed by loading the corresponding data pages and index pages into the buffer pool (buffer pool) of the memory. Then keep the cached data in the buffer pool according to the least recently used principle.

Then when the data page to be read is not in the memory, it is necessary to apply for a data page in the buffer pool, but the data page in the buffer pool is certain. Data pages are evicted from memory. But if dirty pages are eliminated, then the dirty pages need to be flushed to disk for reuse.

You see, it's back to the situation of flushing dirty pages, and you can understand the slowness of the read operation, right?

nip in the bud

Knowing the reason, how can we avoid or alleviate this situation?

First look at the case of missing indexes:

I don't know if you have the habit of using explain in Mysql. Anyway, I use it every time to check the current SQL hit index. Avoid some unknown hidden dangers it brings.

Here is a brief introduction to its usage. By adding explain before the executed SQL, you can analyze the current SQL execution plan:

The summary description of the fields corresponding to the results after execution is shown in the figure below:

Here you need to focus on the following fields:

1、type

Indicates how MySQL finds the desired row in the table. The commonly used types are: ALL, index, range, ref, eq_ref, const, system, NULL These types from left to right, the performance gradually improves.

ALL: Mysql traverses the entire table to find matching rows;
index: The difference from ALL is that the index type only traverses the index tree;
range: only retrieves rows in a given range, using an index to select rows;
ref: Indicates the connection matching conditions of the above tables, which columns or constants are used to find the value on the index column;
eq_ref: Similar to ref, the difference is whether it is a unique index or not. For each index key value, only one record in the table matches. In simple terms, the primary key or unique key is used as the association condition in multi-table joins;
const, system: These types of access are used when Mysql optimizes a certain part of the query and converts it into a constant. If the primary key is placed in the where list, Mysql can convert the query into a constant. system is a special case of const type. When the query table has only one row, use system;
NULL: Mysql decomposes the statement during the optimization process, and does not even need to access the table or index during execution. For example, selecting the minimum value from an index column can be done through a separate index search.

2、possible_keys

Indexes that may be used when querying (but not necessarily used, and displayed as NULL if there is no index).

3、key

The index actually used.

4、rows

Estimate the number of rows needed to find the corresponding record.

5、Extra

The more common ones are the following:

Using index: Indicates that the covering index is used, and there is no need to return to the table;
Using where: instead of reading all the information in the table, the required data can be obtained only through the index. This occurs when all the requested columns of the table are part of the same index, indicating that the mysql server will retrieve rows in the storage engine and then filter;
Using temporary: Indicates that MySQL needs to use a temporary table to store the result set, which is common in sorting and grouping queries, common group by, order by;
Using filesort: When the Query contains an order by operation, and the sorting operation that cannot be completed using the index is called "file sorting".

For the situation of brushing dirty pages, we need to control the proportion of dirty pages so that it does not often approach 75%. At the same time, you must also control the writing speed of the redo log, and tell InnoDB your disk capacity by setting the innodb_io_capacity parameter.

Summarize

We always like to look up to the masters of big factories, but in fact the masters are no more than mortals. Compared with rookie programmers, they spend a little more thought. If you don't work hard, the gap will only grow wider.

The interview questions are more or less helpful for what you are going to do next, but I hope you can summarize your own shortcomings through the interview questions to improve your core technical competitiveness. Every interview experience is a literacy of your skills, and the review and summary effect after the interview is excellent! If you need this full version of the real interview notes , you only need to support my article.

How to get the information: click here to download for free

Summarize your own shortcomings to improve your core technical competitiveness. Every interview experience is a literacy of your skills, and the review and summary effect after the interview is excellent! If you need this full version of the real interview notes , you only need to support my article.