Table of contents
The implementation of count(*)
Keep counts with a cache system
save the count in the database
-
foreword
- When developing a system, you may often need to calculate the number of rows in a table, such as the total number of all change records in a transaction system
- At this time, you may think, wouldn't a select count(*) fromt statement solve the problem?
- However, you will find that as the number of records in the system increases, the execution of this statement will become slower and slower
- Then you may think, why is MySQL so stupid, remember the total, and read it directly every time you want to check, isn’t it all right?
- Let’s talk about how the count(*) statement is implemented, and why MySQL implements it like this
- Then let’s talk about, if there is such a need for frequent changes in the application and the need to count the number of rows in the table, what can be done in business design
-
The implementation of count(*)
- The first thing to be clear is that count(*) has different implementations in different MySQL engines
- The MyISAM engine stores the total number of rows in a table on the disk, so when count(*) is executed, it will directly return this number, which is very efficient
- The InnoDB engine is in trouble. When it executes count(*), it needs to read the data from the engine line by line, and then accumulate the count
- It should be noted here that what is discussed in this article is count(*) without filter conditions. If the where condition is added, the MyISAM table cannot return so quickly
- In the previous article, I analyzed why InnoDB should be used, because InnoDB is superior to MyISAM in terms of transaction support, concurrency and data security.
- Your table must also use the InnoDB engine
- This is why calculating the total number of rows in a table becomes slower and slower when you have more and more records
- Then why doesn't InnoDB save numbers like MyISAM?
- This is because even for multiple queries at the same time, the InnoDB table "how many rows should be returned" is uncertain due to multi-version concurrency control (MVCC)
- Here is an example of count(*) to explain to you
- Assuming that there are 10,000 records in table t, three parallel sessions of users are designed
- Session A first starts the transaction and queries the total number of rows in the table once
- Session B starts the transaction, after inserting a row and recording, query the total number of rows in the table
- Session C starts a separate statement first, and after inserting a row of records, query the total number of rows in the table
- Assuming that it is executed in chronological order from top to bottom, the same row of statements is executed at the same time
- It can be seen that at the last moment, three sessions A, B, and C will query the total number of rows in table t at the same time, but the results obtained are different
- This has something to do with InnoDB's transaction design. Repeatable reading is its default isolation level, which is implemented in code through multi-version concurrency control, that is, MVCC.
- Each row of records has to judge whether it is visible to this session, so for the count(*) request, InnoDB has to read the data row by row and judge in turn, and the visible row can be used to calculate the table "based on this query" The total number of rows
- Of course, this seemingly stupid MySQL is still optimized when executing the count(*) operation
- You should know that InnoDB is an index-organized table, the leaf nodes of the primary key index tree are data, and the leaf nodes of ordinary index trees are the primary key values
- Therefore, the ordinary index tree is much smaller than the primary key index tree
- For operations such as count(*), the results obtained by traversing which index tree are logically the same
- Therefore, the MySQL optimizer will find the smallest tree to traverse
- Under the premise of ensuring the correct logic, minimizing the amount of scanned data is one of the general principles of database system design
- If you have used the showtable status command, you will find that there is also a TABLE_ROWS in the output of this command to display how many rows the table currently has. This command is executed very quickly.
- Can this TABLE_ROWS replace count(*)?
- As mentioned earlier, the value of index statistics is estimated by sampling
- In fact, TABLE_ROWS is estimated from this sample, so it is also very inaccurate
- How inaccurate, the official document says that the error may reach 40% to 50%
- Therefore, the number of rows displayed by the show table status command cannot be used directly
- Here is a summary:
- Although MyISAM table count(*) is very fast, it does not support transactions
- Although the showtable status command returns quickly, it is not accurate
- InnoDB table directly count(*) will traverse the whole table, although the result is accurate, it will cause performance problems
- So, back to the question at the beginning of the article
- If you now have a page that often needs to display the total number of operation records of the trading system, what should you do?
- The answer is, you can only count yourself
- Next, let's discuss and see what methods are available for counting yourself, and what are the advantages and disadvantages of each method
- Let me talk about the basic ideas of these methods first: you need to find a place by yourself to save the number of rows in the operation record table
-
Keep counts with a cache system
- For libraries that are updated frequently, you may immediately think of using a cache system to support
- You can use a Redis service to save the total row count of this table
- Each time a row is inserted into this table, the Redis count is incremented by 1, and each time a row is deleted, the Redis count is decremented by 1.
- In this way, the read and update operations are very fast, but do you think about any problems in this way?
- True, caching systems can lose updates
- Redis data cannot stay in memory permanently, so you will find a place to store this value periodically
- But even then, it is still possible to lose updates
- Just imagine if a row has just been inserted into the data table, the value saved in Redis is also added by 1, and then Redis restarts abnormally
- After restarting, you need to read this value back from the place where the redis data is stored, but the count operation that just added 1 is lost
- Of course, there is still a solution
- For example, after Redis restarts abnormally, execute count(*) in the database to obtain the real number of rows, and then write this value back to Redis.
- Abnormal restart is not a frequent occurrence after all, the cost of this full table scan is still acceptable
- But actually, the way counts are kept in the cache system, it's not just a matter of lost updates
- Even if Redis is working properly, this value is logically imprecise
- You can imagine that there is such a page that displays the total number of operation records and at the same time displays the 100 most recently operated records
- Then, the logic of this page needs to go to Redis to get the count first, and then go to the data table to get the data records
- is so imprecisely defined:
- 1- One is that there are the latest inserted records in the 100 rows found, but 1 has not been added to the count of Redis
- 2- The other is that there is no newly inserted record in the 100 rows found, and 1 has been added to the count of Redis
- Both cases are logically inconsistent.
- Take a look at this timing diagram again:
- In the figure, session A is a logic for inserting transaction records, inserting a row R into the data table, and then adding 1 to the Redis count
- Session B is the data required for querying page display
- In this time series in the figure, when session B queries at T3, the newly inserted record R will be displayed, but the count of Redis has not yet increased by 1
- At this time, there will be inconsistencies in the data we are talking about.
- You will definitely say that this is because when we execute the new record logic, we first write the data table, and then change the Redis count
- When reading, read Redis first, and then read the data table. This order is reversed.
- So, if the order is kept the same, is there no problem?
- Now change the update order of session A, and then look at the execution results:
- You will find that it is reversed at this time. When session B queries at T3, the Redis count increases by 1, but the newly inserted R row cannot be found, which is also a case of data inconsistency.
- In a concurrent system, we cannot precisely control the execution time of different threads, because there is such an operation sequence in the figure, so even if Redis works normally, the count value is still logically inaccurate
-
save the count in the database
- According to the above analysis, using the cache system to save the count has the problems of missing data and inaccurate counting
- So, what if you put this count directly into a separate count table C in the database?
- First of all, this solves the problem of crash loss. InnoDB supports crash recovery without losing data
- Then see if you can solve the problem of inaccurate counting
- Would you say, is this different?
- It is nothing more than changing the operation of Redis in the above figure to the operation of counting table C
- As long as the execution sequence in the above figure appears, this problem is still unsolvable, right?
- This question is not really unsolvable
- The problem to be solved in this article is that InnoDB needs to support transactions, so the InnoDB table cannot store count(*) directly, and then return it directly when querying.
- Now use the feature of "transaction" to solve the problem
- Let's look at the current execution results
- Although the read operation of session B is still performed in T3, because the update transaction has not been committed at this time, the operation of adding 1 to the count value is not yet visible to session B
- Therefore, in the results seen by session B, the lookup value and the results seen in the "last 100 records" are logically consistent
-
different usage of count
- In a query statement such as select count(?) from t, what are the differences in the performance of different usages such as count(*), count(primary key id), count(field) and count(1)
- This time we talked about the performance of count(*), let me explain the performance differences of these usages
- It should be noted that the following discussion is still based on the InnoDB engine
- Here, we must first figure out the semantics of count()
- count() is an aggregation function. For the returned result set, it is judged line by line. If the parameter of the count function is not NULL, the cumulative value will be increased by 1, otherwise it will not be added.
- Finally return the accumulated value
- Therefore, count(*), count(primary key id) and count(1) all indicate the total number of rows returned in the result set that meets the conditions
- And count (field) means to return the total number of data rows that meet the conditions, and the parameter "field" is not NULL
- As for the analysis of performance differences, you can remember the following principles:
- 1-The server layer will give whatever it wants
- 2-InnoDB only gives necessary values
- 3- The current optimizer only optimizes the semantics of count(*) to "fetch the number of rows", and does not do other "obvious" optimizations
- For count (primary key id)
- The InnoDB engine will traverse the entire table, take out the id value of each row, and return it to the server layer
- After the server layer gets the id, it is judged that it is impossible to be empty, and it is accumulated by row
- For count(1)
- The InnoDB engine traverses the entire table, but does not fetch values
- For each line returned by the server layer, put a number "1" into it, judging that it is impossible to be empty, and accumulate by line
- If you just look at the difference between these two usages, it can be compared that count(1) executes faster than count(primary key id)
- Because returning id from the engine will involve parsing data rows and copying field values
- For count (field):
- 1- If this "field" is defined as not null, read this field from the record line by line, judge that it cannot be null, and accumulate by line
- 2- If the definition of this "field" is allowed to be null, then when it is executed, it is judged that it may be null, and the value must be taken out and judged again. It is not null to accumulate
- That is to say, the first principle above, what fields are required by the server layer, and what fields InnoDB will return
- But count(*) is an exception, it does not take out all the fields, but it is specially optimized and does not take values
- count(*) is definitely not null, it is accumulated by row
- Seeing this, you will definitely say, can’t the optimizer judge it by itself? The primary key id must be non-null, why can’t it be processed according to count(*), what a simple optimization
- Of course, it is not impossible for MySQL to optimize this statement specifically.
- But there are too many situations that require special optimization, and MySQL has already optimized count(*), you can use this usage directly
- So the conclusion is: in order of efficiency, count(field)<count(primary key id)<count(1)≈count(*)
- So it is recommended to use count(*) as much as possible