Method 1: Use SQL statements directly provided by the database
-
Sentence pattern: the MySQL, the following methods can be used: SELECT * FROM table name LIMIT M, N
-
Adaptation scenario: applicable to the case of a relatively small amount of data (tuple hundred / one thousand)
-
Cause / Disadvantages: full table scan, the speed will be very slow and some database result set returned instability (such as a particular return 1,2,3, another on a return 2,1,3) Limit restrictions from the result set. M at position N output removed, the rest discarded.
Method 2: Create a primary key or a unique index, using the index (assuming 10 per page)
-
Sentence pattern: the MySQL, the following methods can be used: SELECT * FROM table name WHERE id_pk> (pageNum * 10) LIMIT M
-
Match the scene: suitable for number of cases the amount of data (number of tuples thousands)
-
The reason: an index scan speed will soon have a friend asked: check out the case because the data is not in accordance pk_id sort, so there will be missing data, only method 3
Method 3: based index reordering
-
Sentence pattern: the MySQL, the following methods can be used: SELECT * FROM table name WHERE id_pk> (pageNum * 10) ORDER BY id_pk ASC LIMIT M
-
Adaptation scenario: data contents are large (tuple number thousands) column objects after preferably ORDER BY are primary keys or unique so that ORDERBY operator can use the index be eliminated but the result set is stable (stable meaning. see method 1)
-
The reason: an index scan, but MySQL will soon speed of sorting operations, not only ASC DESC (DESC is false, the future will do the real DESC, expectations ...).
Method 4: Use prepare based index
The first question mark indicates pageNum, the second? It represents the number of page tuples
-
Style statement: MySQL, the following methods are available: PREPARE stmt_name FROM SELECT * FROM table name WHERE id_pk> ORDER BY id_pk ASC LIMIT M (*??)
-
Match the scene: a large amount of data
-
The reason: an index scan, the speed will soon prepare the statement off than the general query faster.
Method 5: Using MySQL supports ORDER index can be used to quickly locate the operating portion tuple, avoid full table scan
For example: the first read line tuples 1000-1019 (pk is the primary key / unique key).
SELECT * FROM your_table WHERE pk>=1000 ORDER BY pk ASC LIMIT 0,20
Method 6: Using location "subquery / + indexes connected" to quickly locate the tuple, then read tuple.
For example, (id is the primary key / unique key, blue font variable)
Subqueries example:
SELECT * FROM your_table WHERE id <=
(SELECT id FROM your_table ORDER BY id desc LIMIT ($page-1)*$pagesize ORDER BY id desc
LIMIT $pagesize
Connection example using:
SELECT * FROM your_table AS t1
JOIN (SELECT id FROM your_table ORDER BY id desc LIMIT ($page-1)*$pagesize AS t2
WHERE t1.id <= t2.id ORDER BY t1.id desc LIMIT $pagesize;
Mysql large amount of data using the limit tab, as the page number increases, the search efficiency is low.
Test test
1. Direct with limit start, count page statement, which I used in the method of the program:
select * from product limit start, count
When the start page is small, there is no query performance issues, we look at each execution time from 10, 100, 1000, 10000 start paging (page take 20).
as follows:
select * from product limit 10, 20 0.016秒
select * from product limit 100, 20 0.016秒
select * from product limit 1000, 20 0.047秒
select * from product limit 10000, 20 0.094秒
We have seen that with the increase of the initial recording, time also with the increase, indicating pagination statement limit with the starting page number is a great relationship, then we start recording the change 40w facie (ie record general about)
select * from product limit 400000, 20 3.229秒
We take a look at the last recording time
select * from product limit 866613, 20 37.44秒
Like this page's largest PAGE clear that this time is intolerable.
From which we can summarize two things:
-
Query time limit is proportional to the position statement and start recording
-
The limit mysql statement is very convenient, but many of the records of the table are not suitable for direct use.
2. Performance optimization problem to limit pagination
Covering the use of the index table to speed up query paging
we all know, the use of the index query if the statement contains only the index column (covering indexes), then this situation will soon queries.
Because the use of the index Finding optimization algorithms and index data in the query above, do not have to go to address the relevant data, this saves a lot of time. In addition Mysql is also related to the index cache, at a time of high concurrency better use of caching effects.
In our example, we know that the id field is the primary key, naturally contains the default primary key index. Now let's see how to use a covering index query results.
The last page of the query data (covering the use of the index contains only the id column), between us as follows:
select id from product limit 866613, 20 0.2秒
With respect to query all the columns of 37.44 seconds to improve by about 100 times the speed of
So if we have to query all the columns, there are two ways, one is the id> = form, and the other is to use join, look at the actual situation:
SELECT * FROM product WHERE ID > =(select id from product limit 866613, 1) limit 20
Query time of 0.2 seconds!
Another way
SELECT * FROM product a JOIN (select id from product limit 866613, 20) b ON a.ID = b.id
Query time is very short!
3. Composite Index Optimization
MySql performance in the end how high? This is definitely for MySql database dba master class to play, and generally do a bit of 10000 News smaller systems can be how to write, with xx framework can achieve rapid development. But the amount of data to 100,000, one million to ten million, his performance still so high? A little mistake could cause rewrite the whole system, and even more the system can not function properly! Well, not so much nonsense.
Speak with the facts, look at an example:
Data Sheet collect (id, title, info, vtype) on four fields, wherein a fixed-length title, info with text, id gradually, the VType is tinyint, vtype is an index. This is a simple model of a basic information system. Now fill the data entered, filling 100,000 news. Finally collect 10 million records, database tables occupied hard 1.6G.
OK, see below this sql statement:
select id,title from collect limit 1000,10;
Soon; substantially 0.01 seconds on OK, look at the following
select id,title from collect limit 90000,10;
From 90,000 to start paging, the result?
8-9 seconds to complete, my god what a problem? In fact, to optimize this data, the Internet to find the answer. Look at the following statement:
select id from collect order by id limit 90000,10;
Soon, 0.04 seconds OK. why? Because with the id primary key indexed fast course. Online reform law are:
select id,title from collect where id>=(select id from collect order by id limit 90000,1) limit 10;
This is done with the results of the id index. But the problem a little bit complicated, would be finished. Look at the following statements
select id from collect where vtype = 1 order by id limit 90000,10; very slow, with the 8-9 seconds!
To here I believe many people will like me, have a feeling of collapse! vtype did indexed ah? How will it slow? vtype do index is good, you directly
select id from collect where vtype=1 limit 1000,10;
Is very fast, basically 0.05 seconds, but a 90-fold increase, from 90,000 to start, that is 0.05 * 90 = speed of 4.5 seconds in the. 8-9 seconds and the test results to an order of magnitude.
From here it was suggested that the idea of the points table, and dis #cuz this forum is the same idea. Ideas are as follows:
Create an index table: t (id, title, vtype) and arranged fixed length, then do tab, and then to collect the results page to look inside info. Feasible? Under the experimental know.
100,000 records to t (id, title, vtype), the data table size is about 20M. use
select id from t where vtype=1 order by id limit 90000,10;
soon. Substantially 0.1-0.2 seconds to finish. Why is this so? My guess is because too many collect data, paging run a long way. full size limit and data tables related. In fact, this is still a full table scan, but because of the small amount of data, only 100,000 was fast. OK, to a crazy experiment, added 1 million, test performance. With 10 times the data on the table immediately to t 200 M, and is a fixed length. Or just the query, time is 0.1-0.2 seconds to complete! Sub-table performance is no problem?
wrong! Because our limit is 90,000, so fast. To a large, 900,000 start
select id from t where vtype=1 order by id limit 900000,10;
Look at the results, the time is 1-2 seconds! why?
Or a sub-table for such a long time, very depressed! Some people settle long will improve the performance limit, and I also began to think, because a record length is fixed, mysql should be able to calculate the position of the fishes 900,000 ah? But we overestimated the intelligence mysql, he is not a business database, not proven to fixed-length and non-fixed-length effect on the limit? No wonder some people say discuz to 1 million records will be very slow, I believe this is true, the database design and relevant!
Is MySQL can not break 1 million limit? ? ? To 1 million pages really to the limit?
The answer is: NO Why not exceeded 100 million mysql because they can not design caused. Here are presumptuous table method to test crazy! A table to get one million records, and 10G database, how fast page!
Well, we went back to collect the test table, began testing concluded that:
300,000 data points table with a feasible method, more than 300,000 of his speed will slow road you can not stand! Of course, if the points table + I this method, it is absolutely perfect. But after I took this method, no sub-table can be the perfect solution!
The answer is: a composite index ! Once the design mysql index when the index stumbled You can use any name, you can choose several fields to come in, what use is it?
Started
select id from collect order by id limit 90000,10;
So fast that they have gone in the index, but if the increase will not go where the index. Try holding the idea of adding such an index search (vtype, id).
Then test
select id from collect where vtype=1 limit 90000,10;
very fast! 0.04 seconds to complete!
Re-test:
select id ,title from collect where vtype=1 limit 90000,10;
Unfortunately, 8-9 seconds, did not walk a search index!
Retest: search (id, vtype), or select id this statement is very regrettable, 0.5 seconds.
To sum up: if there where conditions for, they want to go with the limit of the index, an index must be designed, will put the first one where the primary key limit used to put No. 2, and only select a primary key!
The perfect solution pagination problem. Can quickly return id optimization limit, there is hope, according to this logic, one million of the limit should be able to spread over 0.0x seconds. It seems very important to optimization and indexing mysql statement!