MySQL database optimized to achieve rapid ten million page analysis, look at it.
Data Sheet collect (id, title, info, vtype) on four fields, wherein a fixed-length title, info with text, id gradually, the VType is tinyint, vtype is an index. This is a simple model of a basic information system. Now fill the data entered, filling 100,000 news.
Finally collect 10 million records, database tables take up hard disk 1.6G. OK, see below this sql statement:
select id,title from collect limit 1000,10; 很快；基本上0.01秒就OK，再看下面的
select id,title from collect limit 90000,10; 从9万条开始分页，结果？
8-9 seconds to complete, my god what a problem? ? ? ? In fact, to optimize this data, the Internet to find the answer. Look at the following statement:
select id from collect order by id limit 90000,10; soon, 0.04 seconds on OK. why? Because with the id primary key indexed fast course. Online reform law are:
select id,title from collect where id>=(select id from collect order by id limit 90000,1) limit 10;
This is done with the results of the id index. But the problem a little bit complicated, would be finished. Look at the following statements
select id from collect where vtype=1 order by id limit 90000,10; 很慢，用了8-9秒！
To here I believe many people will like me, have a feeling of collapse! vtype did indexed ah? How will it slow? vtype do index is good, you just select id from collect where vtype = 1 limit 1000,10; is very fast, basically 0.05 seconds, but a 90-fold increase, from 90,000 to start, that is 0.05 * 90 = 4.5 seconds faster. 8-9 seconds and the test results to an order of magnitude. From here it was suggested that the idea of the points table, and discuz this forum is the same idea. Ideas are as follows:
Create an index table: t (id, title, vtype) and arranged fixed length, then do tab, and then to collect the results page to look inside info. Feasible? Under the experimental know.
100,000 records to t (id, title, vtype), the data table size is about 20M. use
select id from t where vtype = 1 order by id limit 90000,10; very quickly. Substantially 0.1-0.2 seconds to finish. Why is this so? My guess is because too many collect data, paging run a long way. full size limit and data tables related. In fact, this is still a full table scan, but because of the small amount of data, only 100,000 was fast. OK, to a crazy experiment, added 1 million, test performance.
With 10 times the data on the table immediately to t 200 M, and is a fixed length. Or just the query, time is 0.1-0.2 seconds to complete! Sub-table performance is no problem? wrong! Because our limit is 90,000, so fast. To a large, 900,000 start
select id from t where vtype=1 order by id limit 900000,10; 看看结果，时间是1-2秒！
why ?? score sheet still such a long time, very depressed! Some people settle long will improve the performance limit, and I also began to think, because a record length is fixed, mysql should be able to calculate the position of the fishes 900,000 ah? But we overestimated the intelligence mysql, he is not a business database, not proven to fixed-length and non-fixed-length effect on the limit? No wonder some people say discuz to 1 million records will be very slow, I believe this is true, the database design and relevant!
Is MySQL can not break 1 million limit? ? ? To 1 million pages really to the limit? ? ?
The answer is: NO !!! Why not exceeded 100 million mysql because they can not design caused. Here are presumptuous table method to test crazy! A table to get one million records, and 10G database, how fast page!
Well, we went back to collect the test table, began testing concluded that: 300,000 data points table with a feasible method, more than 300,000 of his speed will slow road you can not stand! Of course, if the points table + I this method, it is absolutely perfect. But after I took this method, no sub-table can be the perfect solution!
The answer is: a composite index! Once the design mysql index when the index stumbled You can use any name, you can choose several fields to come in, what use is it? select the start of id from collect order by id limit 90000,10; so fast that they have gone in the index, but if the increase will not go where the index. Try holding the idea of adding such an index search (vtype, id). Then test
select id from collect where vtype=1 limit 90000,10; 非常快！0.04秒完成！
Then test: select id, title from collect where vtype = 1 limit 90000,10; regret, 8-9 seconds, did not walk a search index!
Retest: search (id, vtype), or select id this statement is very regrettable, 0.5 seconds.
To sum up: if there where conditions for, they want to go with the limit of the index, an index must be designed, will put the first one where the primary key limit used to put No. 2, and only select a primary key!
The perfect solution pagination problem. Can quickly return id optimization limit, there is hope, according to this logic, one million of the limit should be able to spread over 0.0x seconds. It seems very important to optimization and indexing mysql statement!
Okay, back to the original question, how to quickly apply the above study successfully develop it? If the composite query, I'm no lightweight frame is used. Paging string had to write their own, that much trouble? Here's look at one example, the idea came out:
select * from collect where id in (9000,12,50,7000); 竟然 0秒就可以查完！
mygod, mysql index unexpectedly effective in the same sentence! It seems the Internet that can not be used in the index is wrong!
With this conclusion, it can easily be applied to a lightweight framework:
code show as below:
$db=dblink();
$db->pagesize=20;
$sql="select id from collect where vtype=$vtype";
$db->execute($sql);
$strpage=$db->strpage(); //将分页字符串保存在临时变量，方便输出
while($rs=$db->fetch_array()){
$strid.=$rs['id'].',';
}
$strid=substr($strid,0,strlen($strid)-1); //构造出id字符串
$ Db-> pagesize = 0; // critical, without cancellation of classes, the paging empty, so only need a database connection, do not need to open;
$db->execute("select id,title,url,sTime,gTime,vtype,tag from collect where id in ($strid)");
<?php while($rs=$db->fetch_array()): ?>
<tr>
<td> <?php echo $rs['id'];?></td>
<td> <?php echo $rs['url'];?></td>
<td> <?php echo $rs['sTime'];?></td>
<td> <?php echo $rs['gTime'];?></td>
<td> <?php echo $rs['vtype'];?></td>
<td> <a href="?act=show&id=<?php echo $rs['id'];?>" target="_blank"><?php echo $rs['title'];?></a></td>
<td> <?php echo $rs['tag'];?></td>
</tr>
<?php endwhile; ?>
</table>
<?php
echo $strpage;
By simple transformation, in fact, the idea is very simple: 1) by optimizing the index to find the id, and makes up "123,90000,12000" This string. 2) The second query to find out the results.
Index + a tiny little bit of change to the mysql can support millions or even tens of millions efficient paging!
By way of example here, I reflect on one thing: For large systems, PHP must not use framework, especially the kind of sql statement can not see even a frame! Since the beginning of my lightweight frameworks for almost collapsed! Only suitable for the rapid development of small applications for ERP, OA, large sites, including the data layer logical layer of stuff not use framework. If the programmer lost control of the sql statement, the risk that the project will be a geometric increase! Especially when using mysql, mysql necessarily need professional dba can play his best performance. A performance index difference may be caused by a thousand times!
PS: After the actual test, to one million data, 1.6 million data, 15G table, 190M index, even take the index, limit had 0.49 seconds. So better not let others see page 100 000 after the data, or else will be very slow! Even if the index. After this optimization, mysql to one million pages a limit! But such a result is very good, if you are using sqlserver definitely stuck! 1,600,000 and data in (str) fast with id, or substantially 0 seconds. If so, tens of millions of data, mysql should also be very easy to deal with.