Paging SQL Server Optimization and Row_Number () existing paging problem

A recent item response, the server CPU usage high, our event query page is very slow, even for a few records inquiry 4 minutes or even longer, and when turned to the second page is so much time, this is certainly unacceptable, which is a site with SQLServerProfilerthe statement crawl up.

Paging using the ROW_NUMBER ()

We take a look at the scene caught up pagination statement:

select top 20 a.*,ag.Name as AgentServerName,,d.Name as MgrObjTypeName,l.UserName as userName 
from eventlog as a 
	left join mgrobj as b on a.MgrObjId=b.Id and a.AgentBm=b.AgentBm 
	left join addrnode as c on b.AddrId=c.Id 
	left join mgrobjtype as d on b.MgrObjTypeId=d.Id 
	left join eventdir as e on a.EventBm=e.Bm 
	left join agentserver as ag on a.AgentBm=ag.AgentBm 
	left join loginUser as l on a.cfmoper=l.loginGuid 
where a.OrderNo not in  (
	select top 0 OrderNo  
	from eventlog  as a 
		left join mgrobj as b on a.MgrObjId=b.Id 
		left join addrnode as c on b.AddrId=c.Id  
	where 1=1 and a.AlarmTime>='2014-12-01 00:00:00' and a.AlarmTime<='2014-12-26 23:59:59' 
		and b.AddrId in ('02109000',……,'02109002') 
	order by  AlarmTime desc 
	)  
and 1=1 and a.AlarmTime>='2014-12-01 00:00:00' and a.AlarmTime<='2014-12-26 23:59:59' 
	and b.AddrId in ('02109000',……,'02109002') 
order by  AlarmTime DESC

This is typical to use two top paging wording, the principle is: first find pageSize*(pageIndex-1)(T1) of the number of records, then Topthe PageSizerecords in the article is not in T1, is the current record page. This query efficiency is not high mainly used not in. Prior to refer to my article " program ape is how to solve the CPU100% of SQLServer account " mentioned: "For the expression does not use SARG operators, the index is of no use" .

So instead use ROW_NUMBERtabs:

WITH cte AS(
	select a.*,ag.Name as AgentServerName,d.Name as MgrObjTypeName,l.UserName as userName,b.AddrId
			,ROW_NUMBER() OVER(ORDER BY AlarmTime DESC) AS RowNo
		from eventlog as a WITH(FORCESEEK) 
			left join mgrobj as b on a.MgrObjId=b.Id and a.AgentBm=b.AgentBm 
			left join addrnode as c on b.AddrId=c.Id 
			left join mgrobjtype as d on b.MgrObjTypeId=d.Id 
			left join eventdir as e on a.EventBm=e.Bm 
			left join agentserver As ag on a.AgentBm=ag.AgentBm 
			left join loginUser as l on a.cfmoper=l.loginGuid 
		where a.AlarmTime>='2014-12-01 00:00:00' and a.AlarmTime<='2014-12-26 23:59:59' 
			AND b.AddrId in ('02109000',……,'02109002')
)
SELECT * FROM cte WHERE RowNo BETWEEN 1 AND 20;

Execution time from 14 seconds to 5 seconds lift, which is more efficient paging Row_Number described, but such an approach than top topmany elegant tab.

"Deceived" by the query engine allows queries to query your expectations

But why you would want to query 20 records five seconds of it, especially in this case where the table is to add a time index - Reference " program ape is how to solve SQLServer accounted CPU100% of " index mentioned.

I tried to remove the phrase AND b.AddrId in ('02109000',……,'02109002'), the results of less than 1 second to put 538 records check out, and place restrictions coupled with the phrase, the result is 204 rows. Why is not the result set, but the time it takes to differ so much? View the execution plan and found that taking another index, rather than the time index.

To put this question on SQLServer group, and soon, Sang high to reply: To achieve the effect of restricting the phrase with the site removed, use AdddrId+'' in.

What does this mean? Sometimes do not understand, it is a high-Sang did not understand my statement? Soon, it was added, to deceive the query engine. " Cheat "? Or do not know, but I did, the statements above cte intact Copy out, and then rephrased AND b.AddrId in ('02109000',……,'02109002')to change the order AND b.AddrId+'' in ('02109000',……,'02109002'), is executed, God! ! ! It finished less than a second to execute. In the Plan of Implementation of a pair of really taking the time index:

Later, pondering for a moment, before remembering seeing query engine optimization theory, if your condition with the operator or use functions, etc., will give up the query engine optimization, and perform a table scan. Suddenly turned his head, and in use b.AddrId+''before the query engine attempts to mgrObj table join together to do optimization, then two tables linked to the investigation, the estimated number of records will lead to greatly increased, and the use of the b.AddrId+''query engine will index the press time recording elected brush, so that to achieve the effect that forced cte do first execution incondition, rather than in the cte inconditions selected from the brush. That was it! Sometimes, excessive query optimization engine, will lead to the opposite effect, and you can know if the optimization principle, then you can make some small tips query engine as you expect to be optimized .

ROW_NUMBER () page in the pages of the larger problem

Things here, not the end. Any connection with my colleagues back reaction, queries to the back pages, and card! what? I re-run the above statement, the time frame put 2011-12-01 to 2014-12-26, limit the number of records to 19981-20000, sure enough, about 30 seconds to query, view the execution plan, they are the same ,why?

High Sang suspected key lookup caused by too much, it is recommended to rid paged out to do key lookup. I do not understand what is the meaning of such a sentence. The IO implementation plan and print it out:

看看IO，很明显，主要是越到后面的页数，其他的几个关联表读取的页数就越多。我推测，在Row_Number分页的时候，如果有表连接，则按排序一致到返回的记录数位置，前面的记录都是要参与表连接的，这就导致了越到后面的分页，就越慢，因为要扫描的关联表就越多。

难道就没有了办法了吗？这个时候宋桑英勇的站了出来：“你给表后加一个forceseek提示可破”。这真是犹如天籁之音，马上进行尝试。

使用forceseek提示可以强制表走索引

查了下资料：

SQL Server2008中引入的提示ForceSeek，可以用它将索引查找来替换索引扫描

那么，就在eventlog表中加上这句看看会怎样?

果然，查询计划变了，开始提示，缺少了包含索引。赶紧加上，果然，按这个方式进行查询之后查询时间变为18秒，有进步！但是查看IO，跟上面一样，并没有变少。不过，总算学会了一个新的技能，而宋桑也很热心说晚上再帮忙看看。

把其他没参与where的表放到cte外面

根据上面的IO，很快，又有人提到，把其他left join的表放到cte外面。这是个办法，于是把除eventlog、mgrobj、addrnode的表放到外面，语句如下：

WITH cte AS(
	select a*,b.AddrId,b.Name as MgrObjName,b.MgrObjTypeId          
			,ROW_NUMBER() OVER(ORDER BY AlarmTime DESC) AS RowNo
		from eventlog as a
			left join mgrobj as b on a.MgrObjId=b.Id and a.AgentBm=b.AgentBm 
			left join addrnode as c on b.AddrId=c.Id 
		where a.AlarmTime>='2011-12-01 00:00:00' and a.AlarmTime<='2014-12-26 23:59:59' 
			AND b.AddrId+'' in ('02109000',……,'02109002')
)
SELECT a.* 
	,ag.Name as AgentServerName
	,d.Name as MgrObjTypeName,l.UserName as userName
FROM cte a left join eventdir as e on a.EventBm=e.Bm 
			left join mgrobjtype as d on a.MgrObjTypeId=d.Id 
			left join agentserver As ag on a.AgentBm=ag.AgentBm 
			left join loginUser as l on a.cfmoper=l.loginGuid 
WHERE RowNo BETWEEN 19980 AND 20000;

果然有效，IO大大减少了，然后速度也提升到了16秒。

表 'loginuser'。扫描计数 1，逻辑读取 63 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'agentserver'。扫描计数 1，逻辑读取 1617 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'mgrobjtype'。扫描计数 1，逻辑读取 126 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'eventdir'。扫描计数 1，逻辑读取 42 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'addrnode'。扫描计数 1，逻辑读取 119997 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'Worktable'。扫描计数 0，逻辑读取 0 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'eventlog'。扫描计数 1，逻辑读取 5027 次，物理读取 3 次，预读 5024 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'mgrobj'。扫描计数 1，逻辑读取 24 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。

我们看到，addrNode表还是扫描计数很大。那还能不能提升，这个时候，我想到了，先把addrNode、mgrobj、mgrobjtype三个表联合查询，放到一个临时表，然后再和eventlog做inner join，然后查询结果再和其他表做left join，这样还能减少IO。

使用临时表存储分页记录在进行表连接减少IO

IF OBJECT_ID('tmpMgrObj') IS NOT NULL DROP TABLE tmpMgrObj
SELECT m.Id,AddrId,MgrObjTypeId,AgentBM,m.Name,a.Name AS AddrName 
	INTO tmpMgrObj  
	FROM dbo.mgrobj m
		INNER JOIN dbo.addrnode a ON a.Id=m.AddrId
	WHERE AddrId IN('02109000',……,'02109002');
WITH cte AS(
	select a.*,b.AddrId,b.MgrObjTypeId          
			,ROW_NUMBER() OVER(ORDER BY AlarmTime DESC) AS RowNo
			,ag.Name as AgentServerName
	,d.Name as MgrObjTypeName,l.UserName as userName
		from eventlog as a
			INNER join tmpMgrObj as b on a.MgrObjId=b.Id and a.AgentBm=b.AgentBm
			left join mgrobjtype as d on b.MgrObjTypeId=d.Id 
			left join agentserver As ag on a.AgentBm=ag.AgentBm 
			left join loginUser as l on a.cfmoper=l.loginGuid 
	WHERE AlarmTime>'2011-12-01 00:00:00' AND AlarmTime<='2014-12-26 23:59:59'
) 
SELECT * FROM cte WHERE RowNo BETWEEN 19980 AND 20000
IF OBJECT_ID('tmpMgrObj') IS NOT NULL DROP TABLE tmpMgrObj

这次查询仅用了10秒。我们来看看IO：

表 'Worktable'。扫描计数 0，逻辑读取 0 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'mgrobj'。扫描计数 1，逻辑读取 24 次，物理读取 2 次，预读 23 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'addrnode'。扫描计数 1，逻辑读取 6 次，物理读取 3 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
----------
表 'loginuser'。扫描计数 0，逻辑读取 24 次，物理读取 1 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'Worktable'。扫描计数 0，逻辑读取 0 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'eventlog'。扫描计数 93，逻辑读取 32773 次，物理读取 515 次，预读 1536 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'tmpMgrObj'。扫描计数 1，逻辑读取 3 次，物理读取 0 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'mgrobjtype'。扫描计数 1，逻辑读取 6 次，物理读取 1 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。
表 'agentserver'。扫描计数 1，逻辑读取 77 次，物理读取 2 次，预读 0 次，lob 逻辑读取 0 次，lob 物理读取 0 次，lob 预读 0 次。

除了eventlog之外，其他的表的IO大大减少，有木有？

inner join和left join的区别

但是，多执行几次测试，发现上述语句还是有一点问题：查询第一页的时候，也竟然要用5秒，而查询时间在当前一个月份的，也接近5秒。这是为什么呢？这个时候，宋桑再伸援手，提供了另外一个SQL语句，在查询前面几页的时候1秒就出来了，而后面的页数，则变化不大。我仔细比较了两个语句，原来我用的是inner join，而宋桑给的是left join。这两个有什么区别呢。仔细对比查询计划之后发现，使用inner join的时候，查询引擎会先执行inner join而非子查询，而使用left join则查询引擎先执行子查询。因此如果使用了inner join会导致在查询1个月的数据时，没有有效利用了时间索引。最终，我研究出来的语句如下，在查询最新数据或者前面几页的数据，能够在1秒左右出来，而查询后面的页数，在10秒左右，基本解决了问题。

IF OBJECT_ID('tmpMgrObj') IS NOT NULL DROP TABLE tmpMgrObj
SELECT m.Id,AddrId,MgrObjTypeId,AgentBM,m.Name,a.Name AS AddrName,t.Name AS MgrObjTypeName
	INTO tmpMgrObj  
	FROM dbo.mgrobj m
		INNER JOIN dbo.addrnode a ON a.Id=m.AddrId
		INNER JOIN dbo.mgrobjtype t ON m.MgrObjTypeId=t.Id
	WHERE AddrId+'' IN('02109000',……,'02109002');
SELECT tmp.*
	,ag.Name AS AgentServerName
	, l.UserName AS userName
FROM    ( 
	SELECT    a.* ,b.MgrObjTypeName  , b.AddrId
		,ROW_NUMBER() OVER ( ORDER BY AlarmTime DESC ) AS RowNo
	FROM
		(SELECT    * 
			FROM      eventlog
			WHERE     AlarmTime >= '2011-12-01 00:00:00' AND AlarmTime <= '2014-12-26 23:59:59') AS a
		LEFT JOIN tmpMgrObj AS b ON a.MgrObjId=b.Id AND a.AgentBM=b.AgentBm

) tmp 
	LEFT JOIN eventdir AS e ON tmp.EventBm = e.Bm
		LEFT JOIN agentserver AS ag ON tmp.AgentBm = ag.AgentBm
		LEFT JOIN loginUser AS l ON tmp.cfmoper = l.loginGuid
WHERE tmp.RowNo BETWEEN 1 AND 20;
IF OBJECT_ID('tmpMgrObj') IS NOT NULL DROP TABLE tmpMgrObj

其他优化参考

在另外的群上讨论时，发现使用ROW_NUMBER分页查询到后面的页数会越来越慢的这个问题的确困扰了不少的人。

有的人提出，谁会这么无聊，把页数翻到几千页以后?一开始我也是这么想的，但是跟其他人交流之后，发现确实有这么一种场景，我们的软件提供了最后一页这个功能，结果……当然，一种方法就是在设计软件的时候，就去掉这个最后一页的功能；另外一种思路，就是查询页数过半之后，就反向查询，那么查询最后一页其实也就是查询第一页。

还有一些人提出，把查询出来的内容，放到一个临时表，这个临时表中的加入自增Id的索引，这样，可以通过辨别Id来进行快速刷选记录。这也是一种方法，我打算稍后尝试。但是这种方法也是存在问题的，就是无法做到通用，必须根据每个表进行临时表的构建，另外，在超大数据查询时，插入的记录过多，因为索引的存在也是会慢的，而且每次都这么做，估计CPU也挺吃紧。但是不管怎么样，这是一种思路。

你有什么好的建议？不妨把你的想法在评论中提出来，一起讨论讨论。

总结

现在，我们来总结下在这次优化过程中学习到什么内容：

在SQLServer中，ROW_NUMBER的分页应该是最高效的了，而且兼容SQLServer2005以后的数据库
通过“欺骗”查询引擎的小技巧，可以控制查询引擎部分的优化过程
ROW_NUMBER分页在大页数时存在性能问题，可以通过一些小技巧进行规避
- 尽量通过cte利用索引
- 把不参与where条件的表放到分页的cte外面
- 如果参与where条件的表过多，可以考虑把不参与分页的表先做一个临时表，减少IO
inner join会优先于子查询，而left join不会
使用with(forceseek)可以强制查询因此进行索引查询

转载于:https://www.cnblogs.com/Alenliu/p/5044304.html