Performance test analysis and tuning

Performance analysis

1. premise

   The premise of performance analysis requires not only rich performance test monitoring (such as PTS's own client-side monitoring, basic monitoring-Alibaba Cloud monitoring, application monitoring-ARMS monitoring, etc.), but also relevant technical knowledge (including but not limited to: Operating system, middleware, database, development, etc.).

2. Process
   • In many cases, the pressure measurement traffic does not completely enter the back end (server end). At the network access layer (cloud architecture, such as: SLB / WAF / high-defense IP, or even CDN / full-station acceleration, etc.) There will be restrictions due to various specifications (bandwidth, maximum number of connections, number of new connections, etc.) or because certain characteristics of the pressure test conform to the behavior of CC and DDoS, and the protection strategy is triggered, resulting in the pressure test results not reaching expectations.
   • Next, see if the key indicators meet the requirements. If they do not, you need to determine where there is a problem. In general, the server-side problem is more likely to be the client-side problem (this situation is very small).
   • For server-side problems, it is necessary to locate hardware-related indicators, such as CPU, Memory, Disk I / O, Network I / O. If there is a problem with a hardware indicator, an in-depth analysis is required.
   • If there is no problem with the hardware indicators, you need to check the middleware related indicators, such as: thread pool, connection pool, GC, etc. If these indicators are problems, in-depth analysis is required.
   • If there is no problem with the middleware related indicators, you need to check the database related indicators, such as: slow check SQL, hit rate, lock, parameter settings.
   • If the above indicators are normal, the algorithm, buffering, caching, synchronous or asynchronous of the application may have problems, which requires specific and in-depth analysis.

   The details are shown below:

   
3. Possible bottleneck
   1. Bottlenecks in hardware / specifications

      Generally refers to the CPU, memory, and disk I / O issues, which are divided into server hardware bottlenecks and network bottlenecks (can not be considered for LANs).

   2. Performance bottlenecks on middleware

      It generally refers to application software such as application servers and web servers, and also includes database systems. For example: the parameter setting of the JDBC connection pool configured on the middleware weblogic platform is unreasonable, resulting in a bottleneck.

   3. Performance bottlenecks on the application

      Generally refers to applications developed by developers. For example, the JVM parameters are unreasonable, the container configuration is unreasonable, slow SQL (Alibaba Cloud APM products such as ARMS can be used to assist in positioning), the database design is unreasonable, the program architecture planning is unreasonable, and the program itself has problems (serial processing, request Insufficient processing threads, no buffers, no buffers, uncoordinated producers and consumers, etc.), resulting in a bottleneck caused by poor performance of the system in the direction of a large number of users.

   4. Performance bottlenecks on the operating system

      Generally refers to windows, UNIX, Linux and other operating systems. For example, during performance testing, when there is insufficient physical memory, the setting of virtual memory is not reasonable, and the exchange efficiency of virtual memory will be greatly reduced, resulting in a greatly increased response time of behavior. At this time, it is considered that a performance bottleneck occurs on the operating system.

   5. Performance bottlenecks on network devices

      Generally refers to firewalls, dynamic load balancers, switches and other equipment. Currently more network access products used in cloud service architecture: including but not limited to SLB / WAF / high-defense IP / CDN / full station acceleration, etc. For example, a dynamic load balancing mechanism is set on the dynamic load balancer. When it is found that the hardware resources on an application server have reached the limit, the dynamic load balancer sends subsequent transaction requests to other lightly loaded application servers. . During the test, it was found that the dynamic load balancer did not play a corresponding role. At this time, the network bottleneck can be considered.

4. method
   1. CPU

      If the CPU resource utilization rate is high, you need to look at the state of CPU consumption User, Sys, Wait.

      • If the CPU User is very high, you need to see which process is consumed, you can use the top (linux) command to see, then use top –H –p <pid> to see which thread consumes high resources, if it is a java application, you can use jstack to see Get out the stack that this thread is executing, see which method is used by the resource, and check the source code to know the problem; if it is a C ++ application, you can use the gprof performance tool to analyze it.
      • If the CPU Sys is very high, you can use strace (linux) to see the resource consumption and time of system calls.
      • If the CPU Wait is very high, consider the disk read and write, you can reduce the log output, asynchronous or change the hard disk with fast speed.
   2. Memory

      In order to maximize the use of memory, the operating system generally sets up a large amount of cache. Therefore, it is not a problem that the memory utilization rate is as high as 99%. The problem of memory mainly depends on whether the memory occupied by a process is very large and whether there is a large amount of swap (virtual memory) exchange).

   3. Disk I / O

      One of the most significant indicators of disk I / O is the busy rate, which can be achieved by reducing log output, asynchronously, or changing hard disks with high speed.

   4. Network I / O

      Network I / O mainly considers the size of the transmitted content, which cannot exceed 70% of the maximum value of the hardware network transmission. You can use compression, reduce the content size, set the cache locally, and transmit multiple times.

   5. Kernel parameters

      The kernel parameters generally have default values. These kernel parameter default values ​​are no problem for the general system, but for stress tests, the parameters that may be run will exceed the kernel parameters, causing system problems, you can use sysctl to view and modify.

   6. JVM

      jvm mainly analyzes whether GC / FULL GC is frequent, and the time of garbage collection, you can use the jstat command to check, for each generation size and GC is frequent, dump the memory through jmap, and then use the tool HeapAnalyzer to analyze where the memory occupied High and whether there is a possibility of memory leak. For simplicity, you can use APM tools, such as Alibaba Cloud ARMS, the same below.

   7. Thread Pool

      If the thread is not enough, you can increase the thread through parameter adjustment; for the situation where the thread setting in the thread pool is relatively large, it is still not enough. The possible reason is: a thread is blocked and cannot be released, and it may take more time to wait for the lock and method. Long, long database waiting time and other reasons, need further analysis to locate.

   8. JDBC connection pool

      When the connection pool is not enough, you can adjust and increase it through parameters; but for the slow processing of the database itself, the adjustment does not have much effect. You need to check the database and the reason why the connection is not released due to the code.

   9. SQL

      SQL inefficiency is also a very important reason for poor performance. You can see where SQL is slow by looking at the execution plan. In general, the main reasons for SQL inefficiency are:

      1. 1.  

            category	Subclass	Expression or description	the reason
            index	Unindexed	-	Generate full table scan
            	Unutilized index	substring(card_no,1,4)=′5378′	Generate full table scan
            		amount/30< 1000	Generate full table scan
            		convert(char(10),date,112)=′19991201′	Generate full table scan
            		where salary<>3000	Generate full table scan
            		name like ‘%张’	Generate full table scan
            		first_name + last_name ='beill cliton'	Generate full table scan
            		id_no in(′0′,′1′)	Generate full table scan
            		select id from t where num=@num	A parameter will also generate a full table scan
            	Use inefficient indexes	oder by non-clustered index	Low index performance
            		username = '张三' and age> 20	String index is lower than integer index
            		Table columns and empty NULL values	Low index performance
            		Try not to use IS NULL or IS NOT NULL	Low index performance
            The amount of data	All data	select *	Many columns produce a lot of data
            		select id,name	There are millions of rows in the table, generating a lot of data
            	Nested query	Filter data first, then filter data	Generate a lot of useless data
            	Related query	Multi-table association query, first filter out a small part of the data, filter most of the data	Large number of related operations
            	Large amount of data insertion	Insert again and again	Generate a lot of logs and consume resources
            lock	Lock wait	update account set banlance=100 where id=10	Generate table-level locks that will lock the entire table
            	Deadlock	A:update a;update b;B:update b;update a;	Deadlock
            	cursor	Cursor Open cursor,fetch;close cursor	Very low performance
            	Temporary tables	create tmp table	Generate a lot of logs
            	drop table	Delete temporary table	Need to display delete to avoid long-term locking of system tables
            other	exist instead of IN	select num from a where num in(select num from b)	in will be judged one by one, the end of the existing one will end
            	exist 代替select count(*)	Determine if the record exists	count (*) will accumulate the calculation, and the exit will end
            	between instead of IN	ID in(1,2,3)	IN is judged one by one, between is scope judgment
            	left outer join 代替Not IN	select ID from a where ID not in(select b.Mainid from b)	NOT IN judges one by one, the efficiency is very low
            	union all 代替union	select ID from a union select id from b union	删除重复的行，可能会在磁盘进行排序而union all只是简单的将结果并在一起
            	常用SQL尽量用绑定变量方法	insert into A(ID) values(1)	直接写SQL每次都要编译，用绑定变量的方法只编译一次，下次就可以用了

             

      调优

      1. 调优步骤
         1. 确定问题

            应用程序代码：在通常情况下，很多程序的性能问题都是写出来的，因此对于发现瓶颈的模块，应该首先检查一下代码。 数据库配置：经常引起整个系统运行缓慢，一些诸如大型数据库都是需要DBA进行正确的参数调整才能投产的。 操作系统配置：不合理就可能引起系统瓶颈。 硬件设置：硬盘速度、内存大小等都是容易引起瓶颈的原因，因此这些都是分析的重点。 网络：网络负载过重导致网络冲突和网络延迟。

         2. 分析问题

            当确定了问题之后，我们要明确这个问题影响的是响应时间吞吐量，还是其他问题？是多数用户还是少数用户遇到了问题？如果是少数用户，这几个用户与其它用户的操作有什么不用？系统资源监控的结果是否正常？CPU的使用是否到达极限？I/O 情况如何？问题是否集中在某一类模块中？ 是客户端还是服务器出现问题？ 系统硬件配置是否够用？实际负载是否超过了系统的负载能力？ 是否未对系统进行优化？ 通过这些分析及一些与系统相关的问题，可以对系统瓶颈有更深入的了解，进而分析出真正的原因。

         3. 确定调整目标和解决方案

            高系统吞吐量，缩短响应时间，更好地支持并发。

         4. 测试解决方案

            对通过解决方案调优后的系统进行基准测试。（基准测试是指通过设计科学的测试方法、测试工具和测试系统，实现对一类测试对象的某项性能指标进行定量的和可对比的测试）。

         5. 分析调优结果

            系统调优是否达到或者超出了预定目标？系统是整体性能得到了改善，还是以系统某部分性能来解决其他问题。调优是否可以结束了。 最后，如果达到了预期目标，调优工作可以先告一段落。

      2. 调优注意事项
         • In the process of designing and developing application systems, performance should always be considered, and performance testing should be normalized. Daily intranet performance testing + regular real-world business performance testing can be supported by PTS.
         • Determining clear and clear performance targets is the key, and then transforming the targets into the pressure test scenario in PTS and setting the required target magnitude, and then selecting the concurrent / TPS mode as appropriate, and automatically increasing / manually adjusting the speed for flow control.
         • You must ensure that the tuned program runs correctly.
         • The performance of the system depends to a greater extent on good design, and tuning techniques are only an aid.
         • The tuning process is an iterative and gradual process, and the results of each tuning must be fed back to subsequent code development.
         • Performance tuning cannot be at the expense of code readability and maintainability