Table of contents
Performance Testing Terminology Explained
Performance testing methods and objectives
Performance Requirements Analysis
Performance test cases and scenarios
Performance testing tool selection
Analysis of performance test results
performance test pass standard
Foreword:
Interface performance testing refers to testing the performance of each interface in the system, including indicators such as response time, throughput, concurrency, and performance bottlenecks. The interface performance test can find out whether the interface can run normally under the conditions of high concurrency and large data volume.
Performance Testing Terminology Explained
1. Response time
The response time is the time from when the application system sends out the request to when the client receives the last byte of data. The response time can be subdivided according to the characteristics of the software. For example, the response time of a C/S software can be subdivided into network transmission time, application server processing time, and database server processing time. In addition, the client itself also has analysis time, interface rendering time and so on.
Response time is a performance indicator mainly viewed from the perspective of the client. It is a performance indicator that users care about most and is easy to perceive.
2. Throughput rate
Throughput rate refers to the number of user requests processed by the system per unit time. From a business perspective, throughput rate can be measured in units such as requests per second, transactions per second, pages per second, and queries per second. From a network perspective, throughput can also be measured in bytes per second.
Throughput is a performance indicator mainly viewed from the perspective of the server, which can measure the processing capability of the entire system. For a cluster or cloud platform, the throughput index reflects the pressure that the server cluster can withstand as a whole, which is easier to compare than the number of users.
Remarks: throughput = throughput rate * unit time
3. Number of users
For server clusters or cloud platforms, almost all are multi-user systems, and how many users the system can provide for normal use is also a very important indicator. We distinguish these users according to the timing of using the system as follows.
System Users (SystemUsers): Refers to the number of users that the system can store.
Number of Online Users (OnlineUsers): Refers to the number of users who are in the normal use state after the user has passed the identity verification.
The number of concurrent users (Concurrentusers): refers to the number of users who are using the system at the same time within a certain time range.
Strictly the number of concurrent users: refers to the number of users operating a certain business at the same time.
In the process of performance testing, we have to simulate actual users to send requests. However, in order to put more pressure on the server, there are certain differences between our simulated user operations and actual user operations (for example, simulated user requests are more frequent than actual user requests), and the number of simulated users is different from that of actual users. Numbers are also difficult to convert to each other. Therefore, when measuring the server cluster capability, the throughput index is more practical than the user number index.
Performance testing methods and objectives
1. Performance test method
1.1 Benchmark Testing (BenchmarkTesting)
The benchmark test is based on a certain scale of data volume to test a single service or combine services in the same proportion according to actual user operations. The purpose is to quantify the indicators of response time and throughput rate for subsequent comparison.
The method is to do multiple sets of tests in different scenarios, observe the results, extract a few key data and make records for future performance comparison and evaluation.
1.2 Performance Testing (Performance Testing)
By simulating the business pressure of production operation and the combination of usage scenarios, test whether the performance of the system meets the production performance requirements.
Features:
(1) The main purpose is to verify whether the system has the capabilities claimed by the system.
(2) It is necessary to understand the typical scenarios of the system under test in advance and have a definite performance goal.
(3) It is required to operate in a determined environment.
1.3 Load Testing (LoadTesting)
By continuously increasing the pressure on the system under test until a performance indicator, such as "response time" exceeds a predetermined indicator or a certain resource usage has reached saturation.
Features:
(1) The main purpose is to find the limit of system processing capability.
(2) It needs to be carried out in a given test environment, and usually also needs to consider the business pressure and typical scenarios of the system under test, so that the test results have business significance.
(3) It is generally used to understand the performance capacity of the system, or used in conjunction with performance tuning.
1.4 Stress Testing (Stress Testing)
The test system is in a certain saturated state, such as CPU, memory, etc., under the condition of saturated usage, the session ability that the system can handle, and whether the system will have errors.
Features:
(1) The main purpose is to check the performance of the application when the system is under stress.
(2) Generally, by simulating load and other methods, the resource usage of the system reaches a higher level.
(3) Generally used to test the stability of the system.
1.5 Configuration Testing (ConfigurationTesting)
By adjusting the software/hardware environment of the system under test, we can understand the degree of influence of various environments on system performance, so as to find the optimal allocation principle of various system resources.
Features:
(1) The main purpose is to understand the degree of influence of various factors on system performance, so as to determine the most worthy tuning operation.
(2) Generally, it is carried out after a preliminary understanding of the system performance status.
(3) Generally used for performance tuning and planning capabilities.
1.6 Concurrency Testing (ConcurrencyTesting)
By simulating concurrent access by users, test whether there are deadlocks or other performance problems when multiple users access the same application, module or data record concurrently.
Features:
(1) The main purpose is to discover possible hidden concurrent access problems in the system.
(2) Focus on possible concurrency issues in the system, such as memory leaks, thread locks, and resource contention in the system.
(3) It can be used in various stages of development and requires the cooperation and support of relevant testing tools.
1.7 Reliability Testing (Reliability Testing)
By loading the system with a certain business pressure (for example, the utilization rate of resources is 70%~90%), let the application continue to run for a period of time, and test whether the system can run stably under this condition.
Features:
(1) The main purpose is to verify whether the system supports long-term stable operation.
(2) Need to operate under pressure for a period of time.
(3) It is necessary to pay attention to the operating status of the system.
1.8 Failover Testing (FailoverTesting)
Designed for systems with redundant backup and load balancing, it can be used to test whether users can continue to use the system if a partial failure occurs in the system; and to what extent users will be affected if this happens.
Features:
(1) The main purpose is to verify whether the system can continue to be used under the condition of partial failure.
(2) It is also necessary to point out the conclusion of "how many user accesses can be supported" and the plan of "what emergency measures to take" when a problem occurs.
(3) Generally speaking, this type of test is only required for systems that have clear requirements for continuous system operation indicators.
2. Performance testing objectives
Generally speaking, it can be divided into 4 aspects:
2.1 Proficiency Verification
During system testing or acceptance testing, we need to evaluate the capabilities of the system and measure the performance indicators of the system. The capacity of the system can be the number of concurrent users it can accommodate, or it can be the throughput rate of the system; the performance index of the system can be the response time, or the usage of CPU, memory, disk, and network.
Features:
(1) It is required to be carried out in a determined environment.
(2) Test schemes and use cases need to be designed according to typical scenarios.
The commonly used methods are: performance testing, stress testing, reliability testing, failure recovery testing.
2.2 Capacity planning
Evaluate whether a certain system can support user growth in the future or how to adjust the system configuration so that the system can meet the needs of the increasing number of users.
Features:
(1) It is an exploratory test
(2) It can be used to understand the performance of the system and methods to obtain extended performance, such as system expansion planning. System capacity can be user capacity, data capacity, or system throughput (processing capability of the system). For cluster services, we use throughput more as capacity.
the way is
①Performance tests are performed on each subsystem and component to find out the optimal ratio between them;
② Then through the horizontal expansion of each link, calculate the overall expansion machine ratio.
The commonly used methods are: load testing, stress testing, configuration testing.
2.3 Performance tuning
In order to give full play to the potential of the system, locate the bottleneck of the system, and optimize the system in a targeted manner.
The method is that when performing system tuning, we need to do a benchmark test to compare the changes in performance data, and repeatedly adjust the settings of the system's software and hardware to enable the system to perform optimally. Of course, when optimizing the system, we will select key indicators for optimization, and may sacrifice other performance indicators when returning. If the goal is to optimize the response time, the strategy we may choose is to trade space for time, at the cost of sacrificing memory or expanding the cache, and we need to find a balance among various performance indicators.
Generally, adjustments to the system include the following three aspects:
(1) Adjustment of the hardware environment
(2) Adjustment of system settings
(3) Application-level adjustments
The commonly used methods are: benchmark test, load test, stress test, configuration test and failure recovery test.
2.4 Defects found
Performance testing, like any other testing, can find bugs. In particular, whether there is a slow response time caused by resource contention during strict concurrent access, or whether the program crashes when a large number of users access.
The method is to set up a rendezvous point to achieve strict concurrent user access; or set up performance test cases such as ultra-large-scale user burst access for testing.
The general method used is: concurrent testing.
Performance Requirements Analysis
1. Acquisition of performance requirements
1.1 Functional Specifications
1.2 System design document
1.3 Operation plan
1.4 User Behavior Analysis Records
2. Selection of key performance points
Mainly selected from the following four dimensions:
2.1 Business Analysis
Determine whether the interface under test is a key service interface or first analyze the key service to indirectly obtain the interface accessed by the service.
2.2 Statistical analysis
If there are log analysis records for the access behavior of the interface system, directly obtain the interface with a high daily visit volume; otherwise, according to the type of interface release, select a third-party log analysis tool to obtain indirectly.
(1) IIS log analysis tool: LogParser2.2+LogParserLizardGUI
Download address: http://www.lizard-labs.com/log_parser_lizard.aspx
(2) Tomcat log analysis tool: AWStatsv7.3
Download address: http://www.awstats.org
(3) Nginx log analysis tool: GoAccessv0.9
If the interface application server such as IIS or Tomcat uses Nginx for load, the log access volume should be based on the load to avoid incorrect statistics caused by the interface setting cache in Nginx (that is, no transparent transmission).
Download address: http://www.goaccess.io
2.3 Technical Analysis
(1) Interfaces with high logic implementation complexity (such as judging too many branches or involving CPU/IO-intensive operations, etc.)
(2) Interfaces that consume high hardware resources such as system (memory, CPU, disk IO) and network IO
Remarks: If the interface is refactored due to logic modification, it needs to be re-analyzed.
2.4 Operation Analysis
An interface with a sudden increase in daily visits due to promotional activities.
Remarks: If the operation plan is adjusted, re-analysis is required.
3. Description of performance indicators
3.1 Response time
In general, the average response time of weak interaction interfaces is no more than 1 second, and the average response time of strong interaction interfaces is no more than 200 milliseconds.
3.2 Success rate
Under normal circumstances, the interface response success rate needs to reach more than 99.99%.
3.3 System resources
If it is the optimal load, the recommended range of system CPU and memory usage is [50%, 80%], otherwise it is recommended not to exceed 50%.
3.4 Processing capacity
The project application clearly requires: under XX pressure (concurrent number) the TPS must reach XX or the interface system can support XX million real-time online visits.
3.5 Stability
Under the actual system operating pressure, it can run stably for N*24 (generally N>=7) hours. It can run stably for 12 hours under the condition that the operating pressure is 1 times higher than the actual system.
3.6 Characteristic indicators
Example: Java class application FullGC times <= 1 time/day
Performance test scope
1. Business scope
Description of key business function points.
2. Design scope
The network access layer, interface layer, middleware, storage layer and other tested components and topology description.
Concurrency Calculation Method
Children's shoes who have done some performance tests are just beginning to struggle with how to calculate the number of concurrency for a certain type or type of interface. In fact, the number of concurrency can be viewed from two perspectives: user business and server.
1.80/X principle
Scope of application: unlimited
Taking a project as an example, the traffic distribution of the interface server on Mother’s Day is as follows. How to calculate the average and peak concurrent numbers on that day?
Check that the UV curve distribution on Mother's Day is linearly related to the request volume, as shown below:
Using the idea of calculus, each time point is regarded as a rectangle, and the area of the entire distribution map can be calculated by summing, as shown below:
In fact, the length of each rectangle is 1 (1 hour), so you only need to consider the width when calculating the area, that is, consider
Hourly requests are sufficient.
According to the 80/X principle, find out the time that occupies 80% of the total area, select as large a point as possible to calculate the time that occupies 80% of the total area, and find that the number of points is 7, which means that this time length accounts for 30% of the total time length %, the 80/X principle is transformed into the 80/30 principle, as follows:
Therefore, the average number of concurrency (average number of requests per second) = 80% * daily request volume / 1 day * 30%
Then calculate the multiple of the highest peak value and the average concurrency = 2.25
Therefore, the peak concurrent number (peak number of requests per second) = 2.25 * average concurrent number =
2.25*80%*daily request amount/1 day*30%=6*daily request amount/1 day
Since the distribution of UV and request volume curves is linear, daily request volume=9.25*daily UV
Therefore, the peak concurrent number = 6*9.25* day UV/1 day = 55.5* day UV/1 day
2. Formula method
Scope of application: Web class access
Formula (1) calculates the average number of concurrent users: C=n*L/T
C is the average number of concurrent users; n is the number of loginsessions; L is the average length of loginsessions; T refers to the length of the investigation period.
Formula (2) calculates the peak number of concurrent users: C'≈C+3root C
C' refers to the peak number of concurrent users, and C is the average number of concurrent users obtained in formula (1). The formula is derived by assuming that the user's loginsession is generated according to the Poisson distribution and estimated.
Example 1:
Suppose there is an OA system with 3000 users, and about 400 users access the system every day on average. For a typical user, the average time from logging in to logging out of the system within a day is 4 hours. During a day, users only use the system for 8 hours.
C=400*4/8=200
C'≈200+3*root 200=242
In order to better understand the above formula, it is converted into the following formula:
Formula (3) Number of concurrent users = throughput rate * scenario business time / unit time period
Example 2:
In an OA system, 8000 users log into the system within 1 hour. Every time the user logs in to the system, he needs to start the login page, and then enter the user name and password to enter the home page. Generally, it takes 5 seconds for the user to perform the above operations, and it is required to control the time from clicking the login button to the full display of the home page within 5 seconds.
analyze:
Throughput rate=8000*2 (the entire business operation needs to load the page twice to complete)
Scenario business time=5+5=10 seconds
Unit time period = 1 hour = 3600 seconds
Number of concurrent users (login scenario) = (8000*2)*10/3600=45
After obtaining the number of business concurrency through the above method, we can further analyze which interfaces are accessed by the business. We only need to simulate the calling methods and timing of these interfaces.
Sometimes we need to calculate the concurrency number of a certain or a certain type of interface, we can analyze and calculate according to the following steps:
(1) Sort out the business scenarios where the interface under test is accessed and the number of times each business scenario is accessed
(2) Calculate the number of concurrent users in the business scenario by the above method
Concurrent interface number = number of concurrent users in scenario 1 * number of interface calls in business scenario 1 + number of concurrent users in scenario 2 * number of interface calls 2 +...
If a system needs to support access by 100,000 online users, how to calculate the number of concurrent users through performance requirements analysis? You can learn through the above content and think independently?
Performance test cases and scenarios
script template
scene template
Performance testing tool selection
1. Data Modeling Tools
DataFactory is a powerful data generator that allows developers and QA to easily generate meaningful and correct test databases with millions of rows. This tool supports DB2, Oracle, Sybase, SQLServer databases, and supports ODBC connection methods, which cannot be used directly MySQL database can be indirectly supported.
2. Script development tools
(1) If script operation efficiency is considered, the underlying development language C or the language JS that supports asynchronous communication can be considered. We can choose: Loadrunner or Node.js IDE environment for development.
(2) If script development efficiency is considered, code reusability can be considered, and object-oriented language C# or Java can be selected. For this reason, we can choose: VS2008 and above + corresponding LR.NET control or Eclipse4.0 and above +JDK1.7 and above.
For the development process of protocol interface performance test scripts such as HTTP and Socket, please refer to the attachment for details:
Script development and performance problem analysis of HTTP interface performance test.pdf
Use LR.Net control to complete performance test script writing method.pdf
Node.js Learning Getting Started Manual.pdf
3. Pressure simulation tools
(1) If it is a Java class interface and the number of concurrency on a single machine is controlled within 500, you can choose Jmeter or Loadrunner.
(2) If it is a WebService interface and the number of concurrency on a single machine is controlled within 500, you can choose SoapUI or Loadrunner.
(3) If the number of concurrency on a single machine exceeds 500 and is controlled within 5000, you can choose Loadrunner.
(4) If the number of concurrency on a single machine exceeds 5000, it is recommended to use a load cluster, that is, to adopt the "ControlCenter + LoadGenerator" solution.
4. Performance monitoring tools
4.1 Monitoring tools
Regardless of the Windows or Linux platform, there is generally one or a group of process instances, and we can choose Loadrunner or Nmon to monitor. Sometimes in order to obtain some characteristic indicators of the application under test, you can choose the performance tool set or monitoring system that comes with the component under test. Common application server monitoring tools are recommended as follows:
4.2 Monitoring Platform
The monitoring machine mainly monitors the service or resource usage of the cluster server under test, such as various open source monitoring tools, MRTG: traffic monitoring; CACTI: traffic warning, performance report Smokeping: IDC quality monitoring; comprehensive monitoring: Nagios, Zenoss, Ganglia, Zabbix, Sitescope, HypericHQ, etc., as follows:
4.3 Third-party monitoring cloud service (APM)
APM provides end-to-end application performance management software and application performance monitoring software solutions, including mobile, browser, application, infrastructure, network, database performance management, etc., supporting Java, .NET, PHP, Ruby, Python, Node.js , iOS, Android, HTML5 and other application performance monitoring and management, mainstream cloud services include Tingyun, OneAPM, etc.
Analysis of performance test results
1. Index analysis
Performance testing indicators can be divided into two categories: product indicators and resource indicators. For testers, the requirements for performance testing come from three aspects: users, development, and operation and maintenance. Users and developers focus on product indicators related to business requirements, while operation and maintenance personnel focus on resource indicators related to hardware consumption.
(1) Indicators concerned from the perspective of users
Users pay attention to the experience effect related to a single business, such as the response speed of an operation, whether a request is successful, whether a connection fails, etc. The indicators reflecting a single business include:
a. Success rate b. Failure rate c. Response time
(2) Indicators concerned from the perspective of development
Developers pay more attention to system-level indicators.
a. Capacity: What is the maximum user visit volume that the system can carry? What is the maximum business processing volume of the system?
b. Stability: Whether the system supports 7*24 hours (one week) business access.
(3) Indicators concerned from the perspective of operation and maintenance
Operation and maintenance personnel are more concerned about the consumption of hardware resources.
The above shows that testers need to think from the perspective of users when selecting indicators. In addition, in order to better analyze problems in the future, it is necessary to master performance indicators related to the characteristics or operating principles of the tested components.
For example, usually the interface system will directly or indirectly access the database layer medium (such as Mysql, Oracle, SQLServer, etc.), at this time we need to consider the performance of the storage medium under the pressure generated by the interface system, usually we will choose the analysis index as follows:
(1) Number of connections (Connections)
(2) Queries per second/Transactions per second (QPS/TPS)
(3) Number of disk IOs per second (IOPS)
(4) Cache hit rate (BufferHits)
(5) Number of deadlocks per second (DeadLocks/sec)
(6) Read/write bytes per second (Read/WriteBytes/sec)
For specific indicator monitoring and analysis methods on Windows or Linux platforms, please refer to the attachment:
"Windows operating system performance monitoring tools and indicators analysis V1.0".pdf
"Linux Operating System Performance Monitoring and Analysis Manual V1.0".docx
2. Modeling analysis
2.1 Barbershop model
Shown in the figure is a standard software performance model. There are three curves in the figure, which respectively represent resource utilization (Utilization, including hardware resources and software resources), throughput (Throughput, here refers to the number of transactions per second) and response time (ResponseTime). The horizontal axis of the coordinate axis in the figure shows the continuous growth of the number of concurrent users (Number of ConcurrentUsers) from left to right.
In this figure, we can see that at the beginning, as the number of concurrent users increases, the resource occupancy rate and throughput will increase accordingly, but the response time will not change much; but when the number of concurrent users increases to a certain extent, the resources Occupancy reaches saturation, throughput growth slows down significantly or even stops, while response time is further extended. If the number of concurrent users continues to increase, you will find that the hardware and software resource usage continues to remain saturated, but the throughput begins to decline, and the response time obviously exceeds the user's acceptable range, which eventually causes the user to give up the request or even leave.
According to this performance, three areas are divided in the figure, namely LightLoad (lighter pressure), HeavyLoad (heavier pressure) and BuckleZone (users can't bear it and give up the request). The number of concurrent users at the junction of LightLoad and HeavyLoad is called "The Optimum Number of ConcurrentUsers", and the number of concurrent users at the junction of HeavyLoad and BuckleZone is called "Maximum number of concurrent users ( The Maximum Number of Concurrent Users)".
When the system load is equal to the optimal number of concurrent users, the overall efficiency of the system is the highest, no resources are wasted, and users do not need to wait; when the system load is between the optimal number of concurrent users and the maximum number of concurrent users, the system can continue Work, but the user's waiting time is prolonged, and the satisfaction begins to decrease, and if the load continues, it will eventually cause some users to be unbearable and give up; and when the system load is greater than the maximum number of concurrent users, it is destined to cause some users to fail Put up with the long response time and give up. So we should ensure that the optimal number of concurrent users is greater than the average load of the system.
2.2 Pressure change model
As the traffic per unit time continues to grow, the pressure on the system under test will continue to increase, server resources will continue to be consumed, and the TPS value will change due to these factors, and it conforms to certain rules.
In the picture:
Point a: performance expectations
Point b: higher than expected, system resources are at a critical point
point c: higher than expected, inflection point
Point d: Over load, system crashes
2.3 Capacity Calculation Model
Take a website performance test as a case:
-
By analyzing the operational data, you can know the number of PVs processed by the current system per hour
-
Through the load test, you can know the maximum number of PV processed by the system per hour
That is, tidy up
Remaining amount of PV processed by the system per hour = maximum PV processed by the system per hour - PV processed by the system per hour
Assuming that the user load of the website basically increases linearly, and the number of existing system users is 700,000, according to the operation and promotion plan, the number of users of the website will reach 10 million within one year, which is an increase of 14 times. That is, sorted out:
The increase in PV processing per hour of the system = the number of PVs processed by the current system per hour * 14 - the number of PVs processed by the current system per hour
Daily system load increase rate = 100%/365 = 2.74% (Note: Here, the load of 10 million system users in the future is defined as 100%)
Daily PV processing increase of the system = system PV processing increase per hour * daily system load increase rate * 24
So, we can know that under normal load conditions:
The number of days that the system can support normal operation = the residual amount of PV processing per hour of the system * 24 / the daily PV processing increase of the system
Assume that the number of days for the subsequent deployment and upgrade of the website is known, so that we can know the number of days for the upgrade in advance:
The number of days the system can support uptime—the number of days to deploy upgrades.
performance test pass standard
-
All planned testing has been completed.
-
All performance data planned to be collected has been obtained.
-
All performance bottlenecks are improved and meet design requirements.
As someone who has been here, I also hope that you will avoid some detours. Here I will share with you some necessities on the way forward for automated testing, hoping to help you. (WEB automated testing, app automated testing, interface automated testing, continuous integration, automated test development, big factory interview questions, resume templates, etc.), I believe it can make you better progress!
Just leave [Automated Test] [Automated Test Communication]: 574737577 (remark ccc) 
http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=6gweEAHLIo-VjldEz9_yMjFfeBIGgEu2&authKey=MC4ik2bydrYxO5if1oNFzT6c93XUrxOxYtv7IRY OVRDNh47xpEllgkFVDlFD%2Bf6M&noverify=0&group_code= 574737577
