In-depth talk about performance testing, from getting started to giving up: a first look at performance testing

I originally wanted to divide it into 7 articles to write, but I also considered that I had to turn the pages when I saw the deep feelings, so I simply composed one article to write.
Before writing the text, to share a little story:
People:
Small Cock wire: Test Manager
Gangster: Product Manager,
Small Cock wire xx recently doing a project, the project immediately knot, big brother suddenly asked to do performance testing Small Cock wire.
As the saying goes, to do a project, you must have a standardized process and a project plan.
Xiao Diaosi looked dazed. After 3 seconds, Xiao Diaosi sent out 4 questions~ ~
Xiao Diaosi: Big brother, first of all, we evaluate at the beginning of the project, whether this project does not require performance testing, and secondly, does the customer require performance reports? , Thirdly, does the contract provide software and hardware equipment information, performance expectations and other information, and fourthly, do we need to go to the customer site for deployment?
Big Brother: Um...
Xiao Diaosi: Then, do we perform performance testing to reflect the completeness of the report, or to reflect our team's bullchat?
Big Brother: I just want the performance test report!
Little dick silk:...

Hey! Let’s sigh ~ ~ Let’s
get back to the subject, since as a professional performance test engineer, we must give the project leader/customer the most reasonable and optimal solution. Don’t laugh at them just because they don’t understand.
After all, every other line is like a mountain~~~

Next, follow Xiaoyu, learn in detail, how to do performance!
Xiaoyuer will let you, from getting started to giving up, is an instant~ ~

1. Test process

To quote the words of Mr. Lu Xun: There is no rule without rules.

Of course, our performance test also needs a process specification, and it is in line with the project management process. You say handsome or not~~
Let’s take a look at this process specification

With the above flow chart, next, we will introduce each link in detail:
①Business learning : to understand the function of the system by viewing documents or through actual operations.
②Requirement analysis : Analyze the non-functional requirements of the system, define the scope of performance testing, and understand system performance indicators.
③Work evaluation : work load breakdown, work load evaluation, planned resource input (person, day).
④ Design test model : After delimiting the performance test scope, map the business model to the test model.
Are you confused when you hear this sentence? What is a test model?
In layman's terms, it is: performance test case design + performance test plan
use cases only focus on business, the model also needs to focus on how to implement, whether it is operability, verifiability, etc., and finally, it has to be combined into test scenarios according to different test purposes .
⑤ Plan preparation : plan the test work, clearly list the test scope, manpower input, duration, work content, risk assessment, risk response strategy, etc. in the document.
⑥Script development : record or write performance scripts, develop and test baffle programs, and develop test programs. Even if there is no third-party tool, it is necessary to develop and test the program after the third-party tool.
>Baffle program: We develop our own program to replace the support of a third party.
>> For example, if we need to interact with the aviation system, but the airline does not provide technical support, we have to develop our own program instead.
⑦Test environment preparation : The performance test environment includes two parts: server and load machine.
Server: the operating platform of the system under test (software, hardware)
Load machine: The machine we use to generate load, install load tools, and run scripts.
⑧Test data preparation : prepare the master data and business data of the tested system according to the data model.
⑨ Test execution : This step is the key to success or failure, because different people, design scenarios, and test execution, so the results may be quite different.
⑩Defect management : defects found in the performance test process
⑪Performance analysis : analyze the problems that may be exposed by the performance test results to find out the reasons.
>>For how to perform performance analysis, please see Xiaoyu’s " Performance Analysis Process ".
⑫Performance tuning : This requires collaboration. The performance tester and the development leader should work together to solve performance problems
>>How to perform performance tuning, Please see Xiaoyu’s " How to do performance tuning "
⑬Test report : a summary of the test results.
>>Common performance indicators include: TPS, RT, CPU Using, etc.
⑭Report review : Review the content in the performance report, confirm problems, and evaluate the online/delivery risk.

2. Success or failure factors

Xiaoyu remembers that the article on performance analysis said: If you compare each link of the project to various disciplines, then performance testing is a comprehensive one, integrating requirements, development, testing, operation and maintenance, and coordination and communication.
Let's talk about the major difficulties of performance testing from the following points:
① demand analysis
② scenario design
③ performance diagnosis and tuning
④ environment construction and simulation

Many performance testers did not analyze the requirements in place during the requirements analysis stage, and did not accurately estimate user behaviors. They could not simulate user operations well in the scene, and could not truly simulate the load situation of the system. The
final result is: in the test or The UAT environment is running very well. After going online, problems occur frequently, and the results can be imagined.
Just like: the
2008 Olympic Games,
11.11 2012, a treasure,
11.11 2012, a certain Dong,
12306 on January 9,
2014, and Amazon website on November 6 , 2014.

For laymen, or some big performance testers, they think that performance testing is to use a tool or write a script, get a few servers, "run", and then issue a report, it's ok.
Usually pay attention to how much concurrency, how much response time, can run through and so on.
If the performance test is really that simple, are you worthy of the words from getting started to giving up ~~~
Next, let’s take a look at the important points of performance testing:
1. Evaluate the system, and what needs analysis
does not clearly require the customer? Value,
we need to guide the operation and maintenance leaders and demand leaders to give specific demand data, and conduct a secondary analysis of the data to obtain the information we need.
For the first-time online system: We need to use peer data to analyze user behavior and estimate business data structure as a prerequisite to perform calculations to get the performance requirements we want.
For the online system: we can obtain TPS and time ratio distribution diagram, user number and time distribution diagram, database ER relationship diagram, capacity data, etc. through operation and maintenance personnel, and directly and accurately obtain the current system user behavior and business Data relationship, and then get the performance requirements we want.
2. Scenario design and use case design
We must simulate the load of the online system as realistically as possible.
We have to decide which functions need to participate in performance execution, and how to participate? This is use case design.
How to effectively organize test cases is what the scenario needs to do, such as comprehensively assigning the number of users, execution time, and execution ratio according to business distribution, business volume, business period, and business role.
This scenario design and use case design takes a lot of time.
3. Test execution, whether pass
simulates different load execution test scenarios to identify system weaknesses, do various monitoring, screen various problems, verify system stability, etc.
4. Performance diagnosis optimization
The leader of performance testing needs to have a good and keen awareness of performance, be able to refine the problem (that is, step by step), and assist each development team in problem positioning, analysis and optimization.

Three, big guys look at performance

Since performance is a comprehensive subject, let’s take a look at these big guys’ requirements for the system from different perspectives:
1. Black box testing guys’ perspective: they
care about the single-step response time and performance of the application Whether it is good or bad, it depends on the application time, that is, the total round-trip time of the data stream after the server/server cluster is transmitted over the network.
2. From the perspective of development bosses
① rationality of architecture
② rationality of database design
③ code
④ memory usage in the system
⑤ thread usage in the system
⑥ whether system resources are vicious and unreasonable competition
3. Operation and maintenance boss perspective
① hardware resource utilization Rate
②JVM ③DB
④Does the
system support 7x24 services ⑤Scalability
, compatibility, maximum capacity, possible bottlenecks
4. Performance test from the perspective of ①Server
hardware performance ②Performance goals
based on demand or historical data
③Establish performance passing model
④Yes Develop the code framework and hardware framework for performance analysis
⑤ Perform benchmark tests for the development and release version
⑥ Perform software performance acceptance and stability testing
⑦ Proportion and optimization of the production environment
⑧ Perform performance test cases and scenario design
⑨ Coordination between the part
⑩ Specific performance analysis

Four, related terms

1. Load :
Simulate the process of business operations that put pressure on the server, such as 2000 users placing orders.
2. Performance Testing:
Simulate user load to test whether the system's response time, throughput and other indicators meet performance requirements under load.
3. Load Testing (Load Testing)
in a certain software and hardware environment, by continuously increasing the load (the number of different virtual users) to determine the maximum number of users that can withstand the performance indicators.
>>The performance indicators here include: TPS (transactions per second), RT (average transaction response time), CPU Using (CPU usage), Mem Using (memory usage) and other hardware and software indicators.
4. Configuration Testing
In order to rationally allocate resources and improve system operation efficiency, the process of obtaining, verifying, and adjusting configuration information through testing methods.
5. Stress Testing (Stress Testing)
in a certain software and hardware environment, through high load means to make the server's resources (emphasizing server resources, hardware resources) in a limit state, the test system is in the limit state for a long time Is the operation stable?
>>Determine stable indicators, including: TPS (transactions per second), RT (average transaction response time), CPU Using (CPU usage), Mem Using (memory usage), etc.
6. Endurance Testing
In a certain software and hardware environment, run a certain load for a long time to determine whether the system is running stably under the premise of meeting performance indicators. It is emphasized here that the stability test is not run under the limit state, which is different from the pressure/strength test in Article 5 above.
Generally, we will increase the load to 1.5~2 times for testing under the load that meets the performance requirements.
7. The
number of transactions completed by TPS per second, usually refers to the number of successful transactions per second, an important comprehensive performance index in performance testing. A transaction is a unit of business measurement.
For example: For
payment operations, the back-end system may experience membership systems, financial systems, payment systems, accounting systems, banking gateways, etc. However, for users, I want to know how much I spend from order to payment Long time.
8. RT/ART (Response Time/average Response Time)
response time/average response time refers to how long it takes for a transaction to complete. More often, we count the response time many times, and then average it, which ensures the accuracy of the time and is also more representative.
>>Usually, we write RT instead of ART.
9.PV (Page View)
The number of times the user visits the page per second.
>>This parameter is used to analyze the average number of users visiting the page per second.
10. Vuser (Virtual user)
number of virtual users. That is, the virtual user that simulates the logic steps of the real business, and the operation steps of the virtual user simulation are recorded in the virtual user script. The Vuser script user describes the operations performed by the Vuser in the scenario.
11. Concurrency: There are
two main points for concurrency: point level and line level.
①On the point level:
For example, at 7:30 on Monday morning, elementary school students must go to the playground to raise the national flag.
>>That is: do something at the same time ②On the
line level:
For example: 11:30-13:00 noon, some elementary school students jump rubber bands, some play football, but at the same time exert pressure on the server.
>>That is: do different things in one time period. I will
not introduce too much here. For details, please refer to the " Common Concurrency Problems " written by Xiaoyu , which contains a detailed interpretation of concurrency.
12. In the Scenario
performance test process, in order to simulate the business processing process of real users, a collection of actions based on transactions, scripts, virtual users, running settings, running plans, monitoring, and analysis built in LR is called This is the performance test scenario.
>>The scenario includes: scripts to be executed, script groups, concurrent numbers, load generators, test targets, configuration conditions during test execution, etc.
13. Think Time (Think Time)
simulates the time interval that real users pause during actual operations.
>>From a business perspective: thinking time is the time interval between each request when the user is operating;
>>From the test script: it is the time interval between two request statements.
14. The standard deviation (Std. Deviation)
is derived from the concept of data statistics. The smaller the standard deviation, the smaller the fluctuation and the more stable the system. On the contrary, the more stable the system.
>>Includes: response time standard deviation, TPS standard deviation, Running Vuser standard deviation, Load standard deviation, CPU Using standard deviation, etc.

Five, performance tools

There are quite a few performance testing tools on the market, but we are familiar with and use the most used JMeter and Loadrunner. Then, how do we choose performance testing tools and what performance testing tools are there?
Don't worry, we will introduce it slowly.
1. How to choose performance testing tools When
we choose tools, we can only consider the following aspects:
①Professional, stable and efficient.
②Simple and easy to use, no need to spend too much time on test scripts.
③With technical support and sound documentation.
④The ratio of input to output.
2. What are the common performance tools
①Loadrunner of HP Company
②Apache JMeter
③Grinder ④QALoad of
CompuWare Company
⑤WAS of Microsoft Company
⑥WebLoad of RadView
Company ⑦RPT of IBM Company
⑧OPENSTA etc.
Here, Xiaoyu emphasizes that the core of performance testing is not which tool is selected, but It is performance analysis . The important thing is thought and implementation. It has nothing to do with the choice of tools,
so we must analyze the priority.

Six, pass the standard

To judge whether the performance test passes, not only depends on TPS and RT, but also analyzes from the server performance, front-end performance, and user experience performance.
Common test passing standards are as follows:

Seven, summary

I have shared with you the basic situation of the performance test. I believe everyone can understand and digest these postures well. No, it is knowledge.
Performance testing requires a lot of requirements, not only professional skills, but also communication skills.
In order to better learn and master performance testing, I also think you need to improve yourself from an analysis perspective. After all, the core point is performance analysis.