In performance testing there is a type of testing called benchmarking . In this article, let’s talk about some things about benchmarking.
1. Definition
By designing a reasonable test method, selecting an appropriate test tool and the system under test, a quantitative and comparable test of a certain performance index of a specific target scene can be realized.
2. Traits
①. Repeatability: Repeatable tests can be performed, which is conducive to comparing each test result and obtaining the long-term change trend of performance results, which can be used as a reference for system optimization and capacity planning before going online.
PS: This idiosyncrasy is for the daily polling needs of benchmarks.
② Observability: Through all-round monitoring (including test start to end, execution machine, server, database ), understand and analyze what happened in the test process in a timely manner.
③. Displayability: Relevant personnel can understand the test results intuitively and clearly ( web interface, dashboard, line graph tree diagram, etc.).
④. Authenticity: The test results reflect the real situation experienced by customers (real and accurate business scenarios + configurations consistent with production + reasonable and correct testing methods).
⑤. Executability: Relevant personnel can quickly test, verify, modify, and optimize (positionable and analyzable).
3. Preconditions
Benchmarking must be done under controlled conditions.
In the face of increasingly complex systems, increasing number of users, and multiple business systems that may be involved in performance testing, only when the business scenarios, system architecture, and test environments involved in benchmark testing are under control can we obtain Relatively accurate results provide reference and basis for capacity planning, defect location, and system tuning.
4. Significance
①. Determine system and application limits for capacity planning;
②. Provide a reference basis for the parameters and configuration options of the configuration test;
③. Determine whether the system has the capabilities it claims for acceptance testing;
④. Provide long-term data statistics sources and comparison benchmarks for the establishment of performance baselines.
5. Premise
①. Test purpose: clarify the purpose of the test, what is the test? What testing methods and strategies are used?
②, Test environment: What is the environment of the system under test, SIT or UAT or PAT?
③. Test limitations: What are the limiting factors to perform the test, and how to solve them?
④. Risk factors: What risks may exist in the test, and what are the solutions?
⑤. Result analysis: How to analyze the test results? How to analyze and locate the data generated by the test?
6. Principles
①. Test strategy: Stable and continuous workload, run multiple times, look at the normal distribution trend of the test result data, and try to take the average;
②. Data statistics: the average value and peak value of the test data in the real environment, and the dimension of the value;
③ Differential risks: clarify which risks exist, and whether the impact of risks on test results is ignored;
④. Special circumstances: What are the special circumstances, and whether there are corresponding solutions (such as payment service calls in payment scenarios, whether to use baffles, etc.).
7. Factors to be considered
Transaction ratio: In some business scenarios, a process contains multiple transactions, and in the simulated concurrency, the respective proportions of different transactions;
Sudden read and write operations: In some special business scenarios, there will be a short-term large traffic impact or a sudden drop in the number of requests, how to simulate (surge test);
System configuration: The system configuration of different environments is different, how to convert and compare the test results?
Test duration: during test execution, how long it takes to run, the time allocation of different transactions, etc.;
Result display type: How to display the average value, peak value, and percentage value, and how to compare them?
Success/failure ratio: statistics on the ratio of successful and failed transactions during each test;
Is it reproducible: If an error or some abnormal situation occurs during the test, can it be reproduced?
Is it comparable: Are there other test tools or test results for comparison (try to perform the test as many times as possible to compare the test results: learn about standard deviation and normal distribution?)
8. Simple and feasible method
Gradually increasing the load on the system is an easy way to determine the maximum throughput a system can handle and a viable strategy for finding corners in system performance (step stress testing).
9. Focus
The focus of benchmarking is statistical analysis: statistics can be carried out from the following dimensions:
①. Select appropriate test tools, set reasonable test methods and system performance indicators that need to be confirmed;
②. Choose different test tools, compare the test results, and choose a stable and reflective system that is really the result of performance;
③. Execute tests multiple times and collect a large number of test data sets and indicators;
④. Interpret and analyze data from different dimensions and generate reports.
Finally, I would like to thank everyone who has read my article carefully. Reciprocity is always necessary. Although it is not a very valuable thing, you can take it away if you need it:
These materials should be the most comprehensive and complete preparation warehouse for [software testing] friends. This warehouse has also accompanied tens of thousands of test engineers through the most difficult journey, and I hope it can help you! Partners can click the small card below to receive