Table of contents
When does performance testing start?
1. Formulate performance testing objectives
2. Acquisition of performance test scenarios
3. Determination of performance test data
4. Performance test case design
5. Preparation and construction of performance test environment
11. Drawing and writing reports
When does performance testing start?
Generally, it starts to execute after the system function is stable and there are no major defects. However, the preparatory work can start from the analysis of system requirements: performance target formulation, scene acquisition, environment application, etc.
1. Formulate performance testing objectives
Test the response time of a specific scenario under a specific number of concurrent users
Test the maximum number of concurrent users in a specific scenario under a certain response time requirement
Test the TPS of a specific scenario
1. Online system
Analyze the logs of the online system to obtain the access status of each function of the system, the maximum number of concurrent users, and the average/maximum/minimum response time. Then use the daily growth trend to determine the maximum number of concurrent users, and the response time refers to the results of log analysis, which is equivalent to the average response time.
2. New project
Documentation related to the development process
Documents such as project development plans, requirements specifications, and design specifications may all involve requirements for performance testing. By collecting these materials, preliminary performance requirements can be found. However, these performance test requirements are often not accurate enough and require professional guidance from performance testers.
similar items
Other products or past projects of the company will accumulate some data, which can be used as a reference.
user usage model
Analyzing user usage models is an effective means to obtain performance testing requirements, considering which users use which typical services of the system, and how many users perform which functions at what time. For example: in an OA system, 200 users will log in to the system within 10 minutes at 8:00 every morning; the peak of query transactions every day is 9:00~11:00 and 14:00~16:00 in the afternoon, etc., and then according to This user uses the model and combines the 80/20 principle to calculate the concurrent volume of OA system login and transaction query business.
80/20 principle
The 80/20 principle is that 80% of the business of the system is completed in 20% of the time every working day, or 80% of the users of the system will concentrate on application operations in 20% of the time. Let's take two examples to illustrate:
(1) A website has a total of 100,000 daily visitors, of which 30% are browsing product pages, 20% are search services, and 50% are login + purchase services. Using the 80-20 principle, 20% of 8 hours is used as the benchmark time to calculate the concurrent number of each business.
Search business: (100000*20%*80%)/(8*3600*20%)=2.78 rounded to 3
Browse the single product page: (100000*30%*80%)/(8*3600*20%)=4.17 rounded to 5
Login+purchase: (100000*50%*80%)/(8*3600*20%)=6.94 rounded to 7
(2) The annual business of the system is concentrated in 8 months, with an average of 20 working days per month, and each working day is 8 hours. According to the 80/20 principle, 80% of the business is completed in 1.6 hours per day. Last year, about 1 million business transactions were processed, of which 15% of the business processing needed to submit 7 requests to the application server for each business, of which 70% of the business processing needed to submit 5 requests to the application server for each business, and the remaining 15 In % business processing, each business needs to submit 3 requests to the application server. According to previous statistical results, the annual business growth is 15%. Considering the business development needs in the next three years, the test needs to calculate the TPS that the system should achieve based on the number of requests twice the current business volume.
The total number of requests per year = (1 million*15%*7+1 million*70%*5+1 million*15%*3)*2=10 million
TPS=(10000000*80%)/(8*20*8*3600*20%)=8.68, TPS=9
response time standard
Within 2 seconds, the user feels good
2~5 seconds, the user finds it acceptable
5~10 seconds, the user will feel very irritable, unable to receive, and will frequently refresh the page
For more than 10 seconds, the user cannot receive at all and leaves directly
Performance test engineer video tutorial: 2023 latest detailed explanation of the whole process of real enterprise performance test project, the kind that can be written into the resume interview_哔哩哔哩_bilibili 
https://www.bilibili.com/video/ BV1PW4y1R7ye/?spm_id_from=333.999.0.0
2. Acquisition of performance test scenarios
1. Online system
Single scene:
According to the log analysis results of the online system, the functions with the highest number of visits, the functions that are changed and may be affected by this change, and the functions related to money. To be on the safe side, it is best to confirm with the developer the functions that will be affected.
Mixed scene:
Still based on the log analysis results of the online system, the maximum number of concurrency at the system level is obtained, and then an increment is made according to the daily growth trend to obtain the final maximum number of concurrency. Then assign users according to the proportion of each important function in the log analysis results.
Stability scenario:
After determining the single scene and the mixed scene, the stability scene should also be considered. Its purpose is to test whether the system has memory leaks, and it can also test the mean time between failures of the system. Therefore, long-term stability tests can be done with mixed scenes.
2. New project
Single scene:
Important, core functions
Common Functions
Functions with complex business processes
Resource-intensive functions (such as multi-table query or inserting data into multiple tables)
Mixed scene:
Add all important functions to the mix scene according to a certain ratio
Stability scenario:
Consider using mixed scenarios for long-term stability testing.
3. Determination of performance test data
A very important point in performance testing is the design of scene data. For example, in a data query scenario, if the database table corresponding to the scenario has only 10 pieces of data, then the query result must be relatively fast. However, if the database table corresponding to this query scenario has 10 million pieces of data, the query result will definitely be slower than the query result with only 10 pieces of data. If the performance test does not consider the amount of data, the results of the performance test will be inaccurate, and there will be a high probability of performance problems caused by factors that do not consider the amount of data after going online.
For the online system, the data volume of each table can be determined according to the data volume and increment of each table in the online system. The new system needs to be determined according to the development documents and relevant project stakeholders (such as: customer representatives, project managers, requirements analysts, system architects and product managers)
4. Performance test case design
1. Single scene
Scenario description: simulate a user to log in
Concurrency: Simulate the number of concurrent users as 1, 10, and 50 respectively for testing
Pressure test time: 15 minutes each time
Data volume: There are 700,000 accounts in the MySQL user table
Rendezvous: no rendezvous
Focus on indicators: response time, transaction success rate, application server resource usage (CPU, memory, IO), MySQL database resource usage (CPU, memory, IO), whether there are errors such as deadlock in the application log, whether there is any error in the database log Errors such as deadlocks, JVM memory usage and GC conditions
Expected indicators: response time within 2 seconds, transaction success rate 100%, application server and database server CPU usage ≤ 60%, no memory leaks, database deadlocks, thread deadlocks, etc.
2. Mixed scene
Mixed scenarios do not combine all the test scenarios together to form a large scenario, but should consider different combinations of mixed scenarios first, such as mixed scenarios of database query operations, mixed scenarios of database write operations, and database query and write operations. Great mix scene. as follows:
Scenario description: Simulate a mixed scenario where the system does not require users to perform database read and write operations. The scenarios include user login, ad word default query, new ad group, ad word default creation, ad review, ad validation, and ad words sorted by price.
Concurrency: A total of 300 users are simulated to operate at the same time, of which login operations account for 20%, default query for advertisements accounts for 25%, new ad groups account for 15%, default creation of advertisements accounts for 8%, review of advertisements accounts for 10%, and advertisements take effect 15%, ad words sorted by price 7%
Pressure test time: 15 minutes each time
Data volume: MySQL's cpc table has 1.5 million data, plan table has 100,000 data, group table has 500,000 data, audit table has 1 million data, MongoDB report table has 1TB data, user table has 900,000 Article data.
Rendezvous: no rendezvous
Focus on indicators: response time, transaction success rate, application server resource usage (CPU, memory, IO), MySQL database resource usage (CPU, memory, IO), whether there are errors such as deadlock in the application log, whether there is any error in the database log Errors such as deadlocks, JVM memory usage and GC conditions
Expected indicators: The response time for operations such as login, default query of ad words, and new ad group is within 2 seconds; the response time for operations such as default creation of ad words, ad review, ad activation, and ordering of ad words by price is within 3 seconds; the transaction success rate 100%, application server and database server CPU usage ≤ 60%, no memory leaks, database deadlocks, thread deadlocks, etc.
3. Stability scenario
Scenario description: Simulate a mixed scenario where the system does not require users to perform database read and write operations. The scenarios include user login, ad word default query, new ad group, ad word default creation, ad review, ad validation, and ad words sorted by price.
Concurrency: A total of 300 users are simulated to operate at the same time, of which login operations account for 20%, default query for advertisements accounts for 25%, new ad groups account for 15%, default creation of advertisements accounts for 8%, review of advertisements accounts for 10%, and advertisements take effect 15%, ad words sorted by price 7%
Pressure test time: last 2*24 hours
Data volume: MySQL's cpc table has 1.5 million data, plan table has 100,000 data, group table has 500,000 data, audit table has 1 million data, MongoDB report table has 1TB data, user table has 900,000 Article data.
Rendezvous: no rendezvous
Focus on metrics: JVM memory usage and GC
Expected Metrics: No memory leaks or signs of occurrence
Performance test engineer video tutorial: 2023 latest detailed explanation of the whole process of real enterprise performance test project, the kind that can be written into the resume interview_哔哩哔哩_bilibili https://www.bilibili.com/video/ BV1PW4y1R7ye/?spm_id_from=333.999.0.0
5. Preparation and construction of performance test environment
The performance testing environment includes software environment, hardware environment and network environment. These three environments not only refer to the application server environment, but also include database servers, cache servers, file servers, and other intermediate application server environments.
The hardware environment includes: CPU, memory, disk and other basic factors.
The software environment includes: operating system version number, configuration, Linux disk partition, JDK version, bit number, manufacturer, middleware version number, bit number, database version number, bit number, and the installation paths of these software are also best compared with online The environment is consistent. Configuration files include JVM configuration, middleware configuration, database configuration files, etc.
Network environment includes: network protocol and network bandwidth.
The cluster environment includes: load balancing environment of application-related servers, hot standby or master-slave environment of database, cluster environment, etc.
When applying for an offline simulation test environment, the following principles should be followed:
(1) The hardware environment is as consistent as possible with the production environment
(2) If it is a cluster environment, it is impossible to apply for so many servers in the test environment, so you can consider applying for 3 machines that are consistent with the online production environment as offline performance test machines. In the process of performance testing, you can test the performance of single-machine, double-machine and three-machine load balancing respectively, and then calculate the performance loss of the online production environment (for example, 100 machines) for load balancing according to the performance in the three cases Rate, so as to more realistically calculate the performance index when 100 machines on the line are load balanced.
(3) If the database cluster environment is too large, for example, the database has 8 groups of 32 machines, then offline testing will not apply for 32 machines for performance testing. Generally, in this case, only a set of databases (one master and three slaves) will be applied as the database for performance testing. Because the clusters of large databases basically adopt the strategy of splitting databases and tables, it will lead to huge database clusters. The performance test can be carried out by applying for a set of database machines. It is only necessary to ensure that the user data used for the performance test falls within the applied set of databases.
(4) If it is impossible to ensure that the hardware environment is consistent with the online environment, then only the low-configuration environment can be used for testing. If the results measured in the low-configuration environment can meet the online requirements, then the online high-configuration environment must also meet the established requirements. performance requirements. If it cannot be satisfied, modeling estimation is not recommended, because if the number of CPU granules, cache, physical memory size, and disk speed are different, the performance results obtained by performance modeling will not be accurate enough. If the test on a low-configuration machine fails to meet the requirements, the test environment must be stated in the test report, and it cannot be guaranteed that the test environment will meet the requirements due to the improvement of the test environment.
Mock Server preparation:
It is called Mock Server in the Internet industry, and it is called performance test baffle in banking and other financial industries. Sometimes the business joint debugging of the system needs to call the interface of other systems, but the development of other systems has not been completed. For this situation, a common solution is to build a temporary server, simulate those services, and provide data for joint debugging and testing. The use of Mock Server usually brings the following benefits:
(1) Isolate the test errors of this module caused by other modules or system errors.
(2) Isolate the development status of other modules, as long as the interface is defined, it doesn't matter whether the development is completed or not.
(3) Some slow operations can be replaced by Mock Object and returned quickly.
6. Make a script
Not described in detail here.
Seven, running scene
Run test scenarios based on test cases.
Eight, do monitoring
In the process of performance testing, use commands to monitor first, and then connect to the tool for monitoring if there is a problem.
9. Analysis and tuning
After each tuning, configuration information and test results need to be recorded in detail.
10. Regression testing
After the regression test, after all the goals are achieved, the performance test report is written and sent to the project team members.
11. Drawing and writing reports
1. Test objectives
Which scenarios, number of concurrent users, response time, TPS
2. Test conclusion
pass/fail
3. Optimization of this test
A certain scenario: The TPS was 5 at the beginning of the test, and the TPS reached 30 after optimization. What problems were found and how to solve them.
4. Optimized changes
the code
JVM
database
middleware
Linux server
5. Specific test conditions
system structure
test environment
Test Methods
Test Results
6. Subsequent optimization suggestions
Summarize:
Thanks to everyone who read my article carefully! ! !
I personally sorted out some technical materials I have compiled in my software testing career in the past few years, including: e-books, resume modules, various job templates, interview books, self-study projects, etc. Everyone is welcome to leave a message in the comment area 333 to receive it for free.
