Software performance testing study notes (LoadRunner): starting from scratch

Overview

Software performance testing is completely different from software functional testing, which only focuses on the performance of the server. There are many ways to test software, and there are three commonly used methods, namely stress testing, load testing and stability testing. In addition, there are methods such as failover and recovery testing.

• Stress Test: Put high-intensity pressure on the software in an instant, so stress testing is also called concurrency testing.
• Load testing: Continuously stressing the software.
• Stability test: Keep the pressure to a certain high value and then maintain it to test the endurance of the software. Stability testing often takes several hours.

Before conducting performance testing, preliminary preparations need to be completed:

1. Determine the type of performance test: Determine which way to conduct performance testing. For example, performing concurrency on an interface is a stress test; testing a website's customer capacity is a load test.
2. Determine the indicators for performance testing: Determine the standards for evaluating software performance in advance. For example, for the response time of a website, it is generally considered that within 1 second is very good, within 3 seconds is very good, within 5 seconds is good, within 7 seconds is qualified, and more than 7 seconds is unqualified.
3. Determine the test machine configuration and server configuration: It is necessary to make the strength of the test machine and the server as "equal" as possible, and neither side can be too strong or too weak. It is also necessary to determine the number of servers and test machines (multiple test machines can be used for multi-machine joint testing), because multiple servers or multiple test machines will increase the strength of one party.
4. Determine the IP address of the server: Only by knowing the IP address of the server can the various indicators of the server be remotely monitored.
5. Write performance test cases: That is to design a process for using the software, and test the performance through this process.

To conduct software performance testing, the testing software must be installed. The software used in this blog is LoadRunner. If you need to perform performance testing on the interface, you need to use another software JMeter. This will not be expanded here, but only the performance testing of the business will be introduced.

Business performance testing process:

1. Record performance script: Record the performance script according to the business that needs to be tested;
2. Perform script enhancement: The script obtained directly through recording is only preliminary script content and cannot be used directly. Script enhancement is required. Common techniques in script enhancement include rendezvous points, transactions, associations, checkpoints, parameterization, user think time and other function settings, which will be introduced later. Script enhancement is also an important and difficult point in this blog.
3. Design performance scenario: Import the enhanced performance script and set the number of virtual users, duration, end time, etc. (equivalent to simulating that the website is used by many user usage). Once the design is complete, you can run the scenario.
4. Performance indicator evaluation: Evaluate the scenario running results in the previous step through quantitative indicators. Commonly used indicators include throughput, response time, number of clicks per second, and number of concurrent users. , error information, number of HTTP responses, CPU/memory usage and network delay time, etc.
5. Write a test report: Analyze the results by making an analysis table for the results.

Use of LoadRunner

Notes: It is best to open LoadRunner as an administrator, otherwise some strange errors may occur. For example, it may cause an error message indicating that you cannot connect to the Internet.

Composition of LoadRunner: LoadRunner consists of three parts: creating and editing scripts (Virtual User Generator), running load tests (Controller) and analyzing test results (Analysis). The LoadRunner version I am using is 12.02.

Create editing script (Virtual User Generator)

The interface after opening VUG is as shown below:

Create a new running script: Click the 文件 option in the upper left corner menu and select 新建一个脚本和解决方案 in the drop-down part, as follows shown. In addition, you can also choose to open a running script (solution) in the 文件 menu.

Select单协议→Web - HTTP/HTML to simulate the communication between the browser and the Web server. You also need to set the script name and storage location.

Also, click the 协议分析软件 button in the lower left corner, select the browser used for recording and the URL address to be tested, and then click 开始分析. Note that this step is not necessary, it is just used to determine whether the browser you are currently using is supported.

Note ①: If you are using the LoadRunner 11 series version, it may not be supported by some browsers. At this point you can install a Firefox 24 version of the browser and set it up.
Note ②: Since the virtual users of LoadRunner are aggressive to websites, this software cannot be used to conduct random testing experiments on various websites. LoadRunner provides an airplane booking simulation website for practice.

Click the 创建 button in the previous picture to create an original Action test script. The effect is as shown in the picture below. The script content at this time is empty, and we need to automatically add content to the script through subsequent recording.

Click录制→录制选项→录制 in the menu bar of the above interface, and select 基于HTML的脚本. This is because HTML-based scripts are more concise and cleaner than URL-based scripts.

ClickHTML 高级 on the right and set the script type to仅包含明确URL的脚本.

Click the left脚本 button and set the script language toC语言.

Click the 高级 button on the left and select UTF-8 from the checked character set. If not checked, garbled characters may be returned.

After the above settings are completed, click the 确定 button to complete the recording settings. After that, click the 录制 button in the menu above the main interface to define this recording. Currently, the recording operation is temporarily selected Action, the recording mode is selected Web浏览器, the application is the browser used for recording, and the URL address is the website that needs to be performance tested. . In addition, you also need to set the working directory. After that, click开始录制 to get started. The specified URL will be opened automatically after starting.

Note: In the picture below, the given browser program is Internet Explore, but this browser cannot actually be used because the script that will cause the recording is Blank. Therefore, it is recommended to use the Firefox browser that comes with LoadRunner. Related blog links:How to solve the problem that the LoadRunner recording script is empty. If the recording still fails, you can try closing all browsers in the background, clearing the memory and checking whether the configuration is correct.

Script recording is actually a demonstration of the operation process that requires performance testing. For example, if I need to perform a performance test on the user registration of the website, I will complete a complete user registration during the recording process and then end the recording. The menu after recording starts is as follows. There are buttons such as stop recording and pause recording in the menu. When the recording is complete, just click the 停止录制 button.

After stopping recording, return to the LoadRunner software interface and you will see the corresponding script content. A good habit is to run (play back) the generated script once and observe whether there are any errors. If no errors are reported, proceed with subsequent steps such as script enhancement. Finally, be sure to save the script.

It should be noted that the recording process must have a clear purpose and do not do anything irrelevant to the process that needs to be tested. This will cause the content in the generated script to be messy and cannot be directly used in subsequent processes.

Let’s first introduce a few concepts:

meeting point

• Definition: Used to control the place where pressure is exerted during the recording process. At this time, after each virtual user reaches the meeting point at different speeds, only the last virtual user has to wait. Only when you arrive will you move on. Therefore, the rendezvous point can control the process before the rendezvous point so that large-scale concurrency will not cause the server to crash prematurely, and can also better focus on testing the performance of the interface we care about.

Notes: Try to have only one rendezvous point in a script, unless the distance between the two rendezvous points is very long. Doing so will distort the results at the second rendezvous point and put more stress on the server.

Inserting a rendezvous point is very simple. You only need to click Insert rendezvous point in the recording menu and name the rendezvous point before sending a request to the website.

Thinking time

• Definition: Thinking time is a function used in scripts to simulate staying on the page. Using thinking time can make the script more realistic. The representation method in script is lr_think_time(思考秒数). In addition, the functions that come with LoadRunner all start with lr.

In the script after recording, the thinking time will be automatically assigned according to the length of stay on each page during the recording process. However, if you want think time to take effect, you need to set it up at runtime. Click 查看 in the upper left menu, then 解决方案资源管理器 as shown below:

In the pop-up menu on the left, select运行时设置, and click思考时间 in the pop-up interface. As you can see, it is automatically忽略思考时间 by default. We can adjust it to 按录制参数回放思考时间 as needed, or pass 使用录制思考时间的随机百分比, to simulate multi-user usage, in which case you can also choose to limit the think time to a specified number of seconds.

Note: Thinking time cannot be set before the meeting point.

affairs

• Definition: A wrapped process. You can think of a transaction as a small part of the recording process. By encapsulating transactions, large processes can be broken down into small individual units for processing.

The method of inserting transactions in LoadRunner is very simple. During the recording process, before the event that needs to be analyzed separately, select 插入开始事务 in the recording menu and name it, and then the event After completion, click插入结束事务 and select the corresponding start transaction name. Although transactions can be inserted directly into the script by writing code, it is recommended for beginners to insert them during the recording process.

It should be noted that there can be no thinking time inside the transaction. This is because if there is think time inside the transaction, it will affect the response time results. In addition, the rendezvous point should not be placed inside the transaction, otherwise the rendezvous time will also affect the response time results.

Wasted time: The wasted time in the response time refers to the time it takes for the server to reply to the request after the server receives the request.

checking point

• Definition: Used to ensure that every page is tested correctly and there are no hidden errors.

Checkpoints can be divided into text checkpoints and image checkpoints. Text checkpoints determine whether there is an error by finding whether there is a specified string in the returned web page. It should be noted that the script is inserted after the interface acquisition is completed. We can insert text checkpoints by modifying the script code:

web_find("web_find","What=待查找的内容",LAST)

It should be noted that checkpoints are best placed outside the transaction to prevent detection that affects response time. After modifying the code, go into the script's runtime settings, check to enable image and text inspection, and then the script will be checked when you run it.

Image checkpoints are not covered here.

association

• Definition: Association means that part of the content in the recording result script is not a constant but a variable (such as the website's Token), so it is possible to run the same script repeatedly Will fail to get content requirements that don't meet the test so the association can be defined.

The association needs to be inserted before the variable is obtained, because this can tell the server that a certain value is used as a variable and should be processed in a special way.

The way to establish associations in LoadRunner12.02 is very simple. After each script recording is completed, a window as shown below will be generated:

The above window indicates that LoadRunner automatically scanned an association during the script recording process, that is, the part with the field userSession. After confirming that the result is correct, you can click 关联 in the window to create this association. Then click关闭. After the association is created, you can see that the userSession field in the script file is replaced with a curly bracket representation, as shown below:

to parameterize

• Definition: For the user input part in the script, if the input content is different each time, different parts of the content need to be parameterized so that one script can handle multiple different inputs.

The difference between parameterization and association: Parameterization refers to the difference in script content due to different user input content, and parameterization is caused by the difference in content returned by the server.

The method of parameterizing part of the script in LoadRunner is very simple. First select the part that needs to be parameterized, then right-click and select 使用参数替换 and 新建参数 in the menu to get the interface as shown below:

In this window we can set the name, type and initial value of the parameter. Parameter names and initial values ​​can be customized, but the types are generally of the File type. After setting, click 确定 and you will see that the value in the original script code is replaced. ’

Next, we open the right-click menu of the generated parameters, select 参数属性, and open the window as shown below:

文件路径 in the above figure is actually the values ​​that the parameters can take, and these values ​​are stored in the specified file. You can click the 用记事本编辑 button in the interface to modify the content in the document. For example, the interface for initially opening a document is as follows:

Username in the document represents our parameter name, and T2020308250120-001 represents a value that the parameter needs to take. If we need this parameter to take more values, such as T2020308250120-002, we can modify the document content. The modified document is as follows:

After saving the document and exiting, you will see that the content of the parameter property interface has also been modified:

In addition, you can also click the 添加行 button directly in the interface to quickly add new parameter values.

In addition to setting values ​​row by row, you can also click the 数据向导 button to directly import data sources from the database. SQL statements need to be used here without too much explanation.

If there is more than one column of parameters (for example, there is a password in addition to the account number), you can click the 添加列 button to add column data.

We can see that there are two attributes 选择下一行 and 更新值的时间 in the interface. Both attributes can take three values, so there are nine in total. different combination methods. This part is the focus of parameterization. The effects of different combinations are summarized below. The number of iterations is the number of loop executions of the script.

• Sequential + Each iteration: In each iteration, the next value is taken from the data table. The value is the same for all users in an iteration.
• Random + Each iteration: In each iteration, each virtual user takes out a new value from the data table.
• Unique + Each iteration: In each iteration, the next unique value is taken from the data table.
• Sequential + Each occurrence: Each occurrence of the parameter takes the next value from the data table, whether in the same iteration or not.
• Random + Each occurrence: A value is randomly taken from the data table for each occurrence of the parameter, whether in the same iteration or not.
• Unique + Each iteration: In each iteration, the next unique value will be taken from the data table, whether in the same iteration or not.
• Sequential + Once: The value obtained in the first iteration will be used by each virtual user in all subsequent iterations.
• Random + Once: The random value obtained in the first iteration will be used by each virtual user in all subsequent iterations.
• Unique + Once: The unique value assigned in the first iteration will be used by each virtual user in all subsequent iterations.

Click the 模拟参数 button in the interface, and set 虚拟用户的数量 and 迭代次数 in the pop-up interface, as shown below, Then click 模拟 to see the parameter results for the currently selected attribute combination.

Parameterization can be completed according to the above method.

Other content: In addition to the above script enhancement methods, comments can also be added during the script recording process. After completing the recording and enhancement of the script, you need to use the Controller to run the script.

Run load test (Controller)

Open the Controller and the initial interface looks as follows:

We can import the enhanced script and set its name in the upper left corner of the above interface. At the same time, we can also set the percentage of scripts (because multiple scripts can be run at the same time) and the machine for testing. After importing the script that needs to be run and setting the ratio, the interface is as follows:

Click in the drop-down menu at the Load Generator corresponding line in the script. In the pop-up interface, set the name line to the IP of the test machine address. If you use this machine as a test machine, just enter directly. Then click. 添加localhost确定

In the lower left corner of the Controller interface 全局计划, double-click to start VUser and select the total number of virtual users when starting the script. The number of users is determined based on test preparation. In addition, you can also increase the number of specified virtual users at the same time or at intervals. Click确定 after setting is complete.

Return to the Controller interface, also in the 全局计划 panel, double-click the duration. The duration can be divided into 完成前一直运行 and the specified running time. We choose according to our needs, choose the former here, and then click 确定. When you choose the former, you don't need to consider the exit method; if you choose the latter, if the specified time is not reached after completing once, the operation will be repeated a second time.

After completing the above settings, click 运行, and the software interface will change to the following form. The interface of the Controller includes the 设计 interface and the 运行 interface. The 运行 interface monitors various test indicators.

If you feel that the four test indicator pictures are too few, you can right-click on the blank area of ​​the image, then click 查看图, and then click 显示八张图. The results shown below were obtained. The indicators that can be monitored are on the left 可用图, and we can select them freely by clicking the left mouse button.

Commonly used indicators include:

• Running VUser; (very important)
• Transaction response time; (very important)
• HTTP responses per second;
• VUser with error;
• Telephone information；
• Throughput; (very important);
• Clicks per second; (very important, can reflect the point of concurrency)

LoadRunner does not have a good visualization effect on Windows resources (CPU and memory), and third-party tools are generally used for monitoring. But let’s not remove it here. The eight pictures after adjustment are as follows:

If you still want to use the Windows resource window to monitor the server's resources, right-click the graph and select Add Measurement from the menu. Click Add on the pop-up page and enter the IP address of the server. Here we take this machine as an example:

Click on the selected indicator as shown below:

Next, we set the refresh rate of the image. Right-click on any picture, select Configuration in the menu, open the following window, modify the refresh rate, and check 应用于所有图.

After all designs are completed, click Start Scenario in the upper right corner of the Controller interface, and it will be executed automatically. At this time, the performance indicator graph will also be continuously refreshed, as shown in the following figure:

Click Vuser in the upper right corner to see the current status of each virtual user:

The result after the operation is completed is as shown below:

Notes: If you insert a collection point in the script itself, you need to set the collection method first before running the scene, otherwise the result may be wrong.

Performance test report

The performance test report includes two parts: the scene setting table and the result evaluation table.

Scene setting form

The template is as follows:

Test name (including the interface and test type being tested, as well as the test number)
Test steps
Scenario description (number and changes of users, rendezvous point settings, interfaces requiring concurrency)
Test case (script name + proportion)
operating mode
exit mode
expected outcome

Test indicator record table

Examples are as follows:

Number of concurrent users	Response time	Maximum throughput	CPU usage	Memory usage	Number of failed users

Other miscellaneous topics

• CPU self-protection temperature: Under normal circumstances, when the temperature of the CPU reaches 80 or 90 degrees Celsius, it will alarm and automatically shut down to achieve self-protection.
• Cooling of large servers: Place the server in a place with very low ambient temperature to achieve cooling of the server.
• Server operating system: The operating system of most servers is Linux. This is because the Linux operating system is very stable. The only one in the Windows series of operating systems that can be used as a server operating system is Window 2003.
• Software performance test requirements for test machines: A good CPU (Intel i7 or above) and a large memory (at least 8GB) are required.
• One wrong step and the next wrong: If script recording errors, script enhancement errors, etc. occur during the business performance test, the following content will be wrong, so special care is required. Be careful and make sure every step is correct.
• Performance test time: The results of the performance test are very different when users are intensively using the server and when no one is using the server. Therefore it is best to conduct performance testing when no one is using the server.
• Throughput: Throughput is the core indicator for evaluating servers, indicating the server's ability to process transactions. When the number of users increases from zero, the throughput of the server will increase; but if the number of users continues to increase, the throughput will continue to decrease. When performing load testing, it is best to find the number of users corresponding to the peak load of the server for testing.
• Notes on CPU utilization: During the test, it is generally required that the CPU utilization cannot exceed 70%.
• The number of virtual users used during the exercise: It is best to control the number of virtual users not to be too large during the exercise, otherwise it will cause a great burden on the server.
• TPS: An indicator of server processing speed. The higher the TPS, the faster the server processes.