Reprinted https://blog.csdn.net/liangfengchang/article/details/45070321
1. Commonly used performance testing terms
1. Transaction
In the web performance test , a transaction represents a process of "sending a request from the user -> the web server receives the request and processes it -> the web server obtains data from the DB -> generates the user's object (page) and returns it to the user", Typical response times are for transactions.
2. Request response time
The request response time refers to the time from a request initiated by the client to the end of the client receiving the response returned from the server. In some tools, the response is usually called "TTLB". That is, "time to last byte", which means the time from when a request is initiated to when the client receives the last byte of the response. The unit of response time is generally "seconds" or "milliseconds". A formula can express: response time = network response time + application response time. The standard can refer to the foreign 3/5/10 principle:
(1) Within 3 seconds, the page responds to the user and displays it, which can be considered as "very good";
(2) Within 3 to 5 seconds, the page responds to the user and displays it, which can be considered "good";
(3) Within 5~10 seconds, the page responds to the user and displays it, which can be considered as "reluctantly acceptable";
(4) More than 10 seconds will make people a little impatient, and the user is likely not to continue to wait;
3. Transaction response time
A transaction may be composed of a series of requests. The response time of a transaction is mainly aimed at users. It is a macro concept and is proposed to explain the business response time to users. For example, the response time of an inter-bank withdrawal transaction is determined by a series of The request is composed of. The transaction response time is a parameter that directly measures the performance of the system.
4. Number of concurrent users
Concurrency is generally divided into two situations. One is concurrency in the strict sense, that is, all users do the same thing or operation at the same time, which generally refers to doing the same type of business. For example, in the credit card approval business, a certain number of supporters submit the completed approval business at the same time; there is also a special case, that is, all users perform the exact same operation. For example, in the credit card approval business, all users can apply together business, or modify the same record.
Another type of concurrency is broad scope concurrency. The difference between this concurrency and the previous concurrency is that although multiple users have issued requests or performed operations on the system, these requests or operations can be the same or different. For the whole system, there are still many users operating the system at the same time, so it also belongs to the category of concurrency.
It can be seen that the latter concurrency includes the former concurrency. And the latter kind of concurrency is closer to the actual usage of users, so for most systems, only a small number of users are "strictly concurrent". For WEB performance testing, these two concurrency situations generally need to be tested, and the usual practice is to perform concurrency testing in a strict sense first. User concurrency in the strict sense generally occurs in modules that are used more frequently. Although the probability of occurrence is not very large, once a performance problem occurs, the consequences are likely to be fatal. Concurrency testing in the strict sense is often associated with functional testing , because exceptions encountered in concurrent functions are usually program problems, and this testing is also part of robustness and stability testing.
Number of concurrent users: There are 2 common misconceptions about the number of concurrent users. A wrong view is to interpret the number of concurrent users as the total number of users using the system, on the grounds that these users may use the system at the same time; there is a more correct view that the number of online users is understood as the number of concurrent users. In fact, online users are not necessarily concurrent with other users. For example, users who are browsing web pages have no impact on the server . However, the number of online users is one of the main criteria for calculating the number of concurrent users.
5. Throughput
Refers to the total amount of data transmitted on the network during a performance test. Throughput/transmission time is the throughput rate.
6、TPS（transactionper second）
The number of transactions or transactions that the system can process per second. It is an important indicator to measure the processing power of the system.
7. CTR
The number of HTTP requests submitted by users to the WEB server per second . This indicator is a unique indicator of WEB applications: WEB applications are in the "request-response" mode. Once a user sends an application, the server will process it once, so clicking is a WEB application can The smallest unit of transactions processed. If each click is defined as a transaction, the CTR and TPS are a concept. It is easy to see that the higher the CTR, the greater the pressure on the server. The CTR is only a performance reference indicator, important is to analyze the impact of clicks. It should be noted that the click here does not refer to a single-click operation of the mouse, because in a single-click operation, the client may issue multiple HTTP requests to the server.
8. Resource utilization
Refers to the degree of use of different system resources, such as server CPU utilization, disk utilization, etc. Resource utilization is the main basis for analyzing system performance indicators and improving performance, so it is the focus of WEB performance testing .
Resource utilization is mainly aimed at WEB servers, operating systems , database servers, networks, etc., and is the main reference for testing and analyzing bottlenecks. In the WEB performance test, the corresponding parameters are collected and analyzed as needed.
2. Extension part
Click Graph-->Add New Graph... or press the shortcut key to open a new view, you can also
The Analysis analyzer provides a wealth of analysis graphs, and the common ones are as follows: Vusers graph, transaction graph, Web resource graph, webpage segmentation graph, error graph, system resource graph, Web server resource graph, and database server resource graph.
Uncheck the display only graphs containing data checkbox to display more analysis graphs.
(1) Vusers diagram
During scenario execution, Vusers generate data as they execute transactions. Use the Vusers graph to determine the overall behavior of Vusers during scenario execution. It shows the Vuser status and the number of Vusers who completed the script. It mainly includes running Vuser graph and Vuser summary graph.
(2) Errors graph
During the execution of the scheme, not every Vuser will be executed successfully, and there may be cases where execution fails, stops, or terminates due to errors. The Errors graph is mainly the error information when the statistical scheme is executed, including four graphs: Error Statistics (By Description), Error per Second (by Description), Error Statistics and Error per Second.
(3) Transaction diagram
The transaction graph describes the transaction performance and status during the entire script execution process, including: average transaction response time graph, number of transactions per second graph, total number of transactions per second, transaction summary graph, transaction performance summary graph, transaction response time (under load) ) graph, Transaction Response Time (Percent) graph, and Transaction Response Time (Distribution) graph.
1. Transation Sunmmary Comprehensive analysis
of transactions is the first step in performance analysis. By analyzing the success and failure of user transactions within a period of time,
You can directly determine whether the system is running normally.
2. Average Transaciton Response Time
The "Average Transaction Response Time" shows the average time taken for transaction execution in each second during the running of the test scenario.
It can analyze the performance trend of the application system during the running of the test scenario.
Example: With the change of the test time, the speed of the system processing transactions begins to gradually slow down, which means that the overall performance of the application system will have a downward trend with the change of the production time.
3. Transactions Per Second (transactions per second/TPS)
"Transactions per second/TPS" displays the number of passes, failures, and stops of each transaction in every second of the scenario running, making it an important tool for examining system performance. Important parameters. It is used to determine the time transaction load of the system at any given moment.
Analysis of TPS is mainly to see the performance trend of the curve.
Comparing this to the average transaction response time allows you to analyze the effect of the number of transactions on execution time.
Example: When the pressure increases, if the CTR/TPS curve changes slowly or has a flat trend,
Most likely the server is starting to bottleneck.
4. Total Transactions Per Second
"Total transactions per second" displays the total number of transactions passed in each second when the scene is running
The total number of failed transactions and stopped transactions.
5. Transaction Performance Sunmmary (Transaction Performance Summary)
"Transaction Performance Summary" shows the minimum, maximum and average execution time of all transactions in the scheme,
You can directly judge whether the response time meets the user's requirements.
Focus on the average and maximum execution time of the transaction, if the range is not within the acceptable time range for the user,
Cause analysis is required.
6. Transaction Response Time Under Load (Transaction Response Time and Load)
"Transaction Response Time and Load" is a combination of the "Running Virtual User" graph and the "Average Response Transaction Time" graph.
Through it, we can see the relationship between the transaction response time and the number of users at any point in time,
In this way, we can grasp the performance data of the system in terms of user concurrency, and provide a reference for expanding the user system.
This graph lets you see the overall impact of virtual user load on execution time,
Useful for analyzing test scenarios with gradient loads.
7. Transaction Response Time (Percentile) (Transaction Response Time (Percentile))
"Transaction Response Time (Percentile)" is a comprehensive analysis chart obtained by analyzing the test results.
That is, the graphs obtained indirectly by the tool through some statistical analysis methods.
It allows you to analyze the percentage of transactions that can be executed within a given transaction response time frame.
8. Transaction Response Time (Distribution)
"Transaction Response Time (Distribution)" shows the distribution of the time spent in transaction execution during the scenario running process.
It gives you an idea of the number of transactions with different response times during the test.
If the system predefines the minimum and maximum transaction response time acceptable for the relevant transaction,
You can then use this graph to determine if server performance is within acceptable limits.
(4) Web resource diagram
The web resource graph mainly provides some information about the performance of the web server. Using the web resource graph, you can share the number of hits per second, the throughput of the server, the HTTP status code returned from the server, the number of HTTP responses per second, and the number of page downloads per second during the running of the scheme. count, server retries per second, server retry summary, connections, and connections per second.
1. Hits per Second (Hits per Second)
"Hits per second", even the number of HTTP requests submitted by virtual users to the Web server per second during the running scenario.
It can be used to evaluate the amount of load generated by virtual users, such as comparing it with the "Average Transaction Response Time" graph,
You can view the impact of hits on transaction performance.
By checking the "number of clicks per second", you can judge whether the system is stable.
A drop in the system hit rate usually indicates that the response speed of the server is slowing down, and further analysis is needed to find the bottleneck of the system.
2. Throughput (throughput)
"Throughput" shows the server's throughput per second during the scenario operation. Its unit of measure is bytes,
Indicates the amount of data the virtual uses to get from the server at any given second.
The load generated by virtual users can be evaluated based on the throughput of the server,
And see the processing power of the server in terms of traffic and whether there is a bottleneck.
The difference between the "throughput rate" graph and the "click rate" graph:
The "click rate" graph is the number of HTTP requests processed by the server per second.
The "Throughput" graph is the total amount of data the client gets from the server per second.
Throughput (MB) unit is M
Throughput unit is byte
3. HTTP Status Code Summary
"HTTP Status Code Summary" displays the number of HTTP status codes returned from the web server during a scenario or session step, grouped by code. HTTP status codes indicate the status of the HTTP request.
4. HTTP Responses per Second (HTTP responses per second)
"HTTP responses per second" is to display the number of different HTTP status codes returned from the web server per second during the running scenario, as well as information about various status codes. By analyzing the status code, you can judge the operation of the server under pressure,
You can also locate code scripts that generate errors by grouping the results shown in the graph.
5. Pages Downloaded Per Second "Pages Downloaded Per Second
" displays the number of web pages downloaded from the server in every second that the scenario or session step runs.
Use this graph to calculate the amount of load generated by Vusers based on the number of pages downloaded.
As with the Throughput graph, the Pages Downloads per Second icon is the amount of data a Vuser receives from the server in any given second. But the individual resources considered for throughput are extremely small (e.g. size of each GIF file, size of each web page)
While the number of pages downloaded per second only considers the number of pages.
Note: To view the number of pages downloaded per second graph, you must set "Pages per second (HTML mode only)" in RTS.
6. Retries per Second
"Retries per second" shows the number of connections attempted by the server in each second that the scenario or session step runs.
The server connection will be retried in the following situations:
A. The initial connection is unauthorized
B. Proxy authentication is required
C. The server closes the initial connection
D. The initial connection fails to reach the server
E. The server was initially unable to resolve the IP address of the load generator
7. Retries Summary
The "Retries Summary" displays the number of connections attempted by the server during a scenario or session step.
It is grouped by retry reason.
Use this graph in conjunction with the retries per second graph to determine at what point during a scenario or session step the server retried.
8. Connections
"Connections" displays the number of TCP/IP connections opened at each point in time during the running of the scene or session step.
With the help of this graph, you can know when you need to add connections.
Example: When the number of connections reaches a steady state and the transaction response time increases rapidly, adding connections can greatly improve performance (transaction response time will decrease).
9. Connections Per Second "Connections
per second" displays the number of TCP/IP connections established per second during the running of the program.
Ideally, many HTTP requests should use the same connection, rather than opening a new connection for each request.
Through the number of connections per second graph, you can see the processing of the server, indicating that the performance of the server is gradually declining.
10. SSLs Per Second (SSL connections per second)
"SSL connections per second" shows the new and reused SSLs opened within each second of the scenario or session step running
number of connections. When a TCP/IP connection is opened to a secure server, the browser will open an SSL connection.
(5) Web page segmentation diagram
The webpage segment graph mainly provides some information to evaluate whether the page content affects the transaction response time. If there is a situation that affects the transaction response time, you can further analyze the reasons that affect the webpage transaction response time through the segment graph. Including page segment, page component segment, page component segment (over time), page download time segment, page download time segment (over time) and downloaded component graph.
1. Web Page Diagnostics (page decomposition general diagram)
"Page decomposition" shows the response of a specific transaction in the testing process, and then analyzes whether the related transaction is running normally.
The "Page Breakdown" graph can be further subdivided in the following four ways:
1), DownloadTime Breaddown (Download Time Breakdown)
The "Download Time Breakdown" graph displays the download time of different elements in the web page, and can The time is broken down, and different colors are used to display the respective proportions of DNS resolution time, connection establishment time, and first buffer time.
2), ComponentBreakdown(Over Time) (component subdivision (change over time))
"Component subdivision" graph displays the subdivision graph of the page components of the selected web page over time.
This graph makes it easy to see which elements have unstable download times during testing.
This graph is especially suitable for pages that need to download a lot of controls on the client side. By analyzing the response time of the controls, it is easy to find those controls that are unstable or time-consuming.
3), DownloadTime Breakdown(Over Time) (Download Time Breakdown (Over Time))
The "Download Time Breakdown (Over Time)" figure shows the download time breakdown (over time) of the page elements of the selected web page, It shows very clearly how each element of the page was downloaded during the stress test.
The "Download Time Breakdown" graph shows the time statistical analysis results of page element response during the entire test process, and the "Download Time Breakdown (Varies with Time)" shows the statistics of page element response time in every second during the running of the scenario. result,
The two analyze the download time of page elements from a macro and micro perspective respectively.
4), Time to First Buffer Breakdown (Over Time)
(First Buffer Time Breakdown (vs. time))
The "First Buffering Time Breakdown (vs. Time)" graph shows the successful receipt of the first buffer returned from the web server
The server time and network time (
in seconds). You can use this graph to identify when a server or network problem occurs during a scenario or session step run.
First Buffer Time: means that after the client establishes a connection with the server,
From the time the server sends the first data packet, the data is transmitted over the network to the client,
The elapsed time until the browser receives the first buffer.
3. Page Component Breakdown
The "Page Component Breakdown" graph shows the average download time (in seconds) for each web page and its components.
The graph column can be sorted by the average number of seconds it took to download components, which helps isolate problematic components.
4. Page Component Breakdown (Over Time)
The "Page Component Breakdown (Over Time)" graph displays each web page in every second during the program's operation
and the average response time (in seconds) of its components.
5. Page Download Time Breakdown
The "Page Download Time Breakdown" graph shows a breakdown of the download time of each page component,
It can be used to determine whether slow transaction response times during web page downloads are caused by network errors or server errors.
The "Page Download Time Breakdown" graph is based on DNS resolution time, connection time, first buffer time,
SSL handshake time, receive time, FTP authentication time, client time and error time to download each component
The process is subdivided.
6. Page Download Time Breakdown (Over Time) (Page Download Time Breakdown (Over Time))
The "Page Download Time Breakdown (Over Time)" graph shows each page component in every second during the running of the program
A breakdown of download times. Use this graph to determine at what point during a scenario a problem occurred with the network or server.
"Page Components Breakdown (Over Time)" graph and "Page Download Time Breakdown (Over Time)"
Graphs are often analyzed in combination: first identify problematic components, then analyze their download process,
Find out where the reason is.
7. Time to First Buffer Breakdown The
"First Buffer Breakdown" graph shows each page component during the period before the first buffer returned from the web server was successfully received relative server/network time. If a component takes a long time to download,
You can then use this graph to determine whether the resulting problem is server-related or network-related.
Network time: defined as the average time elapsed from the moment of the first HTTP request until the confirmation.
Server Time: Defined as the time since receipt of the initial HTTP request confirmation,
The average time that elapsed until a buffer was successfully received from the web server.
8. Time to First Buffer Breakdown(Over Time) one
The server time and network time of each web page component for every second of the scenario run during the period before the first buffer.
You can use this graph to identify the point in time during a scenario run that a problem occurs with the server or network.
9. Downloader ComponentSize(KB) (downloaded component size)
The "downloaded component size" figure shows the size of each downloaded web page component. It can directly see which components are large and need further optimization to improve performance.
(6) System resource diagram
The system resource graph mainly monitors the usage of system resources during the running of the scenario. Can monitor Windows resources, UNIX resources, SNMP resources, Antara FlameThrower resources and SiteScope resources.
(7) Web server resource diagram
The Web server resource graph is mainly used to capture the information of the Web server when the scene is running, and it is mainly used to analyze the Microsoft IIS server, Apache server, iPlanet/Netscape server and iPlanet (SNMP) server.
(8) Database server resource diagram
The database server resource graph mainly displays the statistical information of the database server. DB2, Oracle, SQL Server and Sybase databases are currently supported.
3. How to use LoadRunner to analyze resource occupancy
1. Average Transaction Response Time
Average Transation Response Time
优秀：<2s
Good: 2-5s
Pass: 6-10s
Fail: >10s
2. Hits per second
Hits per Second
When the pressure of the system is increased (or the number of concurrent users is increased), the change curve of the throughput rate and TPS is roughly the same, and the system is basically stable. If the pressure increases, the throughput curve increases to a certain level and then changes slowly or even flat, which is likely to be a bandwidth bottleneck in the network. Similarly, if the CTR/TPS curve changes slowly or flat, it is likely that the server response time Increase, observe the server resource usage to determine whether it is a server problem.
3. Request response time
Time to Last Byte: Cumbersome business, generally within 15s; most of the response time for logging in is within 3s; the response time for adding data is within 8s; the page opening is within 5s.
4. The number of transactions processed by the system per second
Transaction per second
5. Throughput
Throughout
6. CPU utilization
Processor / %Processor Time
好：70%
Bad: 85%
Very bad: 90%+
7. CPU time consumed by database operations
If the value of Processor / %User Time
is large, consider whether this value can be reduced by a friendly algorithm or other methods. If the server is a database server,
the reason for the large value of Processor\%User Time is probably that the sorting of the database or the function operation consumes too much CPU time. In this case, you can consider optimizing the database system.
8. Core state CPU average utilization
Processor /%Privileged Time If the value of this parameter and the value of the "Physical
Disk" parameter are consistently high, there is an I/O problem. Consider replacing a faster hard drive system
9. Process the number of threads in the queue
Processor / Processor Queue
Length If the value remains constant (>=2) and the %Processor Time
exceeds 90%, there may be a processor bottleneck. If it is found that more than 2, and the utilization of the processor has been low, then perhaps it is more appropriate to solve the problem of processor blocking, where the processor is generally not the bottleneck.
10. Filesystem cache
Memory / Cache Bytes 50% of available physical memory
11. Remaining free memory
Memory / Avaiable Mbytes must be at least 10%
of the physical memory value
12. Pages downloaded per second
Memory/pages/sec
good: no page swapping
Bad: CPU swaps 10 pages per second
Bad: more page swapping
13. Page read operation rate
Memory / page read/sec
If the rate of page read operations is low and the values of % Disk Time and Avg.Disk Queue
Length are high, there may be a disk bottle diameter. However, if the page read rate does not decrease while the queue length increases, there is insufficient memory.
14. Physical Disk Utilization
Physical Disk / %Disk Time
好：<30%
Bad: <40%
Very poor: <50%+
15. Physical disk average disk I/O queue length
Physical Disk / Avg.Disk Queue Length The value should not exceed 1.5~2 times the number of disks. To improve performance, increase the disk
16. Network throughput
Network Interface / Bytes Total/sec
to determine whether the network connection speed is the bottleneck, you can use the value of this counter and the current network bandwidth, the result should be less than 50%
17. The hit rate of the data cache area
should be greater than 0.90.
Shared area library cache hit rate The hit rate should be greater than 0.99
Monitor the hit rate of dictionary buffer in SGA The hit rate should be greater than 0.85
Detect contention for rollback segments Less than 1%
The hit rate of the redo log buffer in the monitoring SGA should be less than 1%
Monitor the sort ratio of memory and hard disk, preferably keep it less than 10%