1. Classification of performance indicators
System performance indicators
Resource performance indicators
Middleware indicators
Database indicators
Stability indicators
Scalability indicators
Reliability indicators
2. System performance indicators,
response time,
system processing capacity,
throughput,
number of concurrent users,
error rate
2.1 Response Time
Response Time, abbreviated as RT, refers to the time for the system to respond to the request (the time to process the request);
the response time of different functions is not the same, so when discussing the response time of a system, it usually refers to all the functions of the system The average response time or the maximum response time of all functions
Reference standards for different industries:
Internet: less than 500 milliseconds, such as about 10 milliseconds for Taobao business
Finance: less than 1 second is better, complex business less than 3 seconds
Insurance: less than 3 seconds is better
Manufacturing: less than 5 seconds is better
The absolute value of the response time does not directly reflect the performance of the software. The performance of the software actually depends on the user's acceptance of the response time
2.2 System processing capability
System processing capability refers to the ability of the system to process information using the system hardware platform and software platform. The following indicators are used to measure:
HPS (Hits Per Second): Clicks
per second, TPS (Transaction per second): The number of transactions processed by the system per second (transactions),
QPS (Query per second): System Number of queries processed per second, times/second
Under normal circumstances, TPS is used to measure the entire business process, QPS is used to measure the number of interface queries, and HPS is used to indicate server click requests (the larger the better)
Reference standards for different industries:
Finance: 1000TPS—50000TPS
Insurance: 100TPS—100000TPS
Manufacturing: 10TPS-----5000TPS
Internet e-commerce: 10000TPS—1000000TPS
Internet medium-sized website: 1000TPS—50000TPS
Internet small website: 500TPS— 10000TPS
2.3 Throughput
Throughput refers to the number of requests processed by the system per unit time.
For single-user systems, response time can be a good measure of system performance, but for concurrent (multi-user) systems, throughput can usually be used as a performance indicator
2.4 Number of
concurrent users The number of concurrent users refers to the number of users who log in to the system and perform business operations at the same time
2.5 Error rate The
error rate is abbreviated as FR, which refers to the probability of failed transactions when the system is under load. Error rate = (number of failed transactions/total number of transactions) * 100%
reference standard: general success rate is not less than 99.4%
3. Resource performance indicators
CPU
memory
Disk throughput
Network throughput
3.1 CPU
CPU, also known as central processing unit, is a very large-scale integrated circuit, which is the core and control unit of a computer. The main function is to interpret computer instructions and process data in computer software.
Industry reference standards:
CPU indicators mainly refer to CPU utilization, including user mode (user), system mode (sys), waiting state (wait), idle state (idle)
CPU utilization rate <=75%
CPU sys% <= 30%
CPU wait% <=5%
3.2 Memory
Memory is a bridge to communicate with the CPU. All computer programs are run in the memory, and the performance of the memory has a great impact on the system.
Industry reference standard:
In order to maximize the use of memory, a cache is stored in the memory. Therefore, 100% memory utilization does not mean that the memory is bottlenecked. The measurement of whether the system memory is bottlenecked mainly depends on the swap space utilization of SWAP (exchange with virtual memory). Below 70%, too much exchange will cause poor system performance.
3.3 Disk throughput
Disk throughput is abbreviated as Disk Throughput, which refers to the amount of data that passes through the disk in a unit time without disk failure.
Industry reference standards:
disk indicators include the number of megabytes per second read and write, disk busy rate, and the number of disk queues. Average service time, average waiting time, space utilization. The disk busy rate is an important basis that directly reflects whether the disk has a bottleneck, and the disk busy rate should be less than 70%
3.4 Network Throughput
Network Throughput refers to the amount of data that passes through the network per unit time without network failure. Unit: Byte/s. The network throughput index is used to measure the system's demand for network equipment or link transmission capacity. When the network throughput index is close to the maximum transmission capacity of the network equipment or link, you need to consider upgrading the network equipment. Generally does not exceed 70% of the maximum transmission capacity of the device or link
4. Middleware indicators
Commonly used middleware such as Tomcat, weblogic and other indicators mainly include JVM, ThreadPool, JDBC
|GC frequency| times/s | Java virtual machine garbage collection frequency
| Full GC frequency | times/h | Java virtual machine garbage Full Collection Frequency
| Average Full GC Duration| Seconds| Average Duration for
Full Garbage Collection | Maximum Full GC Duration| Seconds| Maximum Duration for Full Garbage Collection
|GC Heap Usage| Percentage| Heap Usage
|Active Thread Count | Number of Active Threads
| Pending User Request | Number | Number of User Requests in the Queue
| JDBC Active Connection| Number | Number of JDBC Active Connections
5. Database metrics.
Commonly used databases such as MySQL metrics mainly include SQL, throughput, cache hit rate, and number of connections.
SQL time-consuming subtle execution of SQL time-consuming
throughput QPS queries per second
throughput TPS transactions per second
hit rate Key Buffer Hit Rate Percent Index Buffer Hit Rate
Hit Rate InnoDB Buffer Hit Rate Percent InnoDB Buffer Hit Rate
Hit Rate QueryCache Hit Rate Percentage Query Cache Hit Rate
Hit Rate TableCache Hit Rate Percentage Table Cache Hit Rate
Hit Rate ThreadCache Hit Rate Percentage Thread Cache Hit Rate
Number of lock wait times Number of lock wait times
Lock wait time subtle lock wait time
Industry reference standard: The
smaller the SQL time is, the better, generally microsecond level
. The higher the hit rate, the better, generally not less than 95%
. The lower the number of lock waits, the better, and the shorter the lock wait time, the better
6. Stability index The
shortest stabilization time: the shortest time that the system can operate stably under 80% of the maximum capacity or under standard pressure.
Generally speaking, for a system that runs on a normal working day (8 hours), it should be able to ensure the stable operation of the system for at least 8 hours.
For a system that runs 7*24 hours, at least ensure stable operation for more than 24 hours.
Reference standards:
TPS curve is stable without significant fluctuations
. There are no leaks or abnormalities in various resource indicators
7. Scalability index
refers to the relationship between increased hardware resources and increased processing power when application software or operating systems are deployed in a cluster.
Calculation formula:
(increase performance/original performance)/(increase resources/original resources) *100%
reference standard: the
ideal expansion capability is that if resources increase several times, the performance will increase several times. The expansion capacity is at least 70%.
8. Reliability indicators
For server performance testing, when analyzing system reliability indicators, it is common to start with three categories:
dual-system hot backup,
cluster
backup and recovery
8.1 dual-system hot backup The
indicators are as follows:
whether the node switch is successful and its consumption time.
Whether there is service interruption during dual-machine switching.
Whether the node switchback is successful and time-consuming.
Whether there is service interruption for dual-machine switchback.
The amount of data loss in the process of node switchback is performed while the two-machine switch is performed. At the same time, the pressure generation tool is used to simulate the actual business situation, to maintain a certain performance pressure on the application, and to ensure that the test results meet the actual production conditions.
8.2 Cluster
For systems that use cluster mode, the cluster reliability is mainly considered in the following ways:
When a node in the cluster fails, is there any business interruption
in the system?
When adding a new node to the cluster, do you need to restart the system?
After the failed node recovers , join the cluster, and whether you need to restart the system When the failed node recovers, join the cluster , Whether there is a business interruption in the system.
How long does it take to switch nodes? While verifying the reliability of the cluster, you need to use pressure tools to simulate actual business related situations based on specific conditions, maintain a certain performance pressure on the application, and ensure that the test results meet the actual production Happening.
8.3 Backup and recovery
In order to verify whether the system's backup/recovery mechanism is effective and reliable, this indicator includes system backup and recovery, database backup and recovery, application backup and recovery, including the following test content:
Whether the backup is successful and how long it takes.
Whether the backup is done automatically using scripts.
Whether the recovery is successful and how long it takes.
Recover whether to use scripts to automate the application principles of the indicator system.
The adoption and investigation of index items depend on the test purpose and test requirements of the corresponding system. The system under test is different, the purpose of the test is different, the test requirements are also different, and the indicators examined are also very different.
If some systems involve additional front-end user access capabilities, it is necessary to examine the user access concurrency indicators.
For the performance verification of the batch processing process, the batch processing efficiency is mainly considered and the batch processing time window is estimated.
If the test target involves system performance capacity, the test requirements should clearly describe the performance index requirements according to the definition of the relevant index items.
After the test indicators are obtained, the relevant prerequisites (such as the amount of business, system resources, etc.) need to be explained.