13 years of testing experience, detailed explanation of performance testing indicators, advanced testing...

Contents: Guide

foreword

Performance test core indicators

Throughput;
Response Time (Rsponse Time);
Concurrent Processing Capability;
Resource Occupancy Capability;

% of time in the test

40% ----- performance test analysis;
30% ----- test execution;
30% ----- test result analysis;
full link monitoring: as long as everything related to the system needs to be monitored

1. Throughput

How many requests the system can handle per unit time, the throughput represents the traffic of the network, the higher the TPS, the greater the throughput, including the data throughput.

The general unit is: second, the amount of requests processed per second

Note: The jmeter aggregation report we see is generally as shown in the figure below. The throughput in the table below is actually the TPS or QPS we mentioned in the article. If you want to calculate throughput, it should be the sum of receiving + sending network traffic.

During performance testing, focus on the relationship between throughput and the network bandwidth of the test environment. If the throughput is close to or equal to the bandwidth limit of the test environment, there may be a network bottleneck.

TPS:
The full name of TPS is Transaction Per Second, that is, the number of transactions processed per second, so what is a transaction?
For example: user operations are accompanied by data changes, [order ----> payment - one request will have multiple operations]; for example: 11.11 place an order with Taobao, generate order data [40W order/second]

To measure the performance of a system, it mainly depends on how much business the system can handle per unit time.

Let’s take an example of e-commerce:
1) Suppose you want to test "ordering", then the "ordering" business can be regarded as a transaction;
2) Suppose you need to test the overall business of "adding shopping cart + placing an order", then the two businesses of "adding a shopping cart" and "placing an order" form a transaction, and TPS is the number of transactions per second that process "adding a shopping cart + placing an order".

When the response time unit is seconds,

TPS = 1/response time * concurrent number

Under normal circumstances, the 28th principle is used to calculate, 80% of the transactions occur in 20% of the time; for example: 10000 transactions a day, TPS = (10000* 80%=8000 transactions) / (24 60 60 20%) 10000 transactions, 2 hours in the morning and 2 hours in the afternoon, TPS = 10000
/ 4 60 60

Before the system reaches the bottleneck, TPS is proportional to the number of concurrency.

QPS:
QPS = concurrent number/response time
The full name of QPS is Request Per Second. The literal meaning is easy to understand. It is the number of requests processed per second (such as: when a user queries data [open a certain page] and opens a Taobao product page), and does not modify the data, but only loads the data into the page.

In the case of testing a single interface, TPS=QPS, such as scenario 1) in the e-commerce example above.

2. TOP response time (Rsponse Time)

From the user's point of view: whether it is fast or not
, the time required for a request from the user to receive the server response
Page: the time to open the response The response time
of a specific single resource

TOP response time is to sort the response time of all requests from large to small, and calculate the specified proportion of requests are all less than a certain time.

This metric counts the time spent on most requests.
Testing with Jmeter usually sees the following data:
Top90 (90% response time): 90% of the request time is lower than a certain response time
Top95 (95% response time): 95% of the request time is lower than a certain response time
Top99 (99% response time): 99% of the request time is lower than a certain response time

3. Average response time

Average response time = average time spent on all requests = ART (Average Response Time)

4. Number of concurrent users/number of virtual users (concurrent processing capacity)

The number of concurrent threads/processes set in the stress testing tool, how many users can use the system at the same time without the system crashing:

It can be understood as: how much concurrency can be handled per second/millisecond

Online at the same time: session information (whether it can support multiple people online at the same time), server storage (the information of multiple people online at the same time needs to be stored by the server, and whether the server memory can support it)

Simultaneous operation: the mainstream definition in the industry, in seconds (extreme case: milliseconds in ms); for example: Double 11 Alipay queues up for payment and cannot make payment temporarily.

The difference with throughput:
Throughput: 1w requests, sent at 10:00, processed at 10:03, it is ok

Concurrency: Is the system able to withstand the high concurrency all the time? For example, if 10000 requests are initiated per second for 10 seconds, is there no problem with the system?

5. Resource occupancy rate

Conceptual understanding:
The two apps have the same function, and they are both used for "picture beautification". Let's judge which app has better performance:

The first App:
It can run on a mobile phone 5 years ago, 2GB of running memory
requires less memory resources

The second app:
it can only run on mobile phones 3 years ago, 4GB of running memory
requires more memory resources

我们可以看出，第一个App所需要的运行内存更小，占用的内存资源更少，而第二个App需要的运行内存是更大的，以及内存资源更多，只能在三年的手机运行，5年前的是运行不了，所以是第一个App相对于与第二个App来说，第一个App性能是更加好的。

性能好一些，对我们实际有什么帮助呢？

假设我们在开发一款应用：
1000使用用户
500w用户-----------3年内的新手机（运行内存4G）
500w用户-----------5年前的老手机（运行内存2G）

如我们开发时没有考虑到老手机运行内存的问题，只可以运行在新手机上，那就意味着50%，500w的老手机的用户是正常使用不了的，如我们所设计时考虑到了，性能足够的好，新、老手机都可使用，那就是100%的用户是都可以正常使用的。

成本的角度：
最小成本【最少资源】支撑最多的吞吐量、支撑最小的响应时间

CPU：作为计算机系统的运算和控制核心，是信息处理、程序运行的最终执行单元。
内存：是计算机的重要部件，也称内存储器和主存储器，它用于暂时存放CPU中的运算数据，以及与硬盘等外部存储器交换的数据。

它是外存与CPU进行沟通的桥梁，计算机中所有程序的运行都在内存中进行，内存性能的强弱影响计算机整体发挥的水平。

只要计算机开始运行，操作系统就会把需要运算的数据从内存调到CPU中进行运算，当运算完成，CPU将结果传送出来。

网络；
磁盘：利用磁记录技术存储数据的存储器。

6、成功率

请求的成功率，一般执行压测后我们会关注请求或者事务的成功率是多少，一般公司可能要求成功率在99.99%以上。

7、PV/UV

PV（Page View）页面/接口的访问量，UV(Unique Visitor)页面/接口的每日唯一访客。

The concept of PV/UV seems to often appear in e-commerce, and it may be mentioned in the analysis of performance requirements, such as the daily PV and UV of a certain page.

8. Assembly point

Rendezvous is not a metric, but a concept in performance testing.

The rendezvous point is a mechanism to increase the instantaneous concurrency pressure. Add a mark to the script, and all virtual users will wait until the mark is executed. After the left and right users arrive, they will continue to execute the next step at the same time.

Choose whether a rendezvous point is needed according to the business. If the business scenario is instantaneous and highly concurrent, such as panic buying, flash sales, etc., you need to add a rendezvous point.

After adding the rendezvous point, for the server, there will be a scenario of instantaneous high concurrency, simulating spikes, etc.; at the same time, it will also reduce the average pressure on the server.

The following is the most complete software test engineer learning knowledge architecture system diagram in 2023 that I compiled