[2023 The Most Diao Tutorial on the Internet] Performance Test Study Notes (Super Detailed~)

Performance testing aims at the performance indicators of the system, establishes a performance test model, develops a performance test plan, formulates monitoring strategies, executes performance scenarios under scenario conditions, analyzes and judges performance bottlenecks and optimizes them, and finally obtains performance results to evaluate system performance indicators Whether it meets the predetermined value.

Performance

Metrics include: Time Metrics, Capacity Metrics, and Resource Utilization Metrics
- The time index refers to the interface response time, business response time
- Capacity indicators refer to interface capacity, service capacity
- Resource utilization indicators refer to the operating system (CPU, IO, Mem, Disk, Network, System), JVM, etc.
The origin of the indicator: It can be obtained through discussion with team members according to the business scenario; or the indicator is not set during the actual stress test, and the purpose is to check the performance bottleneck of the system

Model

The model refers to the abstraction of the real scene, which can tell the performance tester what the business model looks like
In layman's terms, the model can let testers know the concurrency of specific businesses, and it is convenient for testers to design specific pressure ratios according to the model.
Acquisition of model data: generally obtained from statistics in the production environment (such as logging each node, and then analyzing the traffic situation according to the log)

Performance test plan

Contents included in the plan: test environment, test data, test model, performance model, pressure strategy, entry and exit, and schedule risk

performance monitoring

It must have the ability of layering and segmentation, as well as the ability of global monitoring and directional monitoring

Performance testing must have predetermined conditions

The conditions here include hardware and software environment, test data, test execution strategy, pressure compensation, etc.

There must be scenarios in performance testing

Execute performance scripts under the established environment (including strategies such as dynamic expansion), established data (including data changes in scenario execution), established execution strategies, and established monitoring, while observing changes in performance status parameters at all levels of the system , and judge in real time whether the analysis scenario meets expectations
Scenario classification: benchmark performance test, capacity performance scenario, stability performance scenario, abnormal performance scenario

Analysis and tuning of performance testing

Classification of performance items
- New system performance test class: such projects generally require testing the maximum capacity of the system
- Old system new version performance test category: Such projects are generally compared with the old version. As long as the performance does not decline, the capacity can be calculated based on historical data, and the requirements for tuning are generally not large.
- New system performance test optimization class: This type of system not only needs to test the maximum capacity, but also needs to be tuned to the best.

Performance testing must have a result report

Contents include: TPS before and after tuning, response time, resource comparison chart

What is the relationship between TPS and response (RT)

In the actual performance test, assuming that the number of concurrent users is increased by gradient, the TPS will rise slowly at the beginning, and the RT will remain at a low level for a period of time at this time; as the pressure continues to increase, the TPS will also rise. RT also rises slowly; when TPS reaches the limit and the pressure continues to increase, RT will rise rapidly and finally reach the timeout

Performance

demand indicator

business indicators
- Indicators at the business level, if the business side requires 10 million online users, then it can continue to be split into n performance scenarios, and each performance scenario also has a corresponding fixed-value business ratio
technical indicators
- RT Response Time (ReSponse Time), commonly referred to as response time, includes Request Time and Response Time
- HPS hits per second (Hit Per Second)
- TPS Transactions Per Second
- QPS refers to the number of sql per second in mysql
- RPS requests per second
- CPS HTTP return codes per second
- PV Page Views
- UV unique visitor
- IP independent IP number
- Throughput throughput
- IOPS usually describe disk

The concept of performance indicators

TPS
- The granularity of TPS needs to be defined according to the scenario; if it is an interface layer performance test, then T is the interface level; if it is a business level performance test, T can be directly defined as each business step and complete business flow.
concurrent users
- Absolute concurrency: the number of concurrency at the same time
- Relative concurrency: the number of concurrency in a period of time
- Using TPS to carry the concept of concurrency
Number of online users and concurrent users
- The number of online users refers to users on the system within a certain period of time, and these users may not necessarily perform actions
- The specified number of concurrent users is the number of users who perform actions on a certain service by the above-mentioned online users within a certain period of time (number of concurrent users = number of online users * degree of concurrency)
official

Performance Analysis Ideas

general idea
- Accurate judgment of bottlenecks
- Strategies for thread increments
- The process of performance degradation
- Split of Response Time
- Building Analysis Decision Trees
- scene comparison
Accurate judgment of bottlenecks
- TPS curve
- Assuming that the threads are increased in equal proportions, for the above figure, we can see that the performance bottleneck has already appeared in the second step, because theoretically the TPS of the second step should be twice that of the first step, but it is not the case. So there is a performance bottleneck
- The meaning of TPS (information obtained from the TPS curve)
  - Is there a bottleneck: In fact, it is accurate to say that all systems have performance bottlenecks, it just depends on which order of magnitude we are doing performance testing.
  - Is there any relationship between bottleneck and pressure: TPS changes with pressure, so there is a relationship. Regardless of whether the pressure increases or not, TPS will have a curve trend problem, which is irrelevant.
- response time curve
  - Key points of TPS curve and response time curve
    - Response time is used to judge how fast the business is, while TPS is used to judge the capacity.
Strategies for thread increments
- Two thread increment scenarios
- Summarize
  - For a system, if only the pressure strategy is changed (other conditions such as environment, data, software and hardware configuration, etc. remain unchanged), the maximum TPS upper limit of the system is fixed.
  - Regarding the test of the seckill scenario, warm-up must be done in the early stage. Preheating refers to running with a certain amount of traffic, and then suddenly increasing the pressure. This is more similar to the actual scene; instead of directly entering the system with a large amount of traffic at one time.
The process of performance degradation
- As long as the TPS per second per thread starts to decrease, it means that a performance bottleneck has appeared. However, after the bottleneck appears, it does not mean that the processing capacity of the server (here we use TPS to describe it) will decrease. It should be said that TPS will still rise. In the process of continuous performance decline, TPS will reach the upper limit.
Split of Response Time
Building Analysis Decision Trees
- It sorts out the structure, sorts out the system, sorts out the problems, sorts out the process of finding the evidence chain, and sorts out the analysis ideas. It plays a guiding role in looking at the overall situation and looking at the top.
scene comparison
- When you feel that a link in the system is not working, and you have no ability to analyze it, you can directly increase the link.

parametric data

parametric logic
- Analyze business scenarios
- List the data that needs to be parameterized and the corresponding relationship
- Take parameterized data out of the database or design corresponding generation rules
- Reasonably keep parametric data in different files
- Set the corresponding parameter combination relationship in the pressure tool in order to simulate the real scene

Performance Scenarios: Things to Consider Before Doing Parameterization

data that needs attention
- Parameterized data, monitoring data and basic laying data

parametric data

Possible situations for parameterized data
- data imbalance
- Insufficient amount of parameterized data

Questions about parametric data

How much data should be used for parametric data?
Where does the parametric data come from?
- Parametric data is mainly divided into two categories
  - The data entered by the user already exists in the background database, such as the user data in our example above.
    - Data characteristics: stored in the background database; user input is required; the data entered by the user will be compared with the data in the background database.
    - Such data must be queried into the database before being parameterized into the tool.
  - The data entered by the user does not exist in the backend database. In the business flow, these data will be Insert or Update to the database.
    - Data characteristics: These data do not exist in the database originally; these data will be inserted or updated into the database after the script is executed successfully; the data entered by each user may be the same or different, depending on the business characteristics.
    - Such data must be parameterized by pressure tools and must also meet business rules.
      - Data satisfaction conditions: to meet the distribution of data in the production environment; to meet the data volume requirements in performance scenarios.
What effect does the choice of more or less parameters have on the system pressure?
- If too many parameters are obtained, the pressure on the system will be great; if too few parameters are obtained, which does not conform to the data volume in the real scene, the real pressure of the system cannot be tested.
Is the histogram of the parametric data balanced in the database?
- It refers to whether the data distribution of each user conforms to the business scenario; for example, for the same order business, it is obviously unreasonable to create hundreds of thousands of data for user A and a few pieces of data for user B

Performance Scenario Design

preliminary work

Confirm the business that needs to be tested, and the corresponding business proportion of these businesses (can be obtained from the log)
Determine business target TPS
Determine business target response times

Baseline Performance Scenario

Purpose
- In order to test the maximum capacity of a single service, in order to determine which service has the most impact on the overall capacity in a mixed capacity scenario.

Capacity and Performance Scenarios

main points
- scene constantly
- control ratio
Capacity TPS Calculation Method
- Just add up the TPS of each business

Stability Performance Scenarios

main points
- Stability generally emphasizes that the system runs stably for a period of time, such as requiring 20 million business volume to run safely online for a week
- Minimum test duration = 2000w / capacity TPS (this value depends on the calculation method of capacity and performance scenarios)

Abnormal Performance Scenarios

Test Methods
- Generally speaking, it is to make various services unstable; for example, the main redis is down, and see if redis switching will cause functional problems

Performance Monitoring Design

Monitoring Design Steps

Analyze the architecture of the system; monitor individual components
There must be levels and steps in monitoring; first the overall situation, and then targeted quantitative analysis
Analyze the overall, directional, and hierarchical monitoring data, and then decide what information to collect next based on the analysis results, and then find a complete chain of evidence

Global Monitoring Design

os layer

Focus on parameters: CPU, I/O, Memory, Network, System, Swap

CPU parameter	Parameter meaning
idle	Percentage of time the CPU is idle
iowait	Percentage of CPU time spent waiting on I/O
irp	to interrupt
nice	Percentage of CPU time spent running normal processes
softirp	soft interrupt
steal
system	Percentage of CPU time consumed by system processes
user	Percentage of CPU time consumed by user processes
CPU queue

IO/Disk parameters	Parameter meaning
TPS	Total I/O transfers to physical devices per second
rrqm/s	The number of read operations merged per second
wrqm/s	The number of write operations merged per second
r/s	The number of read I/O devices completed per second
w/s	The number of write I/O devices completed per second
bi	The total amount of data read by the block device, that is, read disk
bo	The total amount of data written to the block device, that is, written to disk
r_await	Indicates the average response time for reads
w_await	Write average response time

Memory parameter	Parameter meaning
total	Total physical memory size
free	Available memory (see available)
used	used memory
Buff/cache	Total buffer memory
available	real usable memory

Network parameters	Parameter meaning
TX: send traffic
RX: receive traffic
Send-Q/Recv-Q	send queue, receive queue
full connection queue
semi-join queue

System parameters	Parameter meaning
interrupt	Indicates the number of device interrupts per second observed in a certain time interval
Context switch	The number of context switches generated per second

Swap	Parameter meaning
total	Total exchange area
free
used
and	The memory size of the memory entering the memory swap area
so	The memory size of the memory swap area entering the memory

middleware

message queue
- Indicators include: production speed and consumption speed
- Assuming that there is message accumulation in rabittmq, the solution
  - Increase consumers (this reason is that the speed of consumption cannot keep up with the speed of production)
  - 若增加消费者也不能解决，那么有可能是服务端出bug了，导致无法消费
redis
mysql