Summary from a 13-year veteran, summary of performance testing methods + troubleshooting methods for slow performance response (full details)

---

Preface

1. What does performance testing include?

1) Load testing

Load testing refers to testing and evaluating the performance of software products under specific conditions. Testers can evaluate the performance of software products under different conditions by simulating different numbers of users, concurrent requests, access frequencies and other conditions.

2) Strength test

Strength testing refers to testing and evaluating the performance of software products under limited resources. Testers can evaluate the performance of software products under resource-constrained conditions by simulating resource constraints, such as limiting CPU usage, memory capacity, etc.

3) Database capacity test

Database capacity testing refers to inserting a certain amount of data to evaluate the performance of a software product when processing large amounts of data. Testers can observe the performance of software products by simulating the insertion operation of large amounts of data.

4) Concurrency testing

Concurrency testing refers to testing and evaluating the performance of software products when multiple users access the software products at the same time. Testers can observe the performance of software products under concurrent access conditions by simulating multiple users accessing software products at the same time.

5) Capacity test

Capacity testing refers to evaluating the performance of a software product under a given number of virtual users by constructing record numbers of different orders of magnitude under certain software and hardware conditions.

Testers can simulate the number of users at different levels to observe the performance of software products under different loads.

6) Stability test

Stability testing refers to testing and evaluating the performance and stability of software products during their operation. Testers can simulate actual scenarios, such as network failures, load peaks, etc., to evaluate the performance and stability of software products in the face of abnormal situations.

7) Configuration test

Configuration testing refers to testing to verify and confirm the configuration information of software products before they are released. Testers can ensure that the software product can run normally after release by checking and verifying whether the configuration information of the software product is correct.

8) Network test

Network testing refers to testing and evaluating the performance of software products in a network environment. Testers can evaluate the performance of software products in different network environments by simulating different network environments, such as a stable LAN environment and a busy WAN environment.

9) Security testing

Security testing refers to testing and evaluating software products on the premise of ensuring data security and user privacy.

When performing performance testing, you can also perform stress testing on software products to evaluate their performance under different conditions.

Stress testing is to test the performance of a software product or system under high load conditions by simulating a large number of users accessing it at the same time. By stress testing software products, performance bottlenecks of software products under high load can be discovered and optimized.

2. How to troubleshoot if the performance test response is very slow?

Most performance testing staff are divided into the following three stages:

1) If there is a problem, look at the resources. If the resource usage is very high, report it with a happy attitude. Well, I discovered it. The principle is a resource bottleneck.

2) There is no bottleneck in resources. Through some technical analysis, it is found that there is a problem with the component configuration file, for example: there is a problem with the server's concurrency strategy, there is a bandwidth problem, and the shortcomings in the line performance are found. At this stage, I think it's a pretty cool test.

3) If there are no problems with the above, considering the data structure and algorithm that I have personally come into contact with, most people are now looking up to the second stage and exploring the third stage.

We hope to discover performance problems at the code level and discover and solve them. This is also in line with our theory that the earlier bugs are discovered, the lower the cost of repairing them. At the same time, it is also a symbol of the high salary of a performance test engineer.

Where to start with performance test tuning, as follows:

TPS fluctuates greatly:

Cause analysis: There are generally three reasons for large TPS fluctuations: network fluctuations, competition for other service resources, and garbage collection issues.

The performance test environment is usually on the intranet or the stress test machine and the service are on the same network segment. This can be checked by monitoring the incoming and outgoing traffic of the network;

Competition for other service resources may also cause this problem. You can use the Top command or service sorting method to check whether other services are running during the stress test, causing resource competition;

Tuning plan:
For network fluctuation problems, you can ask operation and maintenance colleagues to help solve the problem (such as switching network segments or selecting intranet stress testing), or wait until the network is more stable to perform stress testing. Test verification;

Resource competition issues: Through command monitoring and service sorting, find out other services that are running during the stress test, and stop the service through communication and coordination (or change the service node without resource competition and re-stress the test);

Garbage collection problem: Through GC file analysis, if frequent FGC is found, you can modify the JVM heap memory parameter Xmx, and then perform another stress test to verify (the maximum value of Xmx should not exceed 50% of the service node memory!)

A large number of errors are reported under high concurrency:

Reason analysis: Common reasons include the port being fully occupied due to short connections, the configuration of the maximum number of threads in the thread pool being small, and the timeout being short.

Tuning plan:
Short connection problem: Modify the tcp_tw_reuse parameter of the service node to 1 and release the TIME_WAIT scoket for new connections;

Thread pool problem: Modify the configuration parameters in the server.xml file of the container in the service node.

In a cluster system, the load of each service node is unbalanced:

Cause analysis: The reason for this type of problem is generally that the SLB service has set session persistence, which will cause the request to be distributed to only one of the nodes.

Tuning plan:
You can modify the session retention parameter of the SLB service (F5/HA/Nginx) to None, and then perform stress testing again;

The number of concurrencies continues to increase, TPS cannot increase, and CPU usage is low:

Reason analysis: SQL does not create an index/SQL statement filtering conditions are unclear, there is a synchronization lock in the code, and lock waiting occurs when concurrency is high;

Tuning plan:
SQL problem: create an index if there is no index, optimize SQL and business logic if the SQL statement filter conditions are not clear;

Synchronization lock issue: Whether to remove synchronization locks is sometimes not just a technical issue, but also involves various judgments of business logic. Whether to remove synchronization locks, it is recommended to communicate with product development colleagues to confirm;

During the stress test, TPS continued to decrease and CPU usage continued to decrease:

Cause analysis: The reason for this problem may be caused by thread block, of course, other possibilities are not ruled out;

Tuning plan:
If it is a thread blocking problem, modify the thread policy and then re-verify;

Key points that should be paid attention to when tuning performance tests:

Point 1: In the design and development process of application systems, performance should always be taken into consideration.
Key Point 2: Defining clear and unambiguous performance tuning goals is key.
Point 3: It is necessary to ensure that the performance-tuned program runs correctly.
Point 4: System performance depends more on good design, and tuning skills are only an auxiliary means.
Point 5: The performance tuning process is an iterative and progressive process, and the results of each tuning must be fed back to subsequent code development.
Point 6: Performance tuning cannot be done at the expense of code readability and maintainability.

The following is the most comprehensive software testing engineer learning knowledge architecture system diagram in 2023 that I compiled.

1. Python programming from entry to proficiency

2. Practical implementation of interface automation projects

3. Web automation project actual combat

4. Practical implementation of App automation project

5. Resumes of first-tier manufacturers

6. Test and develop DevOps system

7. Commonly used automated testing tools

8. JMeter performance test

9. Summary (little surprise at the end)

Success is not something that happens overnight, it requires our unremitting efforts and hard work. Only through continuous learning and accumulation can we achieve substantial progress and success.

No matter where you are, no matter what difficulties you encounter, you should never give up easily. As long as your heart is full of hope, you can always find a solution to the problem and move towards success.

Everyone is a piece of gold, but some people are buried in the excavation, and some people shine in the smelting. So, be brave enough to shine and let the world see your uniqueness.