Performance testing plays an important role in ensuring software quality, and performance testing is an essential link for application systems with large transaction volumes.
Testers usually measure whether the performance of the system is up to standard by monitoring the response time, throughput, CPU and memory of the application server and database server. Then, in the process of performance testing, in the face of some abnormal index data, how should we peel the data layer by layer? Cocoon, find the cause of the problem? How to focus and solve performance problems? This time, the author will summarize the experience of mobile terminal performance testing and share with you how to deal with abnormal CPU curves.
1. Problems found in performance testing
This time, the performance test is conducted for the mobile application. The transactions of the performance test are query transaction 1, application transaction 1, application transaction 2, query transaction 2 and query transaction 3.
During the 8-hour mixed fatigue test on 5 transactions, it was found that the CPU usage curve of the application server showed regular anomalies: after a few hours of fatigue stress in the mixed scene, the CPU usage increased sharply until the stress was over, and the curve As shown in Figure 1.
Figure 1 Application server CPU curve
2. Problem analysis process
Figure 2 Problem Analysis Flowchart
1. When the problem was found, the preliminary guess was that the CPU increased sharply due to the unstable environment. The mixed 8-hour fatigue test was performed again, and the test results were found to be consistent with the original results, excluding environmental factors.
2. Query the running log of XMeter, and find that there is an error message of Java.lang.OutOfMemoryError: Java heap space, and then analyze the Heapdump file, and find no code block that causes the problem. Then modify the heap memory configuration from the original 1.5G to 2.5G, and perform another mixed fatigue stress test for 8 hours. The stress test results are shown in Figure 3.
Figure 3 Application server CPU curve
The pressure test results show that the CPU usage curve of the application server begins to increase sharply for a longer time than before. When the heap memory is 1.5G, the CPU usage increases sharply in about 2-3 hours from the beginning of the pressure, until the end of the pressure; When the memory is 2.5G, the CPU usage increases sharply about 5-6 hours after the start of the pressure, until the end of the pressure.
3. The project team further verified the parameter configuration of the environment, expanded the server from 2C/4G to 2C/8G, increased the number of database connection pools and the number of WAS threads, and conducted a mixed fatigue stress test again. The test results showed that the CPU still experienced a sharp increase. Exclude environment parameter configuration factors.
4. The testers conducted a comprehensive analysis of the application server CPU curve, database server CPU curve, TPS curve and request response time curve, and found that while the application server CPU increased sharply, the database CPU and TPS curve showed a downward trend, and the average response time curve increased , so the cause of the problem is initially focused on the application server program problem.
(1) Request response time
Figure 4 Request response time curve
(2) Request throughput
Figure 5 Request throughput curve
(3) Database server CPU curve
Figure 6 Database server CPU curve
Analyzing the native_stderr.log, it is found that the JVM heap memory usage is constantly increasing, the heap memory recovery is abnormal, and there is a memory leak:
Figure 7 JVM heap memory usage curve
5. After confirming that it is a program problem, locate the transaction that caused the problem by performing pressure tests on different transactions:
(1) Separate pressure test for mobile transactions
Since the time node of the sudden increase is 3 hours after the pressure is released, the pressure test is carried out for each transaction for 4 hours, and then the mixed pressure test is carried out for different transaction combinations for 4 hours, and the phenomenon of CPU sudden increase does not appear.
(2) Pressure testing of PC and mobile transactions
① Since only the application transaction 2 among the 5 transactions is an old transaction, the fatigue stress test was performed on the application transaction 2 on the PC side for 8 hours, and the JVM heap memory usage curve was found to be normal;
② Performed mixed fatigue stress testing for 8 hours on the 4 transactions of the application transaction 2 on the PC side and the mobile terminal: query transaction 1, query transaction 2, and query transaction 3, and found that the JVM heap memory usage curve was normal;
③ Apply for transaction 2 on the PC side, mobile terminal: Query transaction 1, Query transaction 2, Query transaction 3, Application transaction 1 These 5 transactions were mixed fatigue stress test for 8 hours, the problem reappeared, and the test results are shown in Figure 8.
Figure 8 Application server CPU curve
The JVM heap memory usage curve is shown in Figure 9.
Figure 9 JVM heap memory usage curve
It was initially confirmed that the application transaction 1 caused the problem. Then, the fatigue stress test was performed on the application transaction 1 for 8 hours. The problem recurred, and it was confirmed that there was a problem with the transaction. The test results are shown in Figure 10.
Figure 10 Application server CPU curve
The JVM heap memory usage curve is shown in Figure 11.
Figure 11 JVM heap memory usage curve
3. Solutions
The project team debugged the codes of application transaction 1 and application transaction 2 step by step, located the code block with memory problems, and then compared the transaction codes of the PC terminal and the mobile terminal, and found that the transaction code of the mobile terminal did not clean up the cached data of the application form , after adding the method of clearing the cached data, the mixed fatigue stress test was performed on 5 transactions again for 8 hours, and the CPU curve of the application server was stable, and there was no sudden increase.
The code block uses the thread context method to cache the data of each application form. Using this method, you need to use the remove() method to clean it up, which can speed up the recycling of the JVM. Otherwise, in the case of high concurrency, there will be The JVM heap memory usage has been increasing, and the heap memory recovery is abnormal.
4. Test summary and reflection
During the performance test, response time, throughput, and CPU are key indicators for measuring performance. When the response time and throughput meet the passing criteria, it does not mean that the performance must be normal. You must also pay attention to whether the CPU curve is normal.
-
Check relevant logs in time, such as XMeter logs, Heapdump files, java core files, and native_stderr.log files, and analyze the heap memory through log files to see if there are recovery exceptions, memory leaks, or memory overflows.
-
The pressure test time for the problem to reappear is at least 8 hours or more. In this performance test, the pressure test time for the problem to reappear is 3-5 hours. Insufficient pressure test time leads to the failure to reproduce the problem well.
-
After the log shows that there is a memory leak, you should focus on checking the code instead of continuously changing the configuration and parameters.
-
The test results should be comprehensively analyzed in a timely manner, including the application server CPU curve, database server CPU curve, request throughput and request response time, which will help to quickly locate the source of the problem.
-
Since the heap memory will be cleaned up after the server is restarted, in order to make each test start from the initial state, the server should be restarted before the test.
Finally: The complete software testing video tutorial below has been organized and uploaded, and friends who need it can get it by themselves [Guaranteed 100% free]
Software Testing Interview Documentation
We must study to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Ali, Tencent, and Byte, and some Byte bosses have given authoritative answers. Finish this set The interview materials believe that everyone can find a satisfactory job.
Acquisition of complete set of information
