The most detailed in the whole network, performance test - performance analysis cpu usage is too high (practical case)

Contents: Guide

foreword

Architecture:
VM1: used as a web server to simulate performance issues
VM2: used as a client of the web server to increase the pressure on web service requests
Two virtual machines (both Ubuntu 18.04) are used to isolate each other and avoid cross-infection

VM2 runs the ab command (preliminary observation of Nginx performance)

A brief introduction to the ab command
ab (apache bench) is a commonly used HTTP service performance testing tool
that can send requests to the target server concurrently

Run the ab command
to test the Nginx performance of VM1 with 10 concurrent requests, and test 100 requests in total

ab -c 10 -n 10 http://172.20.72.58:10000/

As can be seen from the output of ab, the average number of requests per second that Nginx can bear is only 14.73 (poor)

So what went wrong?
Next, use a series of commands to see what went wrong

In-depth analysis

VM2 runs the ab command for a long time
with 10 concurrent requests to test the Nginx performance of VM1, and a total of 10,000 requests are tested

ab -c 10 -n 10000 http://172.20.72.58:10000/

Run the top command on the VM1 terminal
and press 1 to check the utilization rate of each CPU

Result analysis:
the CPU usage of several php-fpm processes in the system adds up to nearly 200%,
and the user usage (us) of each CPU has exceeded 96%, which is close to saturation

Conclusion: It is the php-fpm process in user space that causes the CPU usage to soar

Analyze the php-fpm process because of which function causes the CPU usage to increase
Run the perf command on the VM1 terminal

perf record -g -p 84408

record: the meaning of recording
-g: open call relationship analysis
-p: specify the process number of php-fpm 84408

After recording for about 30s, ctrl+c terminates the process, and then you can see the perf.data file in the current directory

Then execute the following command to analyze the report (perf.data)

perf report

Press the arrow keys to switch up and down, and press the Enter key to expand if there is a +

Result analysis:
In the end, it is related to the two functions of sqrt and add_function

View the source code of the Nginx application (find the source of the problem)
and find the sqrt function

grep sqrt -r app/

It turned out that only the sqrt function was called in the app/index.php file

Find the add_function function

grep add_function -r app/

You will find that it cannot be found, because add_function is a built-in function of PHP

View index.php source code:

<?php
// test only.
$x = 0.0001;
for ($i = 0; $i <= 1000000; $i++) {
    
    
  $x += sqrt($x);
}

echo "It works!"

As you can see, here is a code segment that loops many times

Solution

Find the root of the problem, you can quickly solve it, delete the loop code block

<?php

echo "It works!"

perf extension
In fact, there is a command to view functions more conveniently

perf top -g -p 84408

Then why do I use perf record and then use perf report?
Because if there is no perf source code, the function of php cannot be read, and only a bunch of hexadecimal codes will be displayed.

After fixing the issue (to verify that there is no change in Nginx performance)

VM2 terminal run ab command again

ab -c 10 -n 10000 http://172.20.72.58:10000/

Result analysis:
The number of requests per second has risen by leaps and bounds to 2500, which is much better than the previous 14

Analyze the overall idea:

Using ab short-term pressure test, it is found that the server performance is low and TPS is relatively low;
using ab long-term pressure test, the server maintains a high-load state, which is convenient for troubleshooting;
monitor system resources through the top command, and find the CPU usage in user mode The rate (us) is very high, and the CPU usage rate of several processes is found to be particularly high in the process list, all of which are the same php process;
record the process for a period of time through the perf record command;
view the result data of the recording process through perf report, you can analyze Which function of the php-fpm process causes high CPU usage;
find the add_function and sqrt functions;
use the grep command to confirm that the function is in the index.php code file;
check the source code of the index.php file to confirm the problem code block;
delete the problem code Block, re-test, the average number of requests per second has a qualitative increase, and the problem is successfully solved;

The following is the most complete software test engineer learning knowledge architecture system diagram in 2023 that I compiled