One, CPU context switch test scenario
Use sysbench to simulate multi-thread scheduling: sysbench --threads=10 --time=300 threads run
Use vmstat to view CPU context switching: The 
number of context switches in the cs column exceeds 1.5 million.
The maximum length of the ready queue of column r is 8, which exceeds the number of system CPUs by 4, and there is a lot of CPU competition.
The sy column exceeds 70%, indicating that the CPU is mainly occupied by the kernel.
The number of interrupts in the in column rose to more than 40,000, indicating that interrupt handling is also a potential problem.
The system ready queue is too long, that is, there are too many processes running and waiting for the CPU, which leads to a large number of context switches, which in turn leads to an increase in the system CPU usage.
Use pidstat -wt 3 to view specific threads: 
The cswch of the sysbench thread exceeds 40,000, and the nvcswch exceeds 100,000.
Observe the interrupt usage of the CPU: watch -d cat /proc/interrupts
rescheduling interrupts (RES) more than 4 million times, used to wake up the idle CPU to schedule new tasks. In a multi-processor system (SMP), Inter-Processor Interrupts (IPI) is a mechanism used to distribute tasks to different CPUs.
Two, CPU load average test scenario
1. CPU-intensive processes
The first terminal runs the stress command to simulate a scenario where the CPU usage is 100%. stress --cpu 1 --timeout 600
Run uptime on the second terminal to check the average load change. watch -d uptime
Run mpstat on the third terminal to view the changes in CPU usage. mpstat -P ALL 5
Monitor all CPUs and output once every 5 seconds.
The 1-minute average load monitored by the second terminal uptime will slowly increase to 1.21.
The utilization rate of CPU3 in the third terminal is 96%, but the corresponding iowait is only 0. The increase in average load is due to the 100% CPU utilization rate.
The fourth terminal runs pidstat to check the CPU usage of the process and finds that the CPU usage of the stress process is 100%.pidstat -u 5 1
2. IO-intensive processes
The first terminal runs the stress command to simulate an IO-intensive process. stress -i 1 --hdd 1 --timeout 600
The second terminal runs uptime to check the average load change. watch -d uptime
The third terminal runs mpstat to view the changes in CPU usage. mpstat -P ALL 5
The fourth terminal runs pidstat to check the CPU usage of the process, and finds that the CPU usage of the stress process is relatively high.pidstat -u 5 1
3. Wait for the CPU process
When the running process in the system exceeds the running capacity of the CPU, a process waiting for the CPU will appear.
The first terminal runs the stress command to simulate the scenario of 10 processes. stress -c 10 --timeout 600
Since the system has only 4 logical CPUs, which is much less than 10 processes, the CPU of the system is in a serious overload state, with an average load of 9.96. 
mpstat view CPU usage is close to 100%. 
10 processes compete for 4 logical CPUs, and each process waits for the CPU time (%wait column) as high as 60%, which seriously exceeds the CPU's computing power and eventually leads to CPU overload.
Three, CPU usage test scenario
1. Open multi-threaded stress test
sysbench --threads=8 --time=600 cpu run
Use sysbench to start the 8-thread stress test, the duration is 600 seconds
2. Real-time analysis
sudo perf top -g -p 30388
Analyze the performance of sysbench process 30388 in real time. 
Select sysbench and press ENTER. 
Select the cpu_execute_eventh function with the largest CPU usage, and press ENTER to enter the view. 
Select Annotate cpu_execute_event and press ENTER to enter to view the CPU occupied by the internal call of the function. 
Judging from the cpu_excute_event function call, the modulo operation occupies more than 70% of the CPU usage.
3. Offline analysis
perf record -g -p 28068
Sampling sysbench process performance data to perf report
analyze performance data