Cost reduction ideas
Within Company A, the online business stability standard is: when the business-approved TPS is reached, the peak CPU utilization does not exceed 60%, which is a stable state.
Against this background, the testing department conducted unified sorting and special performance testing for some systems with high CPU consumption, aiming to reduce the peak CPU utilization of the tested application through performance testing and tuning when meeting business TPS requirements. If the peak CPU utilization drops significantly, reduce the number of CPU cores of the deployed nodes and conduct retesting based on the business scenario. If it is found that the peak CPU utilization is still lower than 60% and there is no abnormality in TPS, the number of CPU cores of the production environment deployment node will be reduced to release more resources for the deployment of other services. In conjunction with production monitoring methods, if there are no abnormalities within a certain period of time, the cost reduction will be considered successful.
The specific idea is shown in the figure.
The cost reduction steps shown in the figure are explained below.
1) Performance stress testing: Conduct high-concurrency stress testing on the application under test.
2) Performance evaluation: Evaluate possible optimization points by looking at the performance indicators during the performance test process. Performance indicators include TPS, response time, CPU utilization, memory usage, code response time, full-link topology, etc.
3) Performance analysis: including the following three dimensions.
❑Conduct CPU hotspot analysis and thread tracking analysis for tested applications with high CPU consumption. Use visual reports to view the code segments and thread IDs of applications running when CPU utilization is high, and conduct in-depth analysis of code segment source code and thread call stacks to locate the root cause of high CPU utilization.
❑For tested applications with high memory consumption, through memory perspective analysis, we can obtain JVM's new generation, old generation, GC, off-heap memory, memory objects, class objects and other indicators in a low-cost method. View abnormal indicators or curves in real time through visual reports, analyze their corresponding memory objects, and locate the root cause.
❑For tested applications with long code response times, through sub-method call analysis, real-time bytecode enhancement technology is used to conduct microsecond-level time-consuming analysis of sub-method calls triggered in the code. After locating the most time-consuming code segment, start source code analysis to view the source code logic and locate the root cause.
4) Performance tuning: Communicate with the project team to fix the found CPU, memory, thread, and code problems.
5) Cost reduction verification: Verify the optimized application under test using the same scenario and number of concurrencies to ensure that the CPU indicators and memory indicators of the application under test are improved. For the application under test that has been confirmed to be optimized, reduce the CPU and memory configurations, and then conduct a performance test using the same scenario and number of concurrencies to confirm that the performance indicators after downgrading meet the online standards.
If the measured application indicators do not meet the online standards after downgrading, start from the performance evaluation step and perform the analysis again.
Cost reduction case
Based on the above cost reduction ideas, the actual system tuning process within Company A is as follows. 1) Description of the problem: In a high-concurrency stress test scenario, the average CPU usage of applications in the system is shown in Figure 10-19. The actual CPU utilization reaches 70.07%, and the highest is 100%.
Figure 10-19 CPU utilization of an application 2) Positioning process description: Through hot spot analysis on the performance analysis platform, it is found that most of the CPU is consumed in the log stack climbing operation.
CPU hotspot analysis chart
3) Determine performance risk points: Frequent log climbing under high concurrency will lead to excessive CPU consumption and affect the overall application experience.
4) Put forward optimization suggestions: From the perspective of development specifications, it is not recommended to locate by line number, because the cost of doing so is too high and wastes too much resources. It is best to use the output log string with accurate signs to clarify which paragraph There is a problem with the code logic. For this case, first of all, it is recommended to delete the parameters %I and %L that print the line number. Secondly, delete the %method or %m configuration. Currently, Debug, Info, Warn, and Error level logs use the same set of output formats, that is, the pattern configuration is the same. It is recommended that Debug and Info do not print method names and line numbers, but directly print Warn and Error is fine.
5) Actual optimization effect: By modifying the configuration, TPS increased from 260 transactions/second to 320 transactions/second, an increase of more than 25%.
6) Cost reduction effect: After the application server configuration is reduced by 50%, the overall indicators still meet the target value.