Linux performance optimization practice - how to learn Linux performance optimization

Performance

The first step in learning performance optimization is to understand the concept of "performance metrics".
"High concurrency" and "fast response" correspond to the two core indicators of performance optimization - "throughput" and "delay". These two indicators examine performance from the perspective of application load , and directly affect the user experience of product terminals. Corresponding to them are indicators from the perspective of system resources , such as resource usage and saturation.

Performance analysis is actually to find out the bottlenecks of the application or system, and try to avoid or alleviate them, so as to use system resources more efficiently to process more requests. This involves the following series of steps:

1. Select metrics to evaluate application and system performance;
2. Setting performance goals for applications and systems;
3. conduct performance benchmarks;
4. Performance analysis to locate bottlenecks;
5. optimize systems and applications;
6. performance monitoring and alerting;

Learning Objectives

Build a holistic view of overall system performance.
Brendan Gregg ( Brendan Gregg )'s Linux performance tool map is as follows.

You need to pay attention to the selection of tools, first understand the principles, and then combine with specific scenarios to integrate different components of the system and use them flexibly.

Learning Linux performance optimization is divided into the following parts:

1. CPU
   1.1 Process and CPU principle
   1.2 Performance indicators (average load, CPU usage, context switching, CPU cache hit rate)
   1.3 Performance analysis (top/ps, vmstat, mpstat, sar, pidstat, strace, perf, execsnoop, proc file system )
   1.4 Tuning methods (CPU binding, process CPU resource limitation, process priority adjustment, interrupt load balancing, CPU cache, NUMA optimization)

2. Memory
   2.1 Memory principle
   2.2 Performance indicators (system memory usage, process memory usage, cache and buffer hit rate, SWAP usage) 2.3
   Performance analysis (free, top, sar, vmstat, cachestat, cachetop, memleak, proc file system )
   2.4 Tuning methods (use cache and buffer, reduce SWAP usage, reduce dynamic memory allocation, optimize NUMA, limit process memory resources, use HugePage)

3. Network
   3.1 Network principle
   3.2 Performance indicators (throughput, delay, packet loss, TCP retransmission)
   3.3 Performance analysis (ethtool, sar, ping, netstat/ss, ifstat, ifconfig, tcpdump, wireshark, iptables, traceroute, ipcontrack, perf)
   3.4 Tuning methods (network card tuning, protocol tuning, resource control, kernel tuning)

4. Disk IO
   4.1 Disk principle
   4.2 Performance indicators (utilization, IOPS, throughput, IOWAIT)
   4.3 Performance analysis (dstat, sar, iostat, pidstat, iotop, iolatency, blktrace, fio, perf)
   4.4 Tuning methods (system bar, I/O resource control, full use of cache, RAID, I/O isolation)

5. File system
   5.1 File system principle
   5.2 Performance index (capacity, IOPS, cache hit rate)
   5.3 Performance analysis (df, strace, vmstat, sar, perf, proc file system)
   5.4 Tuning method (file system selection, use of file system cache , I/O isolation)

6. Linux kernel
   6.1 Kernel principle
   6.2 Performance analysis (BPF, perf, proc file system)
   6.3 Tuning method (kernel option)

7. Application program
   7.1 File system principle
   7.2 Performance indicators (throughput, response time, resource usage)
   7.3 Performance analysis (USE method, process analysis, APM)
   7.4 Tuning method (logic simplification, programming language, algorithm tuning, non-blocking I /O, utilizing caches and buffers, asynchronous processing and concurrency, garbage collection)

8. Architecture design
   8.1 Space for time (cache, buffer, redundant data)
   8.2 Time for space (compression encoding, page swap)
   8.3 Parallel processing (multithreading, multiprocessing, distributed)
   8.4 Asynchronous processing (asynchronous I/O, message queue, event notification)

9. Performance monitoring
   9.1 Time series analysis (historical trend analysis, performance model construction, future trend prediction)
   9.2 Service call tracking (service call process tracking, service call performance analysis, service call chain topology display)
   9.3 Data visualization (trend chart, scatter chart , heat map, pie chart)
   9.4 Alarm notification (threshold selection, alarm strategy, notification channel)
   10. Performance testing
   10.1 Clear requirements (system resource requirements, application requirements)
   10.2 Environmental assumptions (reasonable assumptions, production environment simulation, production load simulation)
   10.3 Performance testing (benchmarking, load testing, stress testing)
   10.4 Results analysis (application bottleneck, database bottleneck, system resource bottleneck)

key point

learn by doing