First, the section reviews
Box update now, let's column last part - Integrated Case modules to a close. Good to see you have not left behind, still in active learning thinking, hands-on, and share your passion in a real environment, we encountered various performance problems
analysis of ideas as well as optimization methods.
Today is the sixth performance optimization of Q & A. As usual, I have a message from the integrated module of the case, pick out some typical problems, as today's Q & A content, focused response. In order to facilitate you learn to understand, they are not strictly in accordance with article order. Each
question, I have attached a screen shot of the message area questions. If you need to review the content of the original, you can scan two-dimensional code to see the bottom right of each question.
Second, Question 1: container cold start performance analysis
1, problem
2. Answer:
Why, after application of container, started a lot slower, we analyzed the application of the container caused by the problem of slow start together. A brief review of the case at that time, Docker container to set the memory limit by Cgroups, but the vessel did not
realize that they still allocate too much memory, causing the system OOM killed.
The root of the case is actually relatively simple, Tony students this raised a deeper problem.
We know that container management applications bring great convenience, such as Serverless (only concerned with running the application, without the need to focus on the server), FaaS (Function as a Service ) and other new software architecture, also based container technology structure
was built. However, although the container has to start soon, but in starting a new vessel, which is a cold start when the start time is relative to the performance requirements of the application, that was too long.
So, how should a cold start to analyze and optimize the performance of it?
This problem is most central point, in fact, is to figure out where to start all spent time in the end. In general, start a Serverless services, including:
- Event triggers (such as HTTP receive a new call request);
- Resource Scheduling;
- Pulling mirror;
- Network Configuration
- Start several process applications.
These processes are time consuming, it can be monitored by way of the link trace, and thus can target the most time-consuming one or more processes.
Then, for the most time-consuming process, we can monitor the application or by the method of dynamic tracking, locating the most time-consuming character module, which will be optimized to find the bottleneck points.
For example, the mirror pulling process, the mirror can be reduced by caching time pull hot mirror; network configuration process can be accelerated by pre-allocating network resources; containers and start the resource scheduling, multiplexing may be predetermined by the created container optimize.
Three, perf probe failure how to do?
1, problem
2. Answer:
Kernel threads in high CPU utilization case, we went through fire and perf drawing tool to generate a dynamic hot spot vector kernel function call stack, and locate when performance problems occur, most frequently executed kernel functions.
As the case study, our main concern is the CPU busy, so this time the flame generated map, known as the on-CPU flame chart. In fact, in addition, there are off-CPU, memory, etc. FIG different flames, respectively blocking and CPU
memory allocation release situations.
Therefore, Li Xiao Yao students raised a good question: are equally flame chart, CPU and memory map flame flame drawing, in the end what is the difference in the generation of data?
This question, just ask on the most central point. Figure flame flame CPU and memory map, in fact, the biggest difference is the different sources of data, which is different from the function stack, and the flame graph format or exactly the same.
- FIG flame on the CPU, the data collection is primarily a function of the consumption of CPU;
- FIG flame and memory, the main data collection memory allocation, release, feed and other memory management functions.
For example, when we use perf record, the default collection event cpu-cycles, data is collected on-CPU, and the resulting figure is the CPU flame flame chart. By perf record -e page-fault events will be collected into page-fault
after, you can collect data memory page fault, the resulting flame Figure naturally became the flame of memory map.
Four, perf probe failure how to do?
1, problem
2. Answer:
How to use dynamic tracking, we have a few cases together, learning how to use the dynamic tracking tools perf, bcc and so on.
These dynamic tracking method, without modifying the code, do not restart the service, so that you understand the dynamic process of execution of the application or kernel. This investigation situation is complex, difficult to reproduce the problem is particularly effective.
When using dynamic tracking tools, because the function address in hexadecimal format is not easy to understand, we need to help debugging information, convert them to a more intuitive function names. For the kernel, I have repeatedly mentioned the need to install debuginfo.
However, for the application how should we do?
In fact, here are two ways.
The first method, if the application provides debugging information packages, then you can install directly to use. For example, in our case bash it, you can use the following command to install it debugging information:
# Ubuntu apt-get install -y bash-dbgsym # Centos debuginfo-install bash
The second method, using recompile the application source code, and turn on debugging compiler, for example can increase the -g option to gcc.
V. question 4: RED method micro-monitor application service
1, problem
2. Answer:
Integrated system monitoring thinking, I introduced the common law when USE resource performance monitoring system for you. USE law the performance of system resources, simplified to three: usage, saturation, and the number of errors. When the three indicators in any of the categories is too high, both
on behalf of the corresponding system resources may have performance bottlenecks.
However, the monitoring of the application, these indicators is clearly inappropriate. Because the application of core indicators is the number of requests, the number of errors and response time. Then how to do it? In fact, this is the RED method classmate Adam mentioned in the message.
RED method is Weave Cloud service performance when the micro-monitor, in conjunction with Prometheus monitor, for monitoring the proposed idea - i.e. micro services, the monitors their number (Rate) the request, the number of errors (the Errors) and a response time
(Duration). So, RED method is suitable for monitoring the micro-service applications, while USE method is suitable for monitoring system resources.
Six Question 5: In-depth kernel method
1, problem
2. Answer:
When positioning performance issues, when we troubleshoot a variety of ways perf, ebpf, systemtap, etc., are likely to find the hot issues of a function in the kernel's. The bluestone and xfan problem is how to understand the depth of the original Linux kernel
management, especially trying to figure out, performance tools show kernel function in the end what is the meaning.
In fact, to understand the meaning of the kernel function, the best way is to query the source code for the kernel version in use. Here, I recommend https://elixir.bootlin.com this site. It is also very simple to use, select the version of the kernel from the left, and then through the kernel function
name to search for it.
The reason why recommend this site because it not only allows you to quickly locate the search function, all functions, variables, macros, etc., provides a quick jump function also. In this way, when you see the function or variable does not understand, you can click to jump to the phase
defined at the corresponding.
In addition, for eBPF, in addition can be learned through the kernel source, I recommend you start from the BPF Compiler Collection (BCC) project. BCC provides many short example, can help you quickly understand how eBPF and cooked
noted the development of ideas eBPF program. Understand these basic usage, to go deep inside eBPF, it will be a lot easier.
Today my main answer these questions, but you are welcome to continue to write down questions and thoughts in the comments section, I will continually communicate with you in the comments section. We are hoping every Q & A and exchange, and you can be together,
The kinds of knowledge into the column of your ability.