1. Background note
线上服务响应时间超过40秒，登录服务器发现cpu将近100%了(如下图)，针对此问题，本文说明排查过程、工具以定位具体的原因。
2. Analysis of the investigation process
此类问题的排查，有两款神器可用，分别是async-profiler和arthas，async-profiler主要用于全局分析，通过此工具可以找到热点方法，
再用arthas对此热点方法进行详细的追踪，trace命令可以追踪方法的具体耗时，watch命令可以查看方法的出入参数，在结合源代码可以比较
方便定位到问题原因。下面记录排查过程：2.1 with a flame async-profiler FIG.
After the download, after decompression as follows:
Run a flame FIG:
./profiler.sh -d 300 1485 -f ./test.svg
wherein the data collected represents -d 300 300s, and 300s after the end of automatic generation of test. svg file to the current directory.
2.2 Flame chart analysis
Svg browser opens the file generated above, as follows:
the flame in the longitudinal direction on behalf of FIG call stack, i.e. the call depth method, the method is transverse occupied cpu time ratio, so the flame plateau phenomenon occurs if the drawing shows the related method is time-consuming (i.e. hot spot method), analysis optimized object;
from FIG seen: java / util / ComparableTimSort.countRunAndMakeAscending method hotspot methods can be tracked correlation method this method call stack:
IO / micrometer / Core /instrument/MeterRegistry.getMappedId
IO / micrometer / Core / Instrument / Tags.and
2.3. Specific time-consuming and tracking method call relations
Arthas tools used at this time, to change the tool installation and use can refer to: " Arthas User's Guide "
2.3.1 Track getMappedId
execute the following command arthas console:
the trace io.micromete / core.instrument.MeterRegistry getMappedId
io.micrometer.core.instrument.MeterRegistry # getMappedId took nearly 40s, during this process, called 2458 times MeterFilter.map () method: the methods source code is as follows:
by command repeatedly found, filters array will increase request with rising; then the filters array in the end what is it?
2.3.2. Monitoring classes return objects
Watch MeterRegistry "returnObj {}". 3 -n. 3 the -X-
storage array is the tag found in the same content continues to view the source code, the source io.micrometer.core.instrument.MeterRegistry class member variable assignment:
2.3.3. Monitoring Tags and methods to the Senate
watch io.micrometer.core.instrument.Tags and "{params}" -x 2 -b -n 4
2.4. The reason
commonTags once, +1 Filters array length per call, will conduct a Copy, growing, and the cycle of operation for tag, getMappedId sorting process, to result in heavy cpu soar;
The following is a simulation result in the problem of code streamlining reasons:
2.5. Solution
commonTags should be initialized when the service settings, such as in spring boot configuration file:
