Use Prometheus to monitor Springboot application reference Prometheus Operator actual combat - Prometheus, Alertmanager, Grafana to monitor Springboot service
Let's take a look at the monitoring indicators of jvm
# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
#This is a Summary indicator, similar to Histogram, which can sample indicator data
Concurrent collector CMS (Concurrent Mark-Sweep)
Garbage collectors that sacrifice throughput to minimize collection pause times
are ideal for applications that require server responsiveness.
CMS is used for the recovery of the tenured generation, that is, the recovery of the old generation. The goal is to minimize the pause time of the application, reduce the probability of full gc, and use the
garbage collection thread concurrent with the application thread to mark and clear the old generation. .
Added to the startup JVM parameter -XX:+UseConcMarkSweepGC, this parameter indicates that CMS is used for the recycling of the old age. The basic algorithm adopted by CMS is: mark-clear.
CMS does not organize and compress the heap space, which saves the pause time of garbage collection, but also wastes heap space. In order to solve the problem of wasting heap space,
the CMS collector no longer uses a simple pointer to point to an available heap space for the next object allocation.
Instead, some unallocated space is summarized into a list. When the JVM allocates object space, it will search this list to find a space large enough to hold the object.
Another disadvantage of CMS is that it requires a larger heap space. Because the threads of the application program are still executing during the CMS marking phase, there will be situations where the heap space will continue to be allocated.
In order to ensure that there is still space allocated to the running application program before the CMS reclaims the heap, a part of the space must be reserved.
There is not enough room on the heap for allocations until the collection is complete! By default, when the old generation uses 68%, CMS starts to act. – XX:CMSInitiatingOccupancyFraction =n to set this threshold.
In general, the CMS collector reduces the pause time of collection, but reduces the utilization of heap space.
If your application is sensitive to pauses, and you can provide more memory and more CPU (that is, the hardware is awesome) when the application is running, then using CMS to collect will benefit you.
Also, if in the JVM, there are relatively many objects with a long survival time (the old age is relatively large), it will be more suitable to use CMS.
ParNew Garbage Collector
Par is the abbreviation of Parallel, which means multi-threading,
but multi-threading here only refers to multi-threaded parallel garbage collection, not parallel running of garbage collection and programs. ParNew also needs to suspend all work, and then multi-threaded parallel garbage collection .
Parameter
"-XX:+UseConcMarkSweepGC": After specifying the use of CMS, ParNew will be used as the new generation collector by default;
"-XX:+UseParNewGC": Forced to specify the use of ParNew;
"-XX:ParallelGCThreads": Specify the number of threads for garbage collection , the default collection thread enabled by ParNew should be equal to the number of CPUs;
# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="ParNew",} 87.0 ==>YGC
jvm_gc_collection_seconds_sum{gc="ParNew",} 15.487 ==>YGCT
jvm_gc_collection_seconds_count{gc="ConcurrentMarkSweep",} 0.0 ==>FGC
jvm_gc_collection_seconds_sum{gc="ConcurrentMarkSweep",} 0.0 ==>FGCT
# HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_used gauge
#jvm已用内存区域
jvm_memory_bytes_used{area="heap",} 2.334615096E9 ==>堆内存使用
jvm_memory_bytes_used{area="nonheap",} 1.8031396E8 ==>非堆内存使用
Remark:
nonheap = "Code Cache" + "Metaspace" + "Compressed Class Space"
heap = "Par Eden Space" + "Par Survivor Space" + "CMS Old Gen"
Conclusion: init is approximately equal to the value of xms, and max is approximately equal to the value of xmx.
used is the used memory size, committed is the currently available memory size (including used), ### committed >= used.
When committed is insufficient, jvm applies to the system, and if it exceeds max, OutOfMemoryError occurs.
# HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_committed gauge
jvm_memory_bytes_committed{area="heap",} 8.267825152E9
jvm_memory_bytes_committed{area="nonheap",} 1.85270272E8
# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_max gauge
#jvm内存区域的最大字节数
jvm_memory_bytes_max{area="heap",} 8.267825152E9
jvm_memory_bytes_max{area="nonheap",} 1.59383552E9
# HELP jvm_memory_bytes_init Initial bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_init gauge
#jvm内存区域的初始化字节数
jvm_memory_bytes_init{area="heap",} 8.589934592E9
jvm_memory_bytes_init{area="nonheap",} 2555904.0
# HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_used gauge
#jvm内存池使用情况
jvm_memory_pool_bytes_used{pool="Code Cache",} 7.2995456E7
jvm_memory_pool_bytes_used{pool="Metaspace",} 9.6191616E7
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 1.1126888E7
jvm_memory_pool_bytes_used{pool="Par Eden Space",} 2.184666584E9
jvm_memory_pool_bytes_used{pool="Par Survivor Space",} 3342224.0
jvm_memory_pool_bytes_used{pool="CMS Old Gen",} 1.46606288E8
# HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_committed gauge
jvm_memory_pool_bytes_committed{pool="Code Cache",} 7.3596928E7
jvm_memory_pool_bytes_committed{pool="Metaspace",} 9.9876864E7
jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 1.179648E7
jvm_memory_pool_bytes_committed{pool="Par Eden Space",} 2.577006592E9
jvm_memory_pool_bytes_committed{pool="Par Survivor Space",} 3.2210944E8
jvm_memory_pool_bytes_committed{pool="CMS Old Gen",} 5.36870912E9
# HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_max gauge
#jvm内存池最大数
jvm_memory_pool_bytes_max{pool="Code Cache",} 2.5165824E8
jvm_memory_pool_bytes_max{pool="Metaspace",} 2.68435456E8
jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
jvm_memory_pool_bytes_max{pool="Par Eden Space",} 2.577006592E9
jvm_memory_pool_bytes_max{pool="Par Survivor Space",} 3.2210944E8
jvm_memory_pool_bytes_max{pool="CMS Old Gen",} 5.36870912E9
# HELP jvm_memory_pool_bytes_init Initial bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_init gauge
#jvm内存池初始化数
jvm_memory_pool_bytes_init{pool="Code Cache",} 2555904.0
jvm_memory_pool_bytes_init{pool="Metaspace",} 0.0
jvm_memory_pool_bytes_init{pool="Compressed Class Space",} 0.0
jvm_memory_pool_bytes_init{pool="Par Eden Space",} 2.577006592E9
jvm_memory_pool_bytes_init{pool="Par Survivor Space",} 3.2210944E8
jvm_memory_pool_bytes_init{pool="CMS Old Gen",} 5.36870912E9
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
#当前jvm已加载类数量
jvm_classes_loaded 16377.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
#从jvm运行开始加载的类的数量，这是一个Counter指标，递增
jvm_classes_loaded_total 16377.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
#jvm运行后卸载的类数量，这是一个Counter指标。生产环境一直是0
jvm_classes_unloaded_total 0.0
# HELP jvm_info JVM version info
# TYPE jvm_info gauge
jvm_info{version="1.8.0_151-b12",vendor="Oracle Corporation",runtime="Java(TM) SE Runtime Environment",} 1.0
# HELP os_free_physical_memory_bytes FreePhysicalMemorySize (java.lang<type=OperatingSystem><>FreePhysicalMemorySize)
# TYPE os_free_physical_memory_bytes gauge
os_free_physical_memory_bytes 1.9491221504E10
# HELP os_committed_virtual_memory_bytes CommittedVirtualMemorySize (java.lang<type=OperatingSystem><>CommittedVirtualMemorySize)
# TYPE os_committed_virtual_memory_bytes gauge
os_committed_virtual_memory_bytes 3.4423967744E10
# HELP os_total_swap_space_bytes TotalSwapSpaceSize (java.lang<type=OperatingSystem><>TotalSwapSpaceSize)
# TYPE os_total_swap_space_bytes gauge
os_total_swap_space_bytes 0.0
# HELP os_max_file_descriptor_count MaxFileDescriptorCount (java.lang<type=OperatingSystem><>MaxFileDescriptorCount)
# TYPE os_max_file_descriptor_count gauge
os_max_file_descriptor_count 1048576.0
# HELP os_system_load_average SystemLoadAverage (java.lang<type=OperatingSystem><>SystemLoadAverage)
# TYPE os_system_load_average gauge
os_system_load_average 3.84
# HELP os_total_physical_memory_bytes TotalPhysicalMemorySize (java.lang<type=OperatingSystem><>TotalPhysicalMemorySize)
# TYPE os_total_physical_memory_bytes gauge
os_total_physical_memory_bytes 2.02692759552E11
# HELP os_system_cpu_load SystemCpuLoad (java.lang<type=OperatingSystem><>SystemCpuLoad)
# TYPE os_system_cpu_load gauge
os_system_cpu_load 0.04950495049504951
# HELP os_free_swap_space_bytes FreeSwapSpaceSize (java.lang<type=OperatingSystem><>FreeSwapSpaceSize)
# TYPE os_free_swap_space_bytes gauge
os_free_swap_space_bytes 0.0
# HELP os_available_processors AvailableProcessors (java.lang<type=OperatingSystem><>AvailableProcessors)
# TYPE os_available_processors gauge
os_available_processors 48.0
# HELP os_process_cpu_load ProcessCpuLoad (java.lang<type=OperatingSystem><>ProcessCpuLoad)
# TYPE os_process_cpu_load gauge
os_process_cpu_load 0.0
# HELP os_open_file_descriptor_count OpenFileDescriptorCount (java.lang<type=OperatingSystem><>OpenFileDescriptorCount)
# TYPE os_open_file_descriptor_count gauge
os_open_file_descriptor_count 163.0
# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 0.00121965
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 0.0
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
#用户和系统的总cpu使用时间
process_cpu_seconds_total 2293.82
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.618365917041E9
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 163.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1048576.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 3.4423963648E10
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.942313984E9
# HELP jvm_buffer_pool_used_bytes Used bytes of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_bytes gauge
#jvm缓冲区使用情况，包括Code Cache(编译后的代码缓存，
不同版本的jvm默认大小不同)、PS Old Gen（老年代）、PS Eden Space（伊甸园）、PS Survivor Space（幸存者）、PS Perm Gen（永久代）
jvm_buffer_pool_used_bytes{pool="direct",} 1729229.0
jvm_buffer_pool_used_bytes{pool="mapped",} 0.0
# HELP jvm_buffer_pool_capacity_bytes Bytes capacity of a given JVM buffer pool.
# TYPE jvm_buffer_pool_capacity_bytes gauge
#给定jvm的估算缓冲区大小
jvm_buffer_pool_capacity_bytes{pool="direct",} 1729229.0
jvm_buffer_pool_capacity_bytes{pool="mapped",} 0.0
# HELP jvm_buffer_pool_used_buffers Used buffers of a given JVM buffer pool.
# TYPE jvm_buffer_pool_used_buffers gauge
#给定jvm的已使用缓冲区大小
jvm_buffer_pool_used_buffers{pool="direct",} 243.0
jvm_buffer_pool_used_buffers{pool="mapped",} 0.0
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
#jvm当前线程数
jvm_threads_current 284.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
#jvm后台线程数
jvm_threads_daemon 241.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
#jvm线程峰值
jvm_threads_peak 286.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
#jvm总启动线程数量，Counter指标
jvm_threads_started_total 1260.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
#死锁线程数量
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0