Table of contents
2. Traditional performance testing
3.1. Maven-related dependencies
4. Basic attribute configuration
1 Introduction
In software development, in addition to writing correct code, you also need to write efficient code. This is even more important in concurrent programming for two main reasons. First of all, some concurrent programs are transformed from serial programs, the purpose of which is to improve system performance. Therefore, it is natural to have a method to compare the performance of the two algorithms. Secondly, multi-threading introduced for business reasons may cause performance loss due to thread concurrency control, so it is necessary to evaluate whether the proportion of loss is acceptable. Whatever the reason for performance evaluation, quantitative metrics are always necessary. In most cases, it is not enough to simply answer who is faster and who is slower. How to quantify program performance? This is the Java micro-benchmarking framework JMH introduced in this section.
2. Traditional performance testing
In traditional performance testing, time stamps are usually printed before and after the method, and then the execution time is judged by the time difference.
public static void dealHelloWorld() throws InterruptedException {
// 这里模拟该方法执行
Thread.sleep(1000);
}
public static void main(String[] args) throws InterruptedException {
long start0 = System.currentTimeMillis();
dealHelloWorld();
long end0 = System.currentTimeMillis();
System.out.println("执行耗时：" + (end0-start0) + "ms");
}Results of the:
However, if the amount of code is large and complex, it is usually necessary to print more time stamps and then perform calculations in segments. like this:
In this case, on the one hand, a lot of computing time code will be incorporated into the business code to increase the readability of the code; Group unoptimized performance data participates in statistical calculations. Then JMH is needed at this time.
3. What is JMH
JMH (Java Microbenchmark Harness) is a micro-benchmark testing framework for the Java language, used to accurately and reliably measure and evaluate the performance of Java code. It was developed by the OpenJDK team specifically for performance testing and benchmarking of Java applications. The performance of multiple methods can be quantitatively analyzed by JMH. For example, when you want to know how long it takes to execute a function, or when there are many different implementations of an algorithm, you need to choose the one with the best performance.
JMH official website address: OpenJDK: jmh
Github address: https://github.com/openjdk/jmh/tags
4、Hello JMH
Let's try it out first. It is very simple to use JMH testing, we can think of the Junit unit test steps:
- Add junt related dependencies
- Declare the test class, @SpringbootTest, if you use Mock, you need to declare the initial configuration of Mock
- Declare the test suite, @JunitSuit; you can also directly write the test class @Test
Similarly, JMH also has these steps, but the dependencies are slightly different.
4.1. Maven-related dependencies
<dependencies>
<!-- JMH核心代码 -->
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-core</artifactId>
<version>1.35</version>
</dependency>
<!-- JMH注解相关依赖 -->
<dependency>
<groupId>org.openjdk.jmh</groupId>
<artifactId>jmh-generator-annprocess</artifactId>
<version>1.35</version>
</dependency>
</dependencies>4.2. Write a simple example
/**
* @author Shamee loop
* @date 2023/7/1
*/
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.SECONDS)
public class JMHTestHello01 {
/**
* @Benchmark 类似于Junit，表示被度量代码标注
*/
@Benchmark
public void dealHelloWorld() throws InterruptedException {
// 这里模拟该方法执行
Thread.sleep(1000);
}
public static void main(String[] args) throws RunnerException {
Options options = new OptionsBuilder()
.include(JMHTestHello01.class.getSimpleName())
.warmupIterations(3) // 预热的次数， 3次
.warmupTime(TimeValue.seconds(2)) // 预热的时间，2s
.forks(1) // 测试的执行线程数量
.build();
new Runner(options).run();
}
}
Results of the:
# ...... 这里省略部分信息，这些都是描述JDK和JMH的基础信息，基本信息等同于当下的环境以及option中的配置
# Benchmark: org.shamee.jmh.demo.JMHTestHello01.dealHelloWorld
# Run progress: 0.00% complete, ETA 00:00:56
# Fork: 1 of 1
# ...... 这里开始预热测试，我们指定了预热3次
# Warmup Iteration 1: 1.005 s/op
# Warmup Iteration 2: 1.010 s/op
# Warmup Iteration 3: 1.007 s/op
# ...... 这里迭代测试进行了5次，以及每次的时间
Iteration 1: 1.007 s/op
Iteration 2: 1.011 s/op
Iteration 3: 1.012 s/op
Iteration 4: 1.008 s/op
Iteration 5: 1.007 s/op
Result "org.shamee.jmh.demo.JMHTestHello01.dealHelloWorld":
1.009 ±(99.9%) 0.009 s/op [Average]
(min, avg, max) = (1.007, 1.009, 1.012), stdev = 0.002
CI (99.9%): [1.000, 1.018] (assumes normal distribution)
# Run complete. Total time: 00:00:59
REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
experiments, perform baseline and negative tests that provide experimental control, make sure
the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
Do not assume the numbers tell you what you want them to tell.
# ...... 这里显示的汇总结果，cnt 执行了5次 score最后的结果 Error误差±0.009s units时间单位
Benchmark Mode Cnt Score Error Units
JMHTestHello01.dealHelloWorld avgt 5 1.009 ± 0.009 s/op
Process finished with exit code 0
5. Basic attribute configuration
From the sample code above, it can be found that the use of JMH is not complicated, and the amount of code is not much; many functions are configured by configuring annotations or generating the properties of Options. Therefore, if we want to better use other functions of JMH, we need to understand some of his basic configurations.
5.1、@BenchmarkMode
The schema of the benchmark. There is only one Mode property. And this Mode attribute represents the mode of JMH measurement, or test mode.
/**
* <p>Benchmark mode declares the default modes in which this benchmark
* would run. See {@link Mode} for available benchmark modes.</p>
*
* <p>This annotation may be put at {@link Benchmark} method to have effect
* on that method only, or at the enclosing class instance to have the effect
* over all {@link Benchmark} methods in the class. This annotation may be
* overridden with the runtime options.</p>
*/
@Inherited
@Target({ElementType.METHOD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface BenchmarkMode {
/**
* @return Which benchmark modes to use.
* @see Mode
*/
Mode[] value();
}Mode provides a variety of ways:
- The overall throughput of Throughput indicates how many calls can be executed within 1 second.
Throughput("thrpt", "Throughput, ops/time")- AverageTime The average time of calls, which refers to the time required for calls per second.
AverageTime("avgt", "Average time, time/op")- SampleTime randomly samples, and finally outputs the distribution of the sampling results, for example, "99% of the calls are within xxx milliseconds, and 99.99% of the calls are within xxx milliseconds".
SampleTime("sample", "Sampling time")- SingleShotTime The above modes all default to an Iteration of 1 second, and this means that it only runs once. The number of warmups is often set to 0 to test performance during cold start.
SingleShotTime("ss", "Single shot invocation time")- All Execute all the above modes.
All("all", "All benchmark modes")5.2、@Benchmark
@Benchmark is similar to @Test and is used to tell JMH which methods are covered by the test. It can only be annotated on the method, a bit similar to when the test project is packaged, JMH will generate the Benchmark method code for the method annotated with @Benchmark. Normally, each Benchmark method runs in an independent process without interfering with each other.
5.3、OptionsBuilder & Options
This is the configuration class, which configures the test. Usually you need to specify some parameters, such as the execution test class (include), the number of processes used (fork), the number of warm-up iterations (warmupInterations), etc. Execute when the configuration starts the test, such as the above code:
Options options = new OptionsBuilder()
.include(JMHTestHello01.class.getSimpleName())
.warmupIterations(3) // 预热的次数， 3次
.warmupTime(TimeValue.seconds(2)) // 预热的时间，2s
.forks(1) // 测试的执行线程数量
.build();5.4, Iteration Iteration
An iteration is a unit of measurement for JMH. In most measurement modes, one iteration represents 1 second. During this second, the method under test will be called continuously, and the throughput, average time, etc. will be calculated by sampling.
It can be configured using OptionsBuilder or using annotations.
Options options = new OptionsBuilder()
.include(JMHTestHello01.class.getSimpleName())
.measurementIterations(3).build(); // 执行的次数， 3次or
@Measurement(iterations = 3)
@Benchmark
public void dealHelloWorld() throws InterruptedException {
// 这里模拟该方法执行
Thread.sleep(1000);
}5.5. Warmup
Due to the existence of the JIT of the Java virtual machine, the time of the same method before and after JIT compilation will be different. Usually only the performance of the method after JIT compilation is considered. The warm-up test will not be used as the final statistical result. The purpose of the warm-up is to allow the Java virtual machine to optimize the code under test enough.
Similarly, preheating can also be configured through OptionsBuilder, and annotations can also be used.
Options options = new OptionsBuilder()
.include(JMHTestHello01.class.getSimpleName())
.warmupIterations(3).build(); // 预热的次数， 3次or
@Warmup(iterations = 3)
@Benchmark
public void dealHelloWorld() throws InterruptedException {
// 这里模拟该方法执行
Thread.sleep(1000);
}5.6, State State
The scope of an object can be specified through State, and instantiation and sharing operations are performed through Scope in JMH.
@Inherited
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface State {
/**
* State scope.
* @return state scope
* @see Scope
*/
Scope value();
}- Scope.Benchmark: Benchmark scope. All test threads share one instance to test the performance of stateful instances under multi-thread sharing;
- Scope.Group: The same thread shares instances in the same group
- Scope.Thread: Default State, thread scope. That is, an object will only be accessed by one thread. When testing in a multi-threaded pool, an object is generated for each thread.
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.SECONDS)
@State(Scope.Benchmark)
public class JMHTestHello01 {
}6. IDEA JMH plugin
Similar to the code generation tool for the Junit test class, JMH also has a corresponding test code automatic generation tool plug-in.
Download and install the plugin JMH Java Microbenchmark Harness.
After the installation is complete, right-click the place where you need to generate the test code -> Generate -> Generate JMH Benchmark, it can be automatically generated.
Then you only need to change the property configuration that needs to be tested according to the actual needs, and you can directly run it with the right mouse button to view the results.
7. Summary
In actual projects, by using JMH, developers can accurately measure and analyze the performance of Java code, and perform performance tuning and optimization. It can help developers better understand the performance of code in different environments, identify performance bottlenecks, and find optimization directions and strategies. However, it should be noted that although JMH is powerful, it is necessary to carefully select test scenarios and parameters when using it, and understand the statistical methods and measurement indicators used to ensure the accuracy and reliability of test results.