how to do benchmarking in java

Recently, the company is working on a new project. Because it is experimental and will not directly face the customer's project, the technology selection this time is very radical. For example, Java 17 is directly used.

As a personal trainee who has practiced in the company for two and a half years, I naturally deeply participated in the work of technology selection. I wonder if you have paid attention to the benchmark test given by the technical components in the technical selection? For example, HikariCP's benchmark :

Or the Caffeine benchmark :

If you have read their benchmark reports carefully, you will find a very interesting technology: Java Microbenchmark Harness, referred to as JMH .

Tips : Some technologies only need to learn how to use, there is no need to "roll" the source code; you have not heard of some "niche" technologies, so don't panic, no one knows everything.

Meet JMH

Before contacting JMH, I usually use System.currentTimeMillis() to calculate the execution time of the method:

long start = System.currentTimeMillis();
......
long duration = System.currentTimeMillis() - start;

Most of the time, this works well, but in some scenarios, the JVM will perform JIT compilation and inline optimization, resulting in a very large difference in the execution efficiency of the code before and after optimization. At this time, this "earth" method will not work. So how to accurately calculate the execution time of the method?

The Java team provides the JMH benchmark suite for developers:

JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.

JMH is a benchmark suite for building, running and analyzing programs written in Java and other JVM-based languages . JMH provides the ability to warm up, let the JVM know which hot codes are hot through warming up, in addition, JMH also provides throughput test indicators. Compared with the "earth" method, JMH can support more kinds of test scenarios , and the test results based on JMH will be more comprehensive and accurate .

Use JMH

Introduce the dependency of JMH in the project:

<dependency>
  <groupId>org.openjdk.jmh</groupId>
  <artifactId>jmh-core</artifactId>
  <version>1.36</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.openjdk.jmh</groupId>
  <artifactId>jmh-generator-annprocess</artifactId>
  <version>1.36</version>
</dependency>

After introducing dependencies, you can write a simple benchmark test. Here is a simplified official example of JMH :

package org.openjdk.jmh.samples;

import org.openjdk.jmh.annotations.Benchmark;  
import org.openjdk.jmh.annotations.BenchmarkMode;  
import org.openjdk.jmh.annotations.Mode;  
import org.openjdk.jmh.annotations.OutputTimeUnit;  
import org.openjdk.jmh.runner.Runner;  
import org.openjdk.jmh.runner.RunnerException;  
import org.openjdk.jmh.runner.options.Options;  
import org.openjdk.jmh.runner.options.OptionsBuilder;  

import java.util.concurrent.TimeUnit;

public class JMHSample_02_BenchmarkModes {

	@Benchmark
	@BenchmarkMode(Mode.AverageTime)
	@OutputTimeUnit(TimeUnit.MILLISECONDS)
	public void measureAvgTime() throws InterruptedException {
		TimeUnit.MILLISECONDS.sleep(100);
	}

	public static void main(String[] args) throws RunnerException {
		Options opt = new OptionsBuilder()
		.include(JMHSample_02_BenchmarkModes.class.getSimpleName())
		.forks(1)
		.build();
		new Runner(opt).run();
	}
}

Executing this example will output the following results:

If divided by empty lines, the output of JMH can be divided into 3 parts:

• Basic information , including environment information and benchmark configuration;
• Test information , each warmup (Warmup) and official execution (Iteration) information;
• Result information , the result of the benchmark test.

Tips：

• IDEA cannot run in Debug mode, otherwise an error will be reported ;
• Note that the scope tag in the dependencies is test, and JMH cannot be accessed under the src\main\java path.

start test

It is not difficult to find from the examples that to execute tests in IDEA, you need to build Options first and execute them through Runner. Let's build the simplest Options:

Options opt = new OptionsBuilder().build();

new Runner(opt).run();

Such Options will execute benchmark methods (methods annotated with Benchmark) scattered throughout the program. If you don't need to execute all benchmark methods, you usually specify the scope of the test when you build Options:

Options opt = new OptionsBuilder().include(JMHSample_02_BenchmarkModes.class.getSimpleName()).build();

At this point benchmarking is limited to the benchmark method in the Test class. In addition, you may also think that the console output style is ugly, or the benchmark test report to be submitted needs to be expressed intuitively with graphics. At this time, you can control the format of the output result and specify the result output file:

Options opt = new OptionsBuilder()
.include(JMHSample_02_BenchmarkModes.class.getSimpleName())
.result("result.json")
.resultFormat(ResultFormatType.JSON)
.build();

Combined with the following websites, you can easily build a test result diagram:

• JMH Visual Chart (deepoove.com)
• JMH Visualizer (morethan.io)

For example, the test results I built through JMH Visual Chart:

In fact, the functions provided by OptionsBuilder are far more than that, but most of them can be configured through the annotations mentioned below, so no redundant explanation is given here.

Common Notes

JMH can easily complete the configuration of the benchmark test through annotations. Next, the 15 commonly used annotations will be described in detail.

Note: Benchmark

Annotate Benchmark's statement:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Benchmark {
}

Benchmark is used on a method and the method must be modified with public, indicating that the method is a benchmark method .

Note: BenchmarkMode

Annotate the declaration of BenchmarkMode:

@Inherited  
@Target({ElementType.METHOD, ElementType.TYPE})  
@Retention(RetentionPolicy.RUNTIME)  
public @interface BenchmarkMode {
	Mode[] value();  
}

BenchmarkMode is used on methods or classes to indicate test indicators . The enumeration class Mode provides 4 test indicators:

• Mode.Throughput, throughput , the number of executions per unit time;
• Mode.AverageTime, the average time , the average time spent executing the method;
• Mode.SampleTime, operation time sampling , and output result distribution;
• Mode.SingleShotTime, single operation time , usually the time to test cold start without preheating.

Let's take a look at the output of Mode.SampleTime:

In addition to using the above test indicators alone, Mode.All can also be specified for benchmarking of all indicators.

Note: OutputTimeUnit

Annotate the declaration of OutputTimeUnit:

@Inherited
@Target({ElementType.METHOD,ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface OutputTimeUnit { 
	TimeUnit value();
}

OutputTimeUnit is used on a method or class to indicate the time unit of the output result . Well, we have finished understanding the annotations in the example, and then we will look at other more critical annotations.

Note: Timeout

Annotate the declaration of Timeout:

@Target({ElementType.METHOD,ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Inherited
public @interface Timeout {

	int time();

	TimeUnit timeUnit() default TimeUnit.SECONDS;
}

Timeout is used on methods or classes to specify the timeout period of the benchmark method .

Note: Warmup

Annotate the declaration of Warmup:

@Target({ElementType.METHOD,ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Inherited
public @interface Warmup {
	int BLANK_ITERATIONS = -1;
	int BLANK_TIME = -1;
	int BLANK_BATCHSIZE = -1;

	int iterations() default BLANK_ITERATIONS;

	int time() default BLANK_TIME;

	TimeUnit timeUnit() default TimeUnit.SECONDS;

	int batchSize() default BLANK_BATCHSIZE;
}

Warmup is used on methods or classes for warm-up configuration . 4 parameters are provided:

• iterations, the number of warm-up iterations;
• time, the time of each warm-up iteration;
• timeUnit, time unit;
• batchSize, the number of times each operation is called.

The warm-up execution results will not be counted in the test results , because of the existence of the JIT mechanism. After some methods are called repeatedly, the JVM will compile them into machine codes, which greatly improves the execution efficiency.

Note: Measurement

Annotate the statement of Measurement:

@Inherited
@Target({ElementType.METHOD,ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface Measurement {
	int BLANK_ITERATIONS = -1;
	int BLANK_TIME = -1;
	int BLANK_BATCHSIZE = -1;

	int iterations() default BLANK_ITERATIONS;

	int time() default BLANK_TIME;

	TimeUnit timeUnit() default TimeUnit.SECONDS;

	int batchSize() default BLANK_BATCHSIZE;
}

Measurement and Warmup are used in exactly the same way, and the parameter meanings are also exactly the same. The difference is that Measurement is a formal test configuration, and the results will be counted .

Note: Group

Annotate the declaration of Group:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Group {
	String value() default "group";
}

Group is used on methods to group test methods .

Note: State

Annotate the declaration of State:

@Inherited
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface State {
	Scope value();
}

State is used on classes to indicate the scope of variables in the class . The enumeration class Scope provides 3 scopes:

• Scope.Benchmark, using a variable in each test method;
• Scope.Group, use the same variable in each group;
• Scope.Thread, use the same variable in each thread.

I forgot where I saw someone say that the scope of Scope.Benchmark is all benchmark methods. This is wrong. Scope.Benchmark will generate an object for each benchmark method, for example:

@State(Scope.Benchmark)
public static class ThreadState {
}

@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
public void test1(State state) {
System.out.println("test1执行" + VM.current().addressOf(state));
}

@Benchmark
@BenchmarkMode(Mode.SingleShotTime)
public void test2(State state) {
System.out.println("test2执行" + VM.current().addressOf(state));
}

In this example, test1 and test2 use different State objects.

Tips : VM.current().addressOf() is a function provided in jol-core.

Note: Setup

Annotate the declaration of Setup:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface Setup {
	Level value() default Level.Trial;
}

Setup is used in the method, the initialization operation before the benchmark test . The enumeration class Level provides 3 levels:

• Level.Trial, when all benchmarks are executed;
• Level.Iteration, each iteration;
• Level.Invocation, each time the method is called.

Tips : In one iteration, there may be multiple method calls.

Note: TearDown

Annotate the declaration of TearDown:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface TearDown {
	Level value() default Level.Trial;
}

TearDown is used in the method. It is the opposite of the function of Setup. It is the operation after the benchmark test . Level is also used to provide 3 levels.

Note: Param

Annotate the declaration of Param:

@Inherited
@Target({ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
public @interface Param {

	String BLANK_ARGS = "blank_blank_blank_2014";

	String[] value() default { BLANK_ARGS };
}

Param is used on fields to specify different parameters and needs to be used with State annotations . for example:

@State(Scope.Benchmark)
public class Test {
	@Param({"10", "100", "1000", "10000"})
	int count;

	@Benchmark
	@Warmup(iterations = 0)
	@BenchmarkMode(Mode.SingleShotTime)
	public void loop() throws InterruptedException {
		for(int i = 0; i < count; i++) {
			TimeUnit.MILLISECONDS.sleep(1);
		}
	}
}

The above code tests the performance of the program when looping 10 times, 100 times, 1000 times and 10000 times.

Note: Threads

Annotate the declaration of Threads:

@Inherited
@Target({ElementType.METHOD,ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface Threads {

	int MAX = -1;

	int value();
}

Threads are used on methods and classes to specify the number of parallel threads in the benchmark . When using MAX, all available threads will be used for testing, which is Runtime.getRuntime().availableProcessors()the number of threads returned.

Note: GroupThreads

Annotate the declaration of GroupThreads:

@Inherited
@Target({ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
public @interface GroupThreads {
	int value() default 1;
}

GroupThreads is used on the method to specify the number of threads used in the benchmark grouping .

Note: Fork

Annotate the statement of Fork:

@Inherited  
@Target({ElementType.METHOD,ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface Fork {
	int BLANK_FORKS = -1;

	String BLANK_ARGS = "blank_blank_blank_2014";

	int value() default BLANK_FORKS;

	int warmups() default BLANK_FORKS;

	String jvm() default BLANK_ARGS;

	String[] jvmArgs() default { BLANK_ARGS };

	String[] jvmArgsPrepend() default { BLANK_ARGS };

	String[] jvmArgsAppend() default { BLANK_ARGS };
}

Fork is used on methods and classes to specify the child process of Fork in the benchmark test . Fork provides 6 parameters:

• value, indicating the number of child processes produced by Fork;
• warmups, warm-up times;
• jvm, the location of the JVM;
• jvmArgs, the JVM parameters that need to be replaced ;
• jvmArgsPrepend, JVM parameters that need to be added ;
• jvmArgsAppend, JVM parameters that need to be appended .

When Fork is set to 0, JMH runs the benchmark in the current JVM. Since it may be in the user's JVM, it cannot reflect the real server scene and cannot accurately reflect the actual performance, so JMH recommends setting Fork .

In addition, you can use the JVM settings provided by Fork to set the JVM to Server mode:

@Fork(value = 1, jvmArgsAppend = {"-Xmx1024m", "-server"})

Note: CompilerControl

Annotate the declaration of CompilerControl:

@Target({ElementType.METHOD, ElementType.CONSTRUCTOR, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
public @interface CompilerControl {

	Mode value();

	enum Mode {
		BREAK("break"),
		PRINT("print"),
		EXCLUDE("exclude"),
		INLINE("inline"),
		DONT_INLINE("dontinline"),
		COMPILE_ONLY("compileonly");
	}
}

CompilerControl is used on methods, constructors or classes to specify the compilation method . Its internal enumeration class provides 6 compilation methods:

• BREAK, insert a breakpoint into the compiled code;
• PRINT, the printing method and its configuration;
• EXCLUDE, prohibit compilation;
• INLINE, use inline;
• DONT_INLINE, prohibit inline;
• COMPILE_ONLY, compile only.

epilogue

We have talked about the use of JMH here, and hope that today's content can help you learn and master a more accurate performance testing method.

Finally, I provide an idea to practice using JMH: Everyone has seen the benchmark test results given by Caffeine at the beginning of the article, but because it is the benchmark test provided by the author of Caffeine, it is inevitable that there are some suspicions of "being both a referee and a player", or He chose some angles that are beneficial to Caffeine to show the results, so you can combine your own actual usage scenarios to do a benchmark test for Caffeine and its competitors.

---

If this article is helpful to you, please give it a lot of praise and support. If there are any mistakes in the article, please criticize and correct. Finally, everyone is welcome to pay attention to Wang Youzhi, a financial man . See you next time!