Preface

Who said that programmers can't stress test! Benchmarking based on JMH, show you how I use code to pressure test code (Doll warning ⚠)
Friends who don’t know much about performance testing can take a look: Summary of performance testing knowledge that programmers also need to know

What is a benchmark

Benchmark testing refers to the realization of quantitative and comparable testing of a certain performance index of a class of test objects by designing scientific test methods, test tools and test systems.

Why use benchmarks?

The characteristics of benchmark tests are as follows:

① Repeatability : Repeatable tests can be performed. This is helpful to compare the results of each test and obtain the long-term trend of performance results, which is a reference for system tuning and capacity planning before going online.

② Observability : Through comprehensive monitoring (including test start to end, execution machine, server, database), we can understand and analyze what happened in the test process in time.

③ Displayability : Relevant personnel can understand the test results intuitively and clearly (web interface, dashboard, line chart tree diagram, etc.).

④ Authenticity : The test result reflects the real situation experienced by the customer (true and accurate business scenarios + consistent configuration with production + reasonable and correct test methods).

⑤ Executability : Relevant personnel can quickly test, verify, modify, and tune (locate and analyze). It can be seen that it is very tedious and difficult to do a benchmark test that meets the characteristics. External factors can easily affect the final test results. Especially for JAVA benchmark tests.

When do I need to benchmark

Want to know exactly how long a method needs to be executed, and the correlation between execution time and input;
Compare the throughput of different interfaces/methods under given conditions;
See what percentage of the request took and how long it took to complete;

How to test java code performance?

Test code preparation

Before doing performance testing, we first prepare the test code.

When java8 turned out, everyone was encouraging the use of lambda expressions, talking about how elegant lambda expressions are, and how lambda expressions can improve efficiency. Don't fall behind! Use it now!

There must be some programmers who said at the time:

The so-called verbal talk is unfounded, we next loop through the List to find the maximum value of the code to verify whether the use of lambda expressions is higher than ordinary loop performance.

Test code:

package com.demo.jmh;

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.Optional;

/**
 * @author lps
 * @title: com.demo.jmh.LoopHelper
 * @projectName testdemo
 * @description: 循环工具类 实现基于java8的多种方式循环List
 * 通过 5 种不同的方式遍历List所有的值来查找最大值
 * @date 2020/7/1215:43
 */
public class LoopHelper {
    
    
    /**
     * 使用迭代器循环
     *
     * @param list
     * @return
     */
    public static int iteratorMaxInteger(List<Integer> list) {
    
    
        int max = Integer.MIN_VALUE;
        for (Iterator it = list.iterator(); it.hasNext(); ) {
    
    
            max = Integer.max(max, (Integer) it.next());
        }
        return max;
    }

    /**
     * 使用foreach循环
     *
     * @param list
     * @return
     */
    public static int forEachLoopMaxInteger(List<Integer> list) {
    
    
        int max = Integer.MIN_VALUE;
        for (Integer n : list) {
    
    
            max = Integer.max(max, n);
        }
        return max;
    }

    /**
     * 使用for循环
     *
     * @return
     */
    public static int forMaxInteger(List<Integer> list) {
    
    
        int max = Integer.MIN_VALUE;
        for (int i = 0; i < list.size(); i++) {
    
    
            max = Integer.max(max, list.get(i));
        }
        return max;
    }

    /**
     * 使用java8并行流
     *
     * @return
     */
    public static int parallelStreamMaxInteger(List<Integer> list) {
    
    
        Optional max = list.parallelStream().reduce(Integer::max);
        return (int) max.get();
    }

    /**
     * 使用 lambda 表达式及流
     *
     * @return
     */
    public static int lambdaMaxInteger(List<Integer> list) {
    
    
        return list.stream().reduce(Integer.MIN_VALUE, (a, b) -> Integer.max(a, b));
    }

}

Common means code test performance

Generally speaking, when we evaluate the performance of the comparison code, we can use the time-consuming index to evaluate, which way to achieve the method takes less time is better.

So we generally write:

    public static void main(String[] args) {
    
    
        List<Integer>list = new ArrayList<Integer>();
        for (int i = 0; i < 100; i++) {
    
    
            list.add(i);
        }
        long start=System.currentTimeMillis();
        System.out.println(LoopHelper.iteratorMaxInteger(list));
        long end=System.currentTimeMillis();
        System.out.println("当前程序1耗时："+(end-start)+"ms");
    }

There is nothing wrong with this, but if the error in the middle is in the case of a large amount of data, the test result is not standard.

Unusual benchmark based on JMH

What is JMH?

JMH, the Java Microbenchmark Harness, is a tool suite dedicated to code microbenchmark testing. What is Micro Benchmark? Simply put, it is based on the method-level benchmark test, and the accuracy can reach the microsecond level. When you locate a hot method and want to further optimize the performance of the method, you can use JMH to quantitatively analyze the optimized results. Compared with other competing products-if any, the most distinctive feature of JMH is that it was developed by the people who implement JIT internally in Oracle. It is accurate to benchmark tests for JIT and the so-called "profile guided optimization" of JVM. The influence of sex can be described as knowingly (smile)

Use JMH for testing

Engineering environment description

Dependency/software	version
java	1.8
maven-compiler-plugin	3.8.1
jmh	1.23

Dependency loading

In pom.xmldependence add the following

<dependencies>
        <!-- https://mvnrepository.com/artifact/org.openjdk.jmh/jmh-core -->
        <dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-core</artifactId>
            <version>1.23</version>
        </dependency>

        <dependency>
            <groupId>org.openjdk.jmh</groupId>
            <artifactId>jmh-generator-annprocess</artifactId>
            <version>1.23</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-math3</artifactId>
            <version>3.6.1</version>
        </dependency>
   </dependencies>

Write test class

package com.demo.jmh;


import com.demo.jmh.LoopHelper;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;


/**
 * @author lps
 * @title: LoopHelperTest
 * @projectName testdemo
 * @description: 测试类
 * @date 2020/7/1216:11
 */
@BenchmarkMode(Mode.Throughput) // 吞吐量
@OutputTimeUnit(TimeUnit.MILLISECONDS) // 结果所使用的时间单位
@State(Scope.Thread) // 每个测试线程分配一个实例
@Fork(2) // Fork进行的数目
@Warmup(iterations = 4) // 先预热4轮
@Measurement(iterations = 10) // 进行10轮测试
public class LoopHelperTest {
    
    
    // 定义2个list大小 分别对不同大小list进行测试
    @Param({
    
    "1000",  "100000", })
    private int n;

    private List<Integer> list;

    /**
     * 初始化方法，在全部Benchmark运行之前进行
     */
    @Setup(Level.Trial)
    public void init() {
    
    
        list = new ArrayList<Integer>();
        for (int i = 0; i < n; i++) {
    
    
            list.add(i);
        }
    }
    @Benchmark
    public void testIteratorMaxInteger() {
    
    
        LoopHelper.iteratorMaxInteger(list);
    }

    @Benchmark
    public void testForEachLoopMaxInteger() {
    
    
        LoopHelper.forEachLoopMaxInteger(list);
    }

    @Benchmark
    public void testForMaxInteger() {
    
    
        LoopHelper.forMaxInteger(list);
    }

    @Benchmark
    public void testParallelStreamMaxInteger() {
    
    
        LoopHelper.parallelStreamMaxInteger(list);
    }

    @Benchmark
    public void testLambdaMaxInteger() {
    
    
        LoopHelper.lambdaMaxInteger(list);
    }

    /**
     *  结束方法，在全部Benchmark运行之后进行
     */
    @TearDown(Level.Trial)
    public void arrayRemove() {
    
    
        for (int i = 0; i < n; i++) {
    
    
            list.remove(0);
        }
    }
    public static void main(String[] args) throws RunnerException {
    
    
        Options options = new OptionsBuilder().include(LoopHelperTest.class.getSimpleName()).build();
        new Runner(options).run();
    }
}

Test Results

After running, the console output test results are as follows:

Benchmark: the method of benchmarking execution	(n): parameter n	Mode test mode, here is throughput	cnt: number of runs	Score: Final score	Units: Unit operations per second
LoopHelperTest.testForEachLoopMaxInteger	1000	thrpt	10	1572.310 ±	ops/ms
LoopHelperTest.testForEachLoopMaxInteger	100000	thrpt	10	15.391 ±	ops/ms
LoopHelperTest.testForMaxInteger	1000	thrpt	10	1555.872 ±	ops/ms
LoopHelperTest.testForMaxInteger	100000	thrpt	10	15.926 ±	ops/ms
LoopHelperTest.testIteratorMaxInteger	1000	thrpt	10	1592.788 ±	ops/ms
LoopHelperTest.testIteratorMaxInteger	100000	thrpt	10	15.151 ±	ops/ms
LoopHelperTest.testLambdaMaxInteger	1000	thrpt	10	295.189 ±	ops/ms
LoopHelperTest.testLambdaMaxInteger	100000	thrpt	10	2.057 ±	ops/ms
LoopHelperTest.testParallelStreamMaxInteger	1000	thrpt	10	55.194 ±	ops/ms
LoopHelperTest.testParallelStreamMaxInteger	100000	thrpt	10	3.818 ±	ops/ms

In order to facilitate comparison, the data is presented using a discount chart

1. This is the result when the size of 1000the List is :Iterator > for > foreach > Lambda > ParallelStream

Which Iterator, for, foreachthe performance difference is not big, ParallelStreamthe performance is relatively poor

2. This is the result when the List size is 100000:for > foreach > Iterator >ParallelStream >Lambda

Which Iterator, for, foreachthe performance difference is not big, Lambdathe performance is relatively poor

Based on the results of the benchmark test, we can draw the following conclusions:

With a small amount of data or when a large amount of data, Lamdbaand ParallelStreamthe cycle performance than List Iterator, for, foreachworse, almost but not, at least 3-4-fold difference in throughput, so if a cyclic operation List, recommended Iterator, for, foreach.

jmh phase attention solution explanation

@BenchmarkMode

Benchmark mode, there are four values

Throughput("thrpt", "Throughput, ops/time"), throughput, the number of times each time unit is executed
AverageTime("avgt", "Average time, time/op"), the average time of each operation
SampleTime("sample", "Sampling time"), random sampling will give the worst time that satisfies what percentage of the situation
SingleShotTime("ss", "Single shot invocation time"), SingleShotTime runs only once. The warmup times are often set to 0 at the same time to test the performance during cold start.
All("all", "All benchmark modes"), all of the above are tested, the most commonly used

@Warmup

Warm-up, due to the existence of JIT, the data after warm-up is more stable.

Why do I need to warm up? Because of the existence of the JIT mechanism of the JVM, if a function is called multiple times, the JVM will try to compile it into machine code to improve the execution speed. In order to make the results of Benchmark closer to the real situation, preheating is required.

@Measurement

Some basic parameters of the test

iterations: the number of tests
time: the time of each test
TimeUnit: Time unit
The number of methods called by each op

@Threads

The number of threads to be tested can be annotated on the class or on the method

@Fork

Several new java virtual machines will be fork to reduce the impact. A series of parameters need to be set. If the number of forks is 2, JMH will fork out two processes for testing.

@outputTimeUnit

Time type of benchmark

@Benchmark

Method-level annotations, each method to be tested.

@Param

Attribute-level annotations can be used to specify multiple situations of a certain parameter, and are especially suitable for testing the performance of a function under different parameter inputs. The @Param annotation receives a String array, which is converted to the corresponding data type before the @Setup method is executed. Multiple @Param annotated members are product relationships. For example, if there are two @Param annotated fields, the first has 5 values, and the second field has 2 values, then each test method will run 5* 2=10 times.

@Setup

Method-level annotations, do some preparatory work before testing, such as initialization parameters

@TearDown

Method-level annotations, some finishing work after testing

name	description
Level.Trial	The default level. Before/after all benchmark runs (a set of iterations)
Level.Iteration	Before/after an iteration (a set of calls)
Level.Invocation	Before/after each method call (not recommended unless you know the purpose of doing so)

@State

Set the shared state of a class between threads during testing:

Thread: thread private
Group: shared by all threads in the same group
benchmark: shared by all threads

to sum up

Through the above cases, we can get more accurate and complete performance test results based on the JMH benchmark test framework. According to the results, we can evaluate the performance of the program interface/method, help us solve performance bottlenecks, optimize system performance, reduce downtime, and round off. It is to improve work efficiency and reduce the chance of overtime, and then rounding up is to be promoted and raise salary, and to the peak of career.