Java8 parallel stream: the execution speed is flying fast!
Before Java 7, if we want to process a collection in parallel, we need the following steps
-
Manually divide into parts
-
Create threads for each part
-
Merge in due course
And also need to pay attention to the modification of shared variables between multiple threads. And Java8 provides us with a parallel stream, which can open the parallel mode with one click. Isn’t it cool? Let's see.
Parallel Streams
Know and start parallel streams.
What is a parallel stream: Parallel stream is to divide the content of a stream into multiple data blocks, and use different threads to process each different data block stream. For example, there is such a demand:
There is a List collection, and each apple object in the list only has weight. We also know that the unit price of apple is 5 yuan/kg. Now we need to calculate the unit price of each apple. The traditional way is like this:
List appleList = new ArrayList<>(); // pretend that the data is found from the library
for (Apple apple: appleList) { apple.setPrice(5.0 * apple.getWeight() / 1000); } Copy code We traverse the apple objects in the list through an iterator to complete the calculation of the price of each apple. The time complexity of this algorithm is O(list.size()) As the size of the list increases, the time consumption will increase linearly. Parallel stream
This time can be greatly shortened. The method of processing the collection in parallel streams is as follows:
. appleList.parallelStream () forEach (apple - > apple.setPrice (5.0 * apple.getWeight () / 1000));
copy the code
and is the difference between the normal stream parallelStream () method is called here. Of course, you can also use stream.parallel() to convert an ordinary stream into a parallel stream. Parallel streams can also be converted to sequential streams through the sequential() method.
But pay attention: the parallel and sequential conversion of the stream will not make any actual changes to the stream itself, it is just a mark. And multiple parallel/sequential conversions are performed on the stream on a pipeline, the effective is the last method call
Parallel streaming is so convenient, where do its threads come from? How many are there? How to configure it?
The parallel stream internally uses the default ForkJoinPool thread pool. The default number of threads is the number of processor cores, and the system core attributes are configured:
java.util.concurrent.ForkJoinPool.common.parallelism can change the thread pool size. However, the value is a global variable. Changing it will affect all parallel streams. It is currently not possible to configure a dedicated number of threads for each stream. Generally speaking, the number of processor cores is a good choice
Testing the performance of parallel streams
In order to test the performance more easily, we let the thread sleep for 1 second after calculating the Apple price, which means that other IO-related operations are performed during this period, and the time consumed for program execution and sequential execution is output. Time:
public static void main(String[] args) throws InterruptedException {
List appleList = initAppleList();
Date begin = new Date();
for (Apple apple : appleList) {
apple.setPrice(5.0 * apple.getWeight() / 1000);
Thread.sleep(1000);
}
Date end = new Date();
log.info("苹果数量：{}个, 耗时：{}s", appleList.size(), (end.getTime() - begin.getTime()) /1000);
}
Copy code
Parallel version
List appleList = initAppleList();
Date begin = new Date();
appleList.parallelStream().forEach(apple ->
{ apple.setPrice(5.0 * apple.getWeight() / 1000); try { Thread.sleep(1000); }catch (InterruptedException e) { e.printStackTrace(); } }); Date end = new Date(); log.info("Number of apples: {}, time-consuming: {}s", appleList.size(), (end.getTime( ) - begin.getTime ()) / 1000 ); duplicated code Processed where
Consistent with our prediction, my computer is a quad-core I5 processor. After enabling parallelism, each of the four processors executes a thread, and the task is completed in 1 second!
Can parallel streams be used casually?
Splitting affects the speed of the stream.
Through the above test, some people will easily come to a conclusion: the parallel stream is very fast, we can completely abandon the external iteration of foreach/fori/iter, and use the internal iteration provided by Stream to achieve it.
Is this really the case? Are parallel streams really so perfect? The answer is of course no. You can copy the code below and test it on your computer. After the test, it can be found that parallel streams are not always the fastest processing method.
- For the first n numbers processed by the iterate method, no matter whether it is parallel or not, it is always slower than the loop. The non-parallel version can be understood as the slowness of the streaming operation that is not looped and more inclined to the bottom layer. But why is the parallel version slow? There are two points to note here:
Iterate generates boxed objects, which must be unboxed into numbers to be summed
It is difficult for us to divide iterate into multiple independent blocks for parallel execution
This question is very interesting, we must realize that some stream operations are easier to parallelize than others. For iterate, each application of this function depends on the result of the previous application.
Therefore, in this case, not only can we not effectively divide the stream into small blocks for processing. On the contrary, because of parallelization, the expenses have increased again.
- For the LongStream.rangeClosed() method, the second pain point of iterate does not exist. What it generates is a basic type of value, no unpacking operation is required, and it can directly split the number 1-n to be generated into 1-n/4, 1n/4-2n/4,… 3n/4-n like this Four parts. Therefore, rangeClosed() in the parallel state is faster than iterating outside the for loop:
package lambdasinaction.chap7;
import java.util.stream. *;
public class ParallelStreams {
public static long iterativeSum(long n) {
long result = 0;
for (long i = 0; i <= n; i++) {
result += i;
}
return result;
}
public static long sequentialSum(long n) {
return Stream.iterate(1L, i -> i + 1).limit(n).reduce(Long::sum).get();
}
public static long parallelSum(long n) {
return Stream.iterate(1L, i -> i + 1).limit(n).parallel().reduce(Long::sum).get();
}
public static long rangedSum(long n) {
return LongStream.rangeClosed(1, n).reduce(Long::sum).getAsLong();
}
public static long parallelRangedSum(long n) {
return LongStream.rangeClosed(1, n).parallel().reduce(Long::sum).getAsLong();
}
}
package lambdasinaction.chap7;
import java.util.concurrent. ;
import java.util.function. ;
public class ParallelStreamsHarness {
public static final ForkJoinPool FORK_JOIN_POOL = new ForkJoinPool();
public static void main(String[] args) {
System.out.println("Iterative Sum done in: " + measurePerf(ParallelStreams::iterativeSum, 10_000_000L) + " msecs");
System.out.println("Sequential Sum done in: " + measurePerf(ParallelStreams::sequentialSum, 10_000_000L) + " msecs");
System.out.println("Parallel forkJoinSum done in: " + measurePerf(ParallelStreams::parallelSum, 10_000_000L) + " msecs" );
System.out.println("Range forkJoinSum done in: " + measurePerf(ParallelStreams::rangedSum, 10_000_000L) + " msecs");
System.out.println("Parallel range forkJoinSum done in: " + measurePerf(ParallelStreams::parallelRangedSum, 10_000_000L) + " msecs" );
}
public static <T, R> long measurePerf(Function<T, R> f, T input) {
long fastest = Long.MAX_VALUE;
for (int i = 0; i < 10; i++) {
long start = System.nanoTime();
R result = f.apply(input);
long duration = (System.nanoTime() - start) / 1_000_000;
System.out.println("Result: " + result);
if (duration < fastest) fastest = duration;
}
return fastest;
}
}
Copy code
Shared variable modification problem
Although parallel stream can easily realize multi-threading, it still does not solve the problem of modifying shared variables in multi-threading. There is a shared variable total in the following code, which uses sequential flow and parallel flow to calculate the sum of the first n natural numbers:
public static long sideEffectSum(long n) {
Accumulator accumulator = new Accumulator();
LongStream.rangeClosed(1, n).forEach(accumulator::add);
return accumulator.total;
}
public static long sideEffectParallelSum(long n) {
Accumulator accumulator = new Accumulator();
LongStream.rangeClosed(1, n).parallel().forEach(accumulator::add);
return accumulator.total;
}
public static class Accumulator {
private long total = 0;
public void add(long value) {
total += value;
}
}
Copy code The
output result of each sequential execution is 50000005000000, while the results of parallel execution are varied.
This is because every time you visit totle, there will be data competition. Regarding the reasons for data competition, you can read the blog about volatile. Therefore, it is not recommended to use parallel streams when there are operations to modify shared variables in the code.
Note
on the use of parallel streams The following points need to be noted in the use of parallel streams:
Try to use LongStream / IntStream / DoubleStream and other raw data streams instead of Stream to process numbers to avoid the extra overhead caused by frequent unpacking.
To consider the total computational cost of the operation pipeline of the flow, suppose N is the total number of tasks to be operated, and Q is the time of each operation. N * Q is the total time of the operation. The larger the value of Q, the greater the possibility of using parallel streams to bring benefits.
For example: Several types of resources come from the front end and need to be stored in the database. Each resource corresponds to a different table. We can regard the number of types as N, and the network time for storing the database + the time for insert operation as Q.
Under normal circumstances, the network time-consuming is relatively large. Therefore, this operation is more suitable for parallel processing. Of course, when the number of types is greater than the number of cores, the performance improvement of this operation will be discounted. Better optimization methods will be provided in future blogs.
For small amounts of data, it is not recommended to use parallel streams
Stream data that is easy to split into blocks, it is recommended to use parallel streams
The following is the splittable performance table of some common collection frameworks corresponding to the stream
The following is the splittable performance table of some common collection framework corresponding streams:
Codewords are not easy, if you think you will gain something after reading it, you might as well click it and let more people see it~
Summary: The
editor summarizes the 2020 interview questions. The modules included in this interview question are divided into 19 modules, namely: Java Basics, Containers, Multithreading, Reflection, Object Copy, Java Web, Exceptions, Networks, Design Patterns, Spring/Spring MVC, Spring Boot/Spring Cloud, Hibernate, MyBatis, RabbitMQ, Kafka, Zookeeper, MySQL, Redis, JVM.