Table of contents
1.1 Algorithm question: How to calculate the sum of a large array?
1.2 forkjoin-Divide and Conquer Thought Computing
1.2.1 fork/join actual combat code
1.2.2 Fork/join source code analysis
1.2.3 jdk8 parallel stream actual calculation
1. Task type
1. CPU-intensive tasks
CPU-intensive tasks are mainly used to utilize the functions of the CPU. For tasks such as encryption, decryption, and calculation, the recommended number of threads is 1-2 times the number of CPU cores.
2. IO-intensive tasks
IO-intensive tasks are mainly used for reading and writing tasks, such as file reading and writing, database reading and writing, communication and other tasks. Generally, the recommended number of threads is many times the number of CPU cores.
The formula for calculating the number of threads: number of threads = number of CPU cores * (1+ average waiting time/average working time)
The final result of the number of threads needs to be obtained through pressure testing to monitor the running status of the JVM and the load of the CPU.
1.1 Algorithm question: How to calculate the sum of a large array?
1. Single thread addition
public class SumTest {
public static void main(String[] args) {
//定义一个存储1亿个整型数字的数组
int[] sum = new int[100000000];
Random random = new Random();
int temp = 0;
int result = 0;
for(int i=0;i < sum.length;i++){
temp = random.nextInt(100);
result +=temp;
}
System.out.println(result);
}
}2. Multi-thread splitting and combining to get the final value
import java.time.Duration;
import java.time.Instant;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
public class SumThreadTest {
private static int SUM = 10000000;//每个任务计算粒度，任务粒度太小会造成计算过慢
private static int initArrLength = 100000000;//1亿个数组长度
public static void main(String[] args) throws Exception{
int result = 0;//最终结果
//0.定义1亿个长度的数组
int[] arr = new int[initArrLength];
Random random = new Random();
for (int i=0;i < arr.length;i++){
arr[i] = random.nextInt(100);
}
Instant now = Instant.now();
//1.定义一个固定线程池
int numThread = arr.length / SUM >0?arr.length / SUM : 1;
ExecutorService executorService = Executors.newFixedThreadPool(numThread);
//2.定义一个继承callable接口的任务计算类
//切分任务提交到线程池中
Task[] tasks = new Task[numThread];
Future<Long>[] taskFuture = new Future[numThread];
for(int j =0 ;j<numThread;j++){
tasks[j] = new Task(arr, j*SUM,(j+1)*SUM);
taskFuture[j] = executorService.submit(tasks[j]);
}
//3.统计线程池中的线程算出的和得出总和
for(int n =0;n < numThread;n++){
result += taskFuture[n].get();
}
System.out.println("执行时间："+ Duration.between(now,Instant.now()).toMillis());
System.out.println("最终的结果"+result);
//关闭线程池
executorService.shutdown();
}
}
class Task implements Callable<Long>{
private int[] arr;//存储数组
private int beginSum = 0;//初始计算粒度
private int endSum = 0;//初始计算粒度
public Task(int[] arr,int beginSum,int endSum){
this.arr = arr;
this.beginSum = beginSum;
this.endSum = endSum;
}
@Override
public Long call() throws Exception {
Long result = 0L;
//计算数组的值
for(int i=beginSum;i < endSum;i++){
result += arr[i];
}
return result;
}
}1.2 forkjoin-Divide and Conquer Thought Computing
Divide and conquer ideas: 1. Decompose tasks 2. Calculate tasks 3. Summarize tasks
Suitable for computationally intensive tasks
Application scenario:
1. Quick sort, merge sort, binary search in the algorithm
2. Big data field framework MapReduce
3. Java provides the Fork/Join framework, and the implementation class is called ForkJoin. This is the focus
Core parameters of the ForkJoin construction method:
int parallelism the number of parallel threads
ForkJoinWorkerThreadFactory factory worker thread factory class
UncaughtExceptionHandler handler exception handling
boolean asyncMode (two modes, FIFO_QUEUE (first in first out): LIFO_QUEUE (default, last in first out))
ForkJoinTask is one of the cores of ForkjoinPool. It is one of the carriers of tasks. It is an abstract class with two important methods: 1. fork() to submit tasks 2. join() to obtain task execution resultsUsually we don't need to implement ForkJoinTask by ourselves, but only need to use the following three inherited abstract classes according to the scenario.
1. RecursiveAction is used for recursively executing tasks that do not need to return results
2. RecursiveTask is used to recursively execute tasks that need to return results, by implementing
protected abstract V compute()3. CountedCompleter<T> will trigger a custom hook function when the task execution is completed, by implementing
The public void onCompletion(CountedCompleter<?> caller) method can be implemented
ForkJoin is a supplement to the traditional thread pool ThreadPoolExecutor. The traditional thread pool is not suitable for computing-intensive tasks and does not have the ability to steal tasks. This is one of the differences from ordinary thread pools and one of the reasons for its performance guarantee.
There are two objects in Forkjoin: thread object and work queue (double-ended queue). Among them, the task operations: push (push tasks to the queue), pop (take tasks from the queue), poll (steal tasks from the tail of the queue). When the queue in the thread is idle, it will go to the tail of the busy queue to steal tasks, and the queue in the busy thread will fetch tasks from the head. This is to reduce the possibility of competition and speed up the efficiency of parallel computing.
1.2.1 fork/join actual combat code
import java.util.concurrent.ForkJoinPool;
import java.util.concurrent.ForkJoinTask;
import java.util.concurrent.RecursiveTask;
public class LongSum extends RecursiveTask<Long> {
// 任务拆分最小阈值
static final int SEQUENTIAL_THRESHOLD = 10000;
int low;
int high;
int[] array;
public LongSum(int[] arr, int min, int max){
this.array = arr;
this.low = min;
this.high = max;
}
@Override
protected Long compute() {
//当任务拆分到小于等于阀值时开始求和
if (high - low <= SEQUENTIAL_THRESHOLD) {
long sum = 0;
for (int i = low; i < high; ++i) {
sum += array[i];
}
return sum;
} else { // 任务过大继续拆分
int mid = low + (high - low) / 2;
LongSum left = new LongSum(array, low, mid);
LongSum right = new LongSum(array, mid, high);
// 提交任务
left.fork();
right.fork();
//获取任务的执行结果,将阻塞当前线程直到对应的子任务完成运行并返回结果
long rightAns = right.join();
long leftAns = left.join();
return leftAns + rightAns;
//优化：第二种方式
// left.fork();
// //获取任务的执行结果,将阻塞当前线程直到对应的子任务完成运行并返回结果
// long leftAns = left.join();
// return leftAns + right.compute();
}
}
}
class Test2{
public static void main(String[] args) throws Exception{
long sum = 0L;
//100亿个数
int[] array = new int[100000001];
for(int i =0;i<100000000;i++){
array[i] = i;
}
long begin = System.currentTimeMillis();
//递归任务 用于计算数组总和
LongSum ls = new LongSum(array, 0, array.length);
// 构建ForkJoinPool
ForkJoinPool fjp = new ForkJoinPool();
//ForkJoin计算数组总和
ForkJoinTask<Long> submit = fjp.submit(new LongSum(array, 0, array.length));
System.out.println("计算总数"+submit.get());
long end = System.currentTimeMillis();
System.out.println("用时:"+(end-begin)/1000+"秒");
}
}1.2.2 Fork/join source code analysis
focus point:
1. A thread is bound to a queue
volatile WorkQueue[] workQueues; final ForkJoinWorkerThreadFactory factory; DefaultForkJoinWorkerThreadFactory implementation class binding queue2. Submit the task to the work queue
public <T> ForkJoinTask<T> submit(ForkJoinTask<T> task)--> externalPush(task); workQueues = new WorkQueue[n];//Create a work queueI don't understand, it seems that there is no need to read
1.2.3 jdk8 parallel stream actual calculation
Using the jdk8 feature to calculate the array performance is worse than forkjoin, and there may be a livelock phenomenon, and other business threads interfere with the calculation performance.
import java.util.Random;
import java.util.stream.IntStream;
public class jdk8Test {
public static void main(String[] args) {
//定义一个1亿个数字数组
int[] sumArr = new jdk8Test().getSum(100000000);
int sum = IntStream.of(sumArr).parallel().sum();
System.out.println(sum);
}
public int[] getSum(int length){
int[] sum = new int[length];
Random random = new Random();
for(int i=0;i<length;i++){
sum[i] = random.nextInt(100);
}
return sum;
}
}