How to read multiple files using thread pool?

huangjs :

I want to read multiple files using a thread pool, but I failed.

@Test
public void test2() throws IOException {
    String dir = "/tmp/acc2tid2999928854413665054";
    int[] shardIds = new int[]{1, 2};
    ExecutorService executorService = Executors.newFixedThreadPool(2);
    for (int id : shardIds) {
        executorService.submit(() -> {
            try {
                System.out.println(Files.readAllLines(Paths.get(dir, String.valueOf(id)), Charset.forName("UTF-8")));
            } catch (IOException e) {
                e.printStackTrace();
            }
        });
    }
}

Above is a simple example I wrote. It cannot reach my purpose.

System.out.println(Files.readAllLines(
        Paths.get(dir, String.valueOf(id)), Charset.forName("UTF-8")));

This line will not run and there were no warnings. I don't know why?

tantalum :

You are submitting tasks to be executed then ending the test before waiting for the tasks to complete. ExecutorService::submit will submit the task to be executed in the future and return immediately. Therefore, your for-loop submits the two tasks then ends, and the test function returns before the tasks had the time to complete.

You might try calling ExecutorService::shutdown after the for-loop to let the executor know that all the tasks have been submitted. Then use ExecutorService::awaitTermination to block until the tasks are complete.

For example:

    @Test
    public void test2() throws IOException {
        String dir = "/tmp/acc2tid2999928854413665054";
        int[] shardIds = new int[]{1, 2};
        ExecutorService executorService = Executors.newFixedThreadPool(2);
        for (int id : shardIds) {
            executorService.submit(
                    () -> {
                        try {
                            System.out.println(Files.readAllLines(Paths.get(dir, String.valueOf(id)), Charset.forName("UTF-8")));
                        } catch (IOException e) {
                            e.printStackTrace();
                        }
                    });
        }
        executorService.shutdown();
        executorService.awaitTermination(60, TimeUnit.SECONDS); //Wait up to 1 minute for the tasks to complete
    }