How can I get 3 thread pool using multiple datasource?

Kirill Sereda :

Help please. I have a class that is responsible for config to the database:

@Getter
@Setter
@ConfigurationProperties(value = "spring")
public class DatabaseConfig {

    @NotEmpty
    private Map<String, Database> databases;

    @Getter
    @Setter
    public static class Database {
    @NotEmpty
    private String url;

    @NotEmpty
    private String username;

    private String password;

    @NotEmpty
    private String driverClassName;
    }
}

There are 3 databases in my application.yml file:

spring:
    H21:
      url: jdbc:h2:tcp://localhost/~/test
      username: sa
      password:
      driver-class-name: org.h2.Driver
    H22:
      url: jdbc:h2:tcp://localhost/~/test2
      username: sa
      password:
      driver-class-name: org.h2.Driver
    H23:
      url: jdbc:h2:tcp://localhost/~/test3
      username: sa
      password:
      driver-class-name: org.h2.Driver

I have a Map, in which when the application is initialized (I use the @PostConstruct method for this), all datasource from the application.yml file are included. Ok. It's fine. But I have a problem.

In the doCheck method method, I create a thread pool = the number of datasource in the Map (3 items) and foreach the Map and call method to the databases.

The line template.query(query, ResultSet::getRow) in performQuery method must be called in parallel for all databases, not sequentially. It's very important for me.

If in the application.yml file 3 databases, it does not work (the application starts and when I send a request through Postman the program waits for a long time and nothing happens). If in the application.yml file 2 databases, then everything works. I read on the Internet that parallelStream() creates only 2 thread pools. Therefore, it works for me when there are only 2 databases in the application.yml file and they are working in parallel. But when there are 3 bases in the application.yml file, it does not work for me.

@Service
@Log
@RequiredArgsConstructor
public class DBServiceImpl implements DBService {

    private final DatabaseConfig databaseConfig;

    private Map<String, JdbcTemplate> templates;

    @PostConstruct
    public void init() {
    templates = databaseConfig.getDatabases()
            .entrySet()
            .stream()
            .collect(Collectors.toMap(Map.Entry::getKey, entry -> createJdbcTemplate(entry.getValue())));
    }

    private JdbcTemplate createJdbcTemplate(DatabaseConfig.Database database) {
    return new JdbcTemplate(DataSourceBuilder
            .create()
            .url(database.getUrl())
            .username(database.getUsername())
            .password(database.getPassword())
            .driverClassName(database.getDriverClassName())
            .build());
    }

    public synchronized PerformanceCheckResult doCheck(String query) throws InterruptedException, ExecutionException {
    PerformanceCheckResult checkResult = PerformanceCheckResult
            .builder()
            .queryExecutionResults(new CopyOnWriteArrayList<>())
            .checkStartedAt(System.currentTimeMillis())
            .build();

    ExecutorService executor = Executors.newFixedThreadPool(templates.size());

    List<QueryExecutionResult> queryExecutionResults = executor.submit(() -> templates
            .entrySet()
            .parallelStream()
            .map(entry -> performQuery(entry.getKey(), entry.getValue(), query))
            .collect(Collectors.toList())).get();

    checkResult.setQueryExecutionResults(queryExecutionResults);
    checkResult.setCheckEndedAt(System.currentTimeMillis());
    checkResult.setTotalCheckDurationTime(getDurationMillis(checkResult.getCheckStartedAt(), checkResult.getCheckEndedAt()));
    return checkResult;
    }

    private QueryExecutionResult performQuery(String databaseName, JdbcTemplate template, String query) {
    QueryExecutionResult executionResult = QueryExecutionResult
            .builder()
            .databaseName(databaseName)
            .queryExecutionStartedAt(System.currentTimeMillis())
            .build();
    template.query(query, ResultSet::getRow);
    executionResult.setQueryExecutionEndedAt(System.currentTimeMillis());
    executionResult.setQueryExecutionDurationTime(
            getDurationMillis(executionResult.getQueryExecutionStartedAt(), executionResult.getQueryExecutionEndedAt()));
    return executionResult;
    }

    private Long getDurationMillis(Long start, Long end) {
    return end - start;
    }

}   

I really need your help. Thanks.

Josh :

You should use invokeAll on the executor rather than submit. This code is just submitting one task to the executor, which then calls parallelStream:

List<QueryExecutionResult> queryExecutionResults = executor.submit(() -> templates
        .entrySet()
        .parallelStream()
        .map(entry -> performQuery(entry.getKey(), entry.getValue(), query))
        .collect(Collectors.toList())).get();

You don't need parallelStream; just create a collection of tasks and use the executor's invokeAll method, which will try to parallelize as much as it can.

Here's a sketch of what I mean:

class Query implements Callable<String> {
    public String call() {
        String s = ""; // do some database query that returns a string
        return s;
    }
}

public static void main(String[] args) {
    var queries = List.of(new Query(...), ...);
    var exec = Executors.newFixedThreadPool(queries.size());
    var futures = exec.invokeAll(queries);

    for (f : futures) {
        f.get();
    }
}