Calling async methods (Vert.x, Java) from necessarily synchronous ones

clixtec :

We have a set of Java applications that were originally written using normal synchronous methods but have largely been converted to asynchronous Vert.x (the regular API, not Rx) wherever it makes sense. We're having some trouble at the boundaries between sync and async code, especially when we have a method that must be synchronous (reasoning explained below) and we want to invoke an async method from it.

There are many similar questions asked previously on Stack Overflow, but practically all of them are in a C# context and the answers do not appear to apply.

Among other things we are using Geotools and Apache Shiro. Both provide customization through extension using APIs they have defined that are strictly synchronous. As a specific example, our custom authorization realm for Shiro needs to access our user data store, for which we have created an async DAO API. The Shiro method we have to write is called doGetAuthorizationInfo; it is expected to return an AuthorizationInfo. But there does not appear to be a reliable way to access the authorization data from the other side of the async DAO API.

In the specific case that the thread was not created by Vert.x, using a CompletableFuture is a workable solution: the synchronous doGetAuthorizationInfo would push the async work over to a Vert.x thread and then block the current thread in CompletableFuture.get() until the result becomes available.

Unfortunately the Shiro (or Geotools, or whatever) method may be invoked on a Vert.x thread. In that case it is extremely bad to block the current thread: if it's the event loop thread then we're breaking the Golden Rule, while if it's a worker thread (say, via Vertx.executeBlocking) then blocking it will prevent the worker from picking up anything more from its queue - meaning the blocking will be permanent.

Is there a "standard" solution to this problem? It seems to me that it will crop up anytime Vert.x is being used under an extensible synchronous library. Is this just a situation that people avoid?

EDIT

... with a bit more detail. Here is a snippet from org.apache.shiro.realm.AuthorizingRealm:

/**
 * Retrieves the AuthorizationInfo for the given principals from the underlying data store.  When returning
 * an instance from this method, you might want to consider using an instance of
 * {@link org.apache.shiro.authz.SimpleAuthorizationInfo SimpleAuthorizationInfo}, as it is suitable in most cases.
 *
 * @param principals the primary identifying principals of the AuthorizationInfo that should be retrieved.
 * @return the AuthorizationInfo associated with this principals.
 * @see org.apache.shiro.authz.SimpleAuthorizationInfo
 */
protected abstract AuthorizationInfo doGetAuthorizationInfo(PrincipalCollection principals);

Our data access layer has methods like this:

void loadUserAccount(String id, Handler<AsyncResult<UserAccount>> handler);

How can we invoke the latter from the former? If we knew doGetAuthorizationInfo was being invoked in a non-Vert.x thread, then we could do something like this:

@Override
protected AuthorizationInfo doGetAuthorizationInfo(PrincipalCollection principals) {
    CompletableFuture<UserAccount> completable = new CompletableFuture<>();
    vertx.<UserAccount>executeBlocking(vertxFuture -> {
        loadUserAccount((String) principals.getPrimaryPrincipal(), vertxFuture);
    }, res -> {
        if (res.failed()) {
            completable.completeExceptionally(res.cause());
        } else {
            completable.complete(res.result());
        }
    });

    // Block until the Vert.x worker thread provides its result.
    UserAccount ua = completable.get();

    // Build up authorization info from the user account
    return new SimpleAuthorizationInfo(/* etc...*/);
}

But if doGetAuthorizationInfo is called in a Vert.x thread then things are completely different. The trick above will block an event loop thread, so that's a no-go. Or if it's a worker thread then the executeBlocking call will put the loadUserAccount task onto the queue for that same worker (I believe), so the subsequent completable.get() will block permanently.

Matt Timmermans :

I bet you know the answer already, but are wishing it wasn't so -- If a call to GeoTools or Shiro will need to block waiting for a response from something, then you shouldn't be making that call on a Vert.x thread.

You should create an ExecutorService with a thread pool that you should use to execute those calls, arranging for each submitted task to send a Vert.x message when it's done.

You may have some flexibility in the size of the chunks you move into the thread pool. Instead of tightly wrapping those calls, you can move something larger higher up the call stack. You will probably make this decision based on how much code you will have to change. Since making a method asynchronous usually implies changing all the synchronous methods in its call stack anyway (that's the unfortunate fundamental problem with this kind of async model), you will probably want to do it high on the stack.

You will probably end up with an adapter layer that provides Vert.x APIs for a variety of synchronous services.