THE CONFIGURATION
Web server : Nginx
App server : Tomcat with default configuration of 200 request serving threads
Expected response time for my server : ~30 seconds(There are lots of third party dependencies)
THE SCENARIO
Every 10 seconds the application will need to generate the token for its use. The expected time for token generation is around 5 seconds, but since its a third party system being contacted over network, this is obviously not consistent and can go up to 10 seconds.
During the token generation process, nearly 80% of the incoming requests per second will need to wait.
WHAT I BELIEVE SHOULD HAPPEN
Since the requests waiting for the token generation will have to wait a "long" time, there is no reason for these request serving to be reused to serve other incoming requests while waiting for token generation process to complete.
Basically, it would make sense if my 20% to keep being served. If the waiting threads are not being utilized for other requests, tomcat request serving limit will be reached and server would essentially choke, not really something any developer will like.
WHAT I TRIED
Initially I expected switching to tomcat NIO connector would do this job. But after looking at this comparison, I was really not hopeful. Nevertheless, I tried by forcing the requests to wait for 10 second and it did not work.
Now I am thinking on the lines that I need to, sort of, shelve the request while its waiting and need to signal the tomcat that this thread is free to reuse. Similarly, I will need tomcat to give me a thread from its threadpool when the request is ready to be moved forward. But I am blindsided on how to do it or even if this is possible.
Any guidance or help?
You need an asynchronous servlet but you also need asynchronous HTTP calls to the external token generator. You will gain nothing by passing the requests from the servlet to an ExecutorService with a thread pool if you still create one thread somewhere per token request. You have to decouple threads from HTTP requests so that one thread can handle multiple HTTP requests. This can be achieved with an asynchronous HTTP client like Apache Asynch HttpClient or Async Http Client.
First you have to create an asynchronous servlet like this one
public class ProxyService extends HttpServlet {
private CloseableHttpAsyncClient httpClient;
@Override
public void init() throws ServletException {
httpClient = HttpAsyncClients.custom().
setMaxConnTotal(Integer.parseInt(getInitParameter("maxtotalconnections"))).
setMaxConnPerRoute(Integer.parseInt(getInitParameter("maxconnectionsperroute"))).
build();
httpClient.start();
}
@Override
public void destroy() {
try {
httpClient.close();
} catch (IOException e) { }
}
@Override
public void doGet(HttpServletRequest request, HttpServletResponse response) {
AsyncContext asyncCtx = request.startAsync(request, response);
asyncCtx.setTimeout(ExternalServiceMock.TIMEOUT_SECONDS * ExternalServiceMock.K);
ResponseListener listener = new ResponseListener();
asyncCtx.addListener(listener);
Future<String> result = httpClient.execute(HttpAsyncMethods.createGet(getInitParameter("serviceurl")), new ResponseConsumer(asyncCtx), null);
}
}
This servlet performs an asynchronous HTTP call using Apache Asynch HttpClient. Note that you may want to configure the maximum connections per route because as per RFC 2616 spec HttpAsyncClient will only allow a maximum of two concurrent connections to the same host by default. And there are plenty of other options that you can configure as shown in HttpAsyncClient configuration. HttpAsyncClient is expensive to create, therefore you do not want to create an instace of it on each GET operation.
One listener is hooked to the AsyncContext, this listener is only used in the example above to handle timeouts.
public class ResponseListener implements AsyncListener {
@Override
public void onStartAsync(AsyncEvent event) throws IOException {
}
@Override
public void onComplete(AsyncEvent event) throws IOException {
}
@Override
public void onError(AsyncEvent event) throws IOException {
event.getAsyncContext().getResponse().getWriter().print("error:");
}
@Override
public void onTimeout(AsyncEvent event) throws IOException {
event.getAsyncContext().getResponse().getWriter().print("timeout:");
}
}
Then you need a consumer for the HTTP client. This consumer informs the AsyncContext by calling complete() when buildResult() is executed internally by HttpClient as a step to return a Future<String> to the caller ProxyService servlet.
public class ResponseConsumer extends AsyncCharConsumer<String> {
private int responseCode;
private StringBuilder responseBuffer;
private AsyncContext asyncCtx;
public ResponseConsumer(AsyncContext asyncCtx) {
this.responseBuffer = new StringBuilder();
this.asyncCtx = asyncCtx;
}
@Override
protected void releaseResources() { }
@Override
protected String buildResult(final HttpContext context) {
try {
PrintWriter responseWriter = asyncCtx.getResponse().getWriter();
switch (responseCode) {
case javax.servlet.http.HttpServletResponse.SC_OK:
responseWriter.print("success:" + responseBuffer.toString());
break;
default:
responseWriter.print("error:" + responseBuffer.toString());
}
} catch (IOException e) { }
asyncCtx.complete();
return responseBuffer.toString();
}
@Override
protected void onCharReceived(CharBuffer buffer, IOControl ioc) throws IOException {
while (buffer.hasRemaining())
responseBuffer.append(buffer.get());
}
@Override
protected void onResponseReceived(HttpResponse response) throws HttpException, IOException {
responseCode = response.getStatusLine().getStatusCode();
}
}
The web.xml configuration for ProxyService servlet may be like
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
id="WebApp_ID" version="3.0" metadata-complete="true">
<display-name>asyncservlet-demo</display-name>
<servlet>
<servlet-name>External Service Mock</servlet-name>
<servlet-class>ExternalServiceMock</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet>
<servlet-name>Proxy Service</servlet-name>
<servlet-class>ProxyService</servlet-class>
<load-on-startup>1</load-on-startup>
<async-supported>true</async-supported>
<init-param>
<param-name>maxtotalconnections</param-name>
<param-value>200</param-value>
</init-param>
<init-param>
<param-name>maxconnectionsperroute</param-name>
<param-value>4</param-value>
</init-param>
<init-param>
<param-name>serviceurl</param-name>
<param-value>http://127.0.0.1:8080/asyncservlet/externalservicemock</param-value>
</init-param>
</servlet>
<servlet-mapping>
<servlet-name>External Service Mock</servlet-name>
<url-pattern>/externalservicemock</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>Proxy Service</servlet-name>
<url-pattern>/proxyservice</url-pattern>
</servlet-mapping>
</web-app>
And a mock servlet for the token generator with a delay in seconds may be:
public class ExternalServiceMock extends HttpServlet{
public static final int TIMEOUT_SECONDS = 13;
public static final long K = 1000l;
public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
Random rnd = new Random();
try {
Thread.sleep(rnd.nextInt(TIMEOUT_SECONDS) * K);
} catch (InterruptedException e) { }
final byte[] token = String.format("%10d", Math.abs(rnd.nextLong())).getBytes(ISO_8859_1);
response.setContentType("text/plain");
response.setCharacterEncoding(ISO_8859_1.name());
response.setContentLength(token.length);
response.getOutputStream().write(token);
}
}
You can get a fully working example at GitHub.