Actor not processing messages for quite some time

whysoseriousson :

In my akka application , i am using a main actor as controller which receives commands and delegates it to processor actor .Processor actor upon finishing (which takes around 2 mins to finish each task) , passes the message to controller and then controller actor , sends message to Database actor for persistence. Both processor actor and db database actor are managed using router each with say 5 routees. I am using default dispatcher and all other akka configuration are default only .Now below is the scenario.

The controller actor is receiving around 100 messages which are passed to processor actor , i can see from the log that the processor has finished processing some messages (around 5) and passed on completion message to controller actor. But database actor starts executing after some 5 mins. During this 5 mins , however the processor actor is processing its pending messages. So its not like the application is idle.

When the volume of message is less , the flow from controller -> processor -> controller -> db actor is almost instantaneously and there is hardly any lag.

I dont want this lag after processing , DB execution should happen as soon as processing is finished. But seems threads are busy executing processor task.How can i overcome this situation , ideally i want the turn around time of my task execution to be less , but due to above behavior i am not able to achieve it.

Anand Sai :

By default, all Akka actors use the same executor which is limited to use 64 threads maximum. From https://doc.akka.io/docs/akka/current/general/configuration-reference.html :

# This will be used if you have set "executor = "default-executor"".
      # If an ActorSystem is created with a given ExecutionContext, this
      # ExecutionContext will be used as the default executor for all
      # dispatchers in the ActorSystem configured with
      # executor = "default-executor". Note that "default-executor"
      # is the default value for executor, and therefore used if not
      # specified otherwise. If no ExecutionContext is given,
      # the executor configured in "fallback" will be used.
      default-executor {
        fallback = "fork-join-executor"
      }

and fork-join-executor config:

# This will be used if you have set "executor = "fork-join-executor""
      # Underlying thread pool implementation is java.util.concurrent.ForkJoinPool
      fork-join-executor {
        # Min number of threads to cap factor-based parallelism number to
        parallelism-min = 8

        # The parallelism factor is used to determine thread pool size using the
        # following formula: ceil(available processors * factor). Resulting size
        # is then bounded by the parallelism-min and parallelism-max values.
        parallelism-factor = 1.0

        # Max number of threads to cap factor-based parallelism number to
        parallelism-max = 64

        # Setting to "FIFO" to use queue like peeking mode which "poll" or "LIFO" to use stack
        # like peeking mode which "pop".
        task-peeking-mode = "FIFO"
      }

The problem could be related to blocking calls in processor actors. Akka assigns separate threads from a pool of 64 to handle these blocking calls in processor actors and waits for one of them to complete message processing to be able to handle messages for other actors. This might cause the time lag that you observe between actors.

A key aspect on which Akka is based is that systems should remain responsive at all times. If you used the same dispatcher/thread pool for the blocking DB operations or processing messages as your main Akka routing infrastructure, it is possible that all the Akka threads could be occupied by the processing actors or DB operations, and your system would effectively be deadlocked until one of the blocking operations is completed. This might not be a problem for a simple system on a single JVM that only performs this task, but when it is scaled, it might cause a lot of problems.

In cases like yours where you cannot avoid blocking, a dedicated dispatcher for blocking operations should be used. This link talks about this aspect(though it is titled for Akka-Http, it can be generalized). You can create two types of dispatchers to handle the two different blocking operations. I also, think that you should throttle your blocking requests as to not overwhelm your system(use dispatchers for throttling). You could also implement buffers within your actors to deal with backpressure situations.

EDIT

The controller mailbox has 100 messages and 5 messages are taken and delegated to processor actors. Each processor actor takes 2 mins of time and sends the response back to the controller and the response gets queued in the controller's mailbox. But before processing these messages, the controller needs to process messages which were added before these messages effectively increasing the service time for processing the messages for controller. The lag is the culmination of this whole process. As soon as the controller got the response message, it delegated to the actor.I think that the processing time increases as the messages increase.

Let me know if it helps!!