Firebase ML-Kit run multiple face detection sessions in parallel

rTECH :

As part of my ML project, I want to generate training data for analyzing the face of multiple individuals from different images using the face detector Google Firebase ML-Kit Face detection library. I created a very simple service class to encapsulate the initialization and start the process of face detection:

class FaceDetectorService(private val act: MainActivity)  {

private var opts: FirebaseVisionFaceDetectorOptions? = null
private var detector: FirebaseVisionFaceDetector? = null


init {
    FirebaseApp.initializeApp(act)

    opts = FirebaseVisionFaceDetectorOptions.Builder()
        .setPerformanceMode(FirebaseVisionFaceDetectorOptions.ACCURATE)
        .setLandmarkMode(FirebaseVisionFaceDetectorOptions.NO_LANDMARKS)
        .setClassificationMode(FirebaseVisionFaceDetectorOptions.NO_CLASSIFICATIONS)
        .setContourMode(FirebaseVisionFaceDetectorOptions.ALL_CONTOURS)
        .build()

    detector = FirebaseVision.getInstance()
        .getVisionFaceDetector(opts!!)
}


suspend fun analyzeAsync(cont: Context, uri: Uri) : Pair<String, Task<List<FirebaseVisionFace>>> {
    val image = FirebaseVisionImage.fromFilePath(cont, uri)

    // this is for the UI thread
    withContext(Main){
        act.addItemToAnalyze(uri.lastPathSegment)
    }

    // return the filename too
    return Pair(uri.lastPathSegment, detector!!.detectInImage(image))
}

}

The function detector!!.detectInImage (FirebaseVisionImage.detectInImage ) returns a Task that represents async operations.

In the onResume() function of my MainActivity, inside a CoroutineScope, I fire up the lib and start iterating over the images converting them to an Uri first then passing it to the face detector:

CoroutineScope(IO).launch {
        val executeTime = measureTimeMillis {
            for (uri in uris){

                    val fileNameUnderAnalysis = uri.lastPathSegment
                    //val tsk = withContext(IO) {
                    //    detector!!.analyzeAsync(act, uri)
                    //}
                val tsk = detector!!.analyzeAsync(act, uri)
                tsk.second.addOnCompleteListener { task ->
                    if (task.isSuccessful && task.result!!.isNotEmpty()) {
                        try {
                           // my best
                        } catch (e: IllegalArgumentException) {
                           // fire
                        }
                    } else if (task.result!!.isEmpty()) {
                       // not today :(
                    }

                }

                tsk.second.addOnFailureListener { e ->
                    // on error
                }
            }

        }
        Log.i("MILLIS", executeTime.toString())
    }

Now, although my implementation runs concurrently (that is, starting at the same time), what I actually want is to run them in parallel (running in the same time depending on the number of threads, which is 4 in my case on an emulator), so my goal would be to take the number of available threads and assign an analysis operation to each of them quartering the execution time.

What I tried so far is, inside the CoroutineScope(IO).launch block, encapsulating the call to the library in a task:

val tsk = async {
    detector!!.analyzeAsync(act, uri)
}

val result = tsk.await()

and a job:

val tsk = withContext(IO) {
   detector!!.analyzeAsync(act, uri)
}

but the async operations I manually start always last only as long as the Firebase tasks are started, not waiting for the inner task to run to completion. I also tried adding different withcontext(...) and ...launch {} variations inside the class FaceDetectorService, but to no avail.

I'm obviously very new to kotlin coroutines, so I think I'm missing something very basic here, but I just cannot wrap my head around it.

(PS: please do not comment on the sloppiness of the code, this is just a prototype :) )

Marko Topolnik :

analyzeAsync() is a suspend fun and also returns a future-like Task object. Instead it should return the result of Task.await(), which you can easily implement basically by factoring out your addOnCompleteListener call:

suspend fun <T> Task<T>.await(): T = suspendCancellableCoroutine { cont ->
    addOnCompleteListener {
        val e = exception
        when {
            e != null -> cont.resumeWithException(e)
            isCanceled -> cont.cancel()
            else -> cont.resume(result)
        }
    }
}

An optimzied version is available in the kotlinx-coroutines-play-services module).

Since the face detection API is already async, it means the thread you call it on is irrelevant and it handles its computation resources internally. Therefore you don't need to launch in the IO dispatcher, you can use Main and freely do GUI work at any point, with no context switches.

As for your main point: I couldn't find explicit details on it, but it is highly likely that a single face detection call already uses all the available CPU or even dedicated ML circuits that are now appearing in smartphones, which mean there's nothing to parallelize from the outside. Just a single face detection request is already getting all the resources working on it.

Firebase ML-Kit run multiple face detection sessions in parallel

Guess you like