Application of Facial Expression Recognition Based on Convolutional Neural Network--AR Face Changing in Sichuan Opera (2)

This project will implement an AR software on Android that can put different styles of makeup on people's faces by recognizing expression categories. The effect is as follows:

Demonstration of the effect of face-changing AR Android APP based on facial expression recognition

To realize such a software, the core consists of two parts:
1) Use convolutional neural network to train a facial expression recognition model,
2) Transplant the trained model to the Android platform, and at the same time realize the Facebook AR effect on Android, combined with facial expressions Recognize the recognition results of the model and render different Facebook styles

This article talks about the second part, how to port the convolutional neural network model to Android, and perform Facebook AR effects on Android.

See the first part: Convolutional Neural Network-Based Facial Expression Recognition Application – AR Sichuan Opera Face Changing (1)

---

---

In the first part, we trained a convolutional neural network model that can recognize seven facial expressions: angry, disgust, fear, happy, sad, surprised, and normal, and the analysis found that the effect is good.

This article will build on the basis of expression recognition, design an AR face-changing effect software, obtain face images through the front camera, use the trained model for facial expression recognition, and then realize the rendering of different facial makeup according to different expressions .

The screenshot of the software effect is as follows. First, there is a button in the upper right corner. When FPS is turned on, the current frame number is displayed. When Emotion is turned on, the current facial expression recognition result is displayed. When Mask is turned on, the facial makeup effect is rendered.

1. Model conversion

Our model is trained using Keras, but such a model cannot be used directly on the Android mobile terminal, so it needs to be converted to be available on Android.

Fortunately, TensorFlow Lite (tflite) of TensorFlow can be used on Android. If you can convert the model to tflite, you can port the model to Android.

Because Keras is included in TensorFlow, the conversion is also very convenient:

from keras.models import Model,load_model
import tensorflow as tf

# keras模型的文件按路径
model=load_model(r"keras.hdf5",compile=False)   # keras.hdf5是Keras训练好的模型

converter =tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

open("output.tflite", 'wb').write(tflite_model)  # 转换成功后会得到output.tflite文件，这个就是转换后的模型，可以在Android中使用

2. Android deployment

Now that we have the model file, we need an inference framework for Android deployment:tensorflow.lite

For model deployment in Android, expression recognition is written as a class: TFLiteClassificationUtil, there are two main methods in this class, preProcess is used for preprocessing: when we get the face image from the front camera, we need to perform image scaling and type conversion , dimension transformation, grayscale transformation and other processing, so that the input meets the requirements of the model, and predict is used for prediction: that is, to receive an image data, use our tflite model to predict the result and return it.

package com.google.ar.core.examples.java.augmentedfaces;

import android.graphics.Bitmap;

import org.tensorflow.lite.DataType;
import org.tensorflow.lite.Interpreter;
import org.tensorflow.lite.nnapi.NnApiDelegate;
import org.tensorflow.lite.support.tensorbuffer.TensorBuffer;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;

import java.io.File;

public class TFLiteClassificationUtil {
    
    
    private Interpreter tflite;
    private final TensorBuffer outputProbabilityBuffer;
    private static final int NUM_THREADS = 4;
    int[] imageShape;   //用以储存model的input_shape    [1,n,n,channel]


    /**
     * @param modelPath model path
     */
    public TFLiteClassificationUtil(String modelPath) throws Exception {
    
    

        File file = new File(modelPath);
        if (!file.exists()) {
    
    
            throw new Exception("model file is not exists!");
        }

        try {
    
    
            Interpreter.Options options = new Interpreter.Options();
            // 使用多线程预测
            options.setNumThreads(NUM_THREADS);
            // 使用Android自带的API或者GPU加速
            NnApiDelegate delegate = new NnApiDelegate();
//            GpuDelegate delegate = new GpuDelegate();
            options.addDelegate(delegate);
            tflite = new Interpreter(file, options);
            // 获取输入，shape为{1, height, width, 3}
            String input="conv2d_input";
            String output="Identity";
            imageShape = tflite.getInputTensor(tflite.getInputIndex(input)).shape();
            DataType imageDataType = tflite.getInputTensor(tflite.getInputIndex(input)).dataType();
            // 获取输入，shape为{1, NUM_CLASSES}
            int[] probabilityShape = tflite.getOutputTensor(tflite.getOutputIndex(output)).shape();
            DataType probabilityDataType = tflite.getOutputTensor(tflite.getOutputIndex(output)).dataType();
            outputProbabilityBuffer = TensorBuffer.createFixedSize(probabilityShape, probabilityDataType);

            // 添加图像预处理方式
            //about input/output more info: https://www.tensorflow.org/lite/inference_with_metadata/lite_support
        } catch (Exception e) {
    
    
            e.printStackTrace();
            throw new Exception("load model fail!");
        }
    }


    private ByteBuffer preProcess(Bitmap bitmap) {
    
    

        // 设置想要的大小（也即是model的input shape）
        int newWidth = imageShape[1];
        int newHeight = imageShape[2];

        bitmap=Bitmap.createScaledBitmap(bitmap, newWidth, newHeight, false);

        //buffer初始化    //定义一个Buffer,可以直接加载run()
        ByteBuffer inputBuffer = ByteBuffer.allocateDirect(4 * 1 * newWidth * newHeight * 1);
        inputBuffer.order(ByteOrder.nativeOrder());
        inputBuffer.rewind();

        // The bitmap shape should be 28 x 28
        int[] pixels=new int[newWidth * newHeight];

        bitmap.getPixels(pixels, 0, newWidth, 0, 0, newWidth, newHeight);


        for(int i =0;i<newWidth * newHeight;i++){
    
    
            int pixel = pixels[i];
            float avg = (((pixel >> 16) & 0xFF) * 38 + ((pixel >> 8) & 0xFF) * 75 + (pixel & 0xFF) * 15) >> 7;  //RGB图像转灰度图，pixels是多字节直接换算后的整数，所以还是用位运算更直接简便
            inputBuffer.putFloat(avg);
        }
        return inputBuffer;

    }

    // 执行预测
    float[] predict(Bitmap bmp) throws Exception {
    
       //传入一张bitmap图

        ByteBuffer inputImageBuffer= preProcess(bmp);

        try {
    
    
            tflite.run(inputImageBuffer, outputProbabilityBuffer.getBuffer().rewind());   //开始推理
        } catch (Exception e) {
    
    
            throw new Exception("predict image fail! log:" + e);
        }

        float[] results = outputProbabilityBuffer.getFloatArray();
        int l = getMaxResult(results);
        return new float[]{
    
    l, results[l]};
    }

    // 获取概率最大的标签
    public static int getMaxResult(float[] result) {
    
    

        float probability = 0;
        int r = 0;
        for (int i = 0; i < result.length; i++) {
    
    
            if (probability < result[i]) {
    
    
                probability = result[i];
                r = i;
            }
        }
        return r;
    }

}

3. Facebook AR

Now that the facial expression recognition problem has been solved, let's solve the realization of Facebook AR.

First of all, the overall framework of the Android software:

Among them, AR is implemented using the Google ARCore framework. The following figure is the flow chart of ARCore. By taking the camera image data, position analysis, rendering effect, and repeating it repeatedly, it feels that the AR effect has been realized.

Combined with this design, the final design flow chart is as follows:

The Tflite module is facial expression recognition. After obtaining the image data, analyze the facial expressions, load different facial makeup styles according to different expressions, and then render the facial makeup on the human face.

For example, the following code is to render different facial makeup through different expression categories (0~6)

switch (maxIndex) {
    
    
    case 0:
        augmentedFaceRenderer_0.draw(projectionMatrix, viewMatrix, modelMatrix, colorCorrectionRgba, face);
        break;
    case 1:
        augmentedFaceRenderer_1.draw(projectionMatrix, viewMatrix, modelMatrix, colorCorrectionRgba, face);
        break;
    case 2:
        augmentedFaceRenderer_2.draw(projectionMatrix, viewMatrix, modelMatrix, colorCorrectionRgba, face);
        break;
    case 3:
        augmentedFaceRenderer_3.draw(projectionMatrix, viewMatrix, modelMatrix, colorCorrectionRgba, face);
        break;
    case 4:
        augmentedFaceRenderer_4.draw(projectionMatrix, viewMatrix, modelMatrix, colorCorrectionRgba, face);
        break;
    case 5:
        augmentedFaceRenderer_5.draw(projectionMatrix, viewMatrix, modelMatrix, colorCorrectionRgba, face);
        break;
    case 6:
        augmentedFaceRenderer_6.draw(projectionMatrix, viewMatrix, modelMatrix, colorCorrectionRgba, face);
        break;
}

4. Demonstration effect

By changing different expressions, the face of Facebook can be changed:

Facebook AR Demo_x264

Code resource:
Link: https://pan.baidu.com/s/1Gb_dQwF8yq1SL8Zg76iZZw?pwd=4c4w
Extraction code: 4c4w