Facial expression recognition 4: C++ realizes expression recognition (including source code, real-time detection)

Table of contents

1. Facial expression recognition method

2. Face detection method

3. Facial expression recognition model (Python)

(1) Training of facial expression recognition model

(2) Convert the Pytorch model to the ONNX model

(3) Convert ONNX model to TNN model

4. C/C++ deployment of facial expression recognition model

(1) Project structure

(2) Configure the development environment (OpenCV+OpenCL+base-utils+TNN)

(3) Deploy the TNN model

(4) CMake configuration

(5) main source code

(6) Compile and run the source code

(7) Demo test effect

5. Project source code download

This is the project "Facial Expression Recognition" series " C++ realizes expression recognition (including source code, real-time detection) ", mainly sharing the deployment of the facial expression recognition model (mobilenet_v2) trained in Python to the C/C++ platform. We will develop a simple C/C++ Demo that can run in real time for facial expression recognition. The accuracy rate is quite high. The facial expression recognition accuracy rate using the lightweight mobilenet_v2 model can also be as high as 94.72%, which basically meets the business performance requirements. The C/C++ version of expression recognition model reasoning supports CPU and GPU acceleration, and GPU (OpenCL) acceleration can be enabled to achieve real-time detection and recognition results, basically meeting the performance requirements of the business.

Let me first show you the C/C++ version of the facial expression recognition Demo effect (different expressions are marked with different color boxes)

[ Respect originality, please indicate the source for reprinting ] https://blog.csdn.net/guyuealian/article/details/129467023

For more articles on the " Facial Expression Recognition " series, please refer to:

1. Facial expression recognition method

There are many implementation schemes for facial expression recognition methods. The most conventional method is used here: based on face detection + facial expression classification recognition method , that is, first use a general face detection model to perform face detection, then crop the face area, and then Train a facial expression classifier to complete facial expression recognition;

The advantage of this is that the existing face detection model can be used without retraining the face detection model, which can reduce the cost of manual labeling; while face data is relatively easy to collect, and the classification model can be optimized in a targeted manner .

2. Face detection method

For the face detection training code of this project, please refer to: https://github.com/Linzaer/Ultra-Light-Fast-Generic-Face-Detector-1MB

This is an improved and lightweight face detection model based on SSD. It is very slim. The whole model is only about 1.7M, and it can be detected in real time on ordinary Android phones. There are a lot of ready-made methods for face detection on the Internet, and I don't need to limit my method at all.

For the method of face detection, you can refer to my other blog:

Pedestrian detection, face detection and face key point detection (C++/Android source code)

3. Facial expression recognition model (Python)

(1) Training of facial expression recognition model

This blog post does not contain the python version of the facial expression model and related training codes. For the training method of the facial expression recognition model, please refer to my other blog post " Facial Expression Recognition 2: Pytorch Realizes Expression Recognition (Including Expression Recognition Dataset and Training Code) ": Facial Expression Recognition 2: Pytorch realizes expression recognition (including expression recognition data set and training code)

(2) Convert the Pytorch model to the ONNX model

At present, there are many deployment methods for the CNN model, and deployment tools such as TNN, MNN, NCNN, and TensorRT can be used. I use TNN to deploy on the C/C++ side. The deployment process can be divided into four steps: training model -> converting model to ONNX model -> converting ONNX model to TNN model -> deploying TNN model in C/C++.

After training the Pytorch model, we need to convert the model to ONNX model for subsequent model deployment.

The original project provides a conversion script, you only need to modify model_file to your model path
convert_torch_to_onnx.py implements the script to convert the Pytorch model to the ONNX model

python libs/convert/convert_torch_to_onnx.py

"""
This code is used to convert the pytorch model into an onnx format model.
"""
import sys
import os

sys.path.insert(0, os.getcwd())
import torch.onnx
import onnx
from classifier.models.build_models import get_models
from basetrainer.utils import torch_tools


def build_net(model_file, net_type, input_size, num_classes, width_mult=1.0):
    """
    :param model_file: 模型文件
    :param net_type: 模型名称
    :param input_size: 模型输入大小
    :param num_classes: 类别数
    :param width_mult:
    :return:
    """
    model = get_models(net_type, input_size, num_classes, width_mult=width_mult, is_train=False, pretrained=False)
    state_dict = torch_tools.load_state_dict(model_file)
    model.load_state_dict(state_dict)
    return model


def convert2onnx(model_file, net_type, input_size, num_classes, width_mult=1.0, device="cpu", onnx_type="default"):
    model = build_net(model_file, net_type, input_size, num_classes, width_mult=width_mult)
    model = model.to(device)
    model.eval()
    model_name = os.path.basename(model_file)[:-len(".pth")] + ".onnx"
    onnx_path = os.path.join(os.path.dirname(model_file), model_name)
    # dummy_input = torch.randn(1, 3, 240, 320).to("cuda")
    dummy_input = torch.randn(1, 3, input_size[1], input_size[0]).to(device)
    # torch.onnx.export(model, dummy_input, onnx_path, verbose=False,
    #                   input_names=['input'],output_names=['scores', 'boxes'])
    do_constant_folding = True
    if onnx_type == "default":
        torch.onnx.export(model, dummy_input, onnx_path, verbose=False, export_params=True,
                          do_constant_folding=do_constant_folding,
                          input_names=['input'],
                          output_names=['output'])
    elif onnx_type == "det":
        torch.onnx.export(model,
                          dummy_input,
                          onnx_path,
                          do_constant_folding=do_constant_folding,
                          export_params=True,
                          verbose=False,
                          input_names=['input'],
                          output_names=['scores', 'boxes', 'ldmks'])
    elif onnx_type == "kp":
        torch.onnx.export(model,
                          dummy_input,
                          onnx_path,
                          do_constant_folding=do_constant_folding,
                          export_params=True,
                          verbose=False,
                          input_names=['input'],
                          output_names=['output'])
    onnx_model = onnx.load(onnx_path)
    onnx.checker.check_model(onnx_model)
    print(onnx_path)


if __name__ == "__main__":
    net_type = "mobilenet_v2"
    width_mult = 1.0
    input_size = [128, 128]
    num_classes = 2
    model_file = "work_space/mobilenet_v2_1.0_CrossEntropyLoss/model/best_model_022_98.1848.pth"
    convert2onnx(model_file, net_type, input_size, num_classes, width_mult=width_mult)

(3) Convert ONNX model to TNN model

At present, there are many deployment methods for the CNN model, and deployment tools such as TNN, MNN, NCNN, and TensorRT can be used. I use TNN for deployment on the C/C++ side.

TNN conversion tool:

(1) To convert ONNX model to TNN model, please refer to TNN official instructions: TNN/onnx2tnn.md at master Tencent/TNN GitHub

(2) One-click conversion, a must for lazy people: One-click conversion of Caffe, ONNX, TensorFlow to NCNN, MNN, Tengine (there may be version problems, the TNN model converted by this tool may not be compatible, it is recommended to build the source code to convert, September 25, 2022 test available)

4. C/C++ deployment of facial expression recognition model

The project IDE development tool uses CLion, and the relevant dependent libraries mainly include OpenCV, base-utils, TNN and OpenCL (optional), among which OpenCV must be installed, OpenCL is used for model acceleration, and base-utils and TNN have been configured without installation;

The project is only tested on Ubuntu 18.04, please configure the development environment by yourself under the Windows system.

(1) Project structure

(2) Configure the development environment (OpenCV+OpenCL+base-utils+TNN)

The project is only tested on Ubuntu18.04, please configure and compile it yourself under Windows system

Install OpenCV: image processing

Image processing (such as reading pictures, image cropping, etc.) needs to be processed using the OpenCV library

Installation tutorial: Install opencv and opencv_contrib on Ubuntu 18.04

The OpenCV library uses the opencv-4.3.0 version, and the opencv_contrib library is not used temporarily, so it is not necessary to install it

Install OpenCL: Model Acceleration

Installation tutorial: Ubuntu16.04 installs OpenCV&OpenCL_xiaozl_284's blog-CSDN blog_clinfo source code download

OpenCL is used for model GPU acceleration. If OpenCL is not used for model inference acceleration, the pure C++ inference model will be extremely slow

base-utils: C++ library

GitHub: https://github.com/PanJinquan/base-utils (No need to install, the project is already configured)

base_utils is a commonly used C++ library for personal development, which integrates commonly used algorithms such as C/C++ OpenCV

TNN: model inference

GitHub: https://github.com/Tencent/TNN (No need to install, the project is already configured)

The high-performance, lightweight neural network inference framework open sourced by Tencent Youtu Lab has many outstanding advantages such as cross-platform, high performance, model compression, and code pruning. On the basis of the original Rapidnet and ncnn frameworks, the TNN framework further strengthens the support and performance optimization of mobile devices. At the same time, it draws on the high performance and good scalability characteristics of the industry's mainstream open source frameworks, and expands the support for background X86 and NV GPU. TNN on the mobile phone side has been implemented in many applications such as mobile QQ, Weishi, and Ptu. The TNN on the server side, as the basic acceleration framework of Tencent Cloud's AI, has provided acceleration support for the implementation of many businesses.

(3) Deploy the TNN model

The project implements C/C++ version of license plate detection and license plate recognition, license plate detection model YOLOv5 and license plate recognition model PlateNet, model reasoning uses TNN deployment framework (supports multi-threaded CPU and GPU accelerated reasoning); image processing uses OpenCV library, model acceleration uses OpenCL can achieve real-time processing on common devices.

If you want to deploy your own trained license plate detection model YOLOv5 and license plate recognition model PlateNet in this Demo, you can convert the trained Pytorch model to ONNX, then convert it to a TNN model, and then replace the original model with your own TNN model That's it.

(4) CMake configuration

This is CMakeLists.txt, which mainly configures the four libraries of OpenCV+OpenCL+base-utils+TNN , please configure and compile by yourself under Windows system

cmake_minimum_required(VERSION 3.5)
project(Detector)

add_compile_options(-fPIC) # fix Bug: can not be used when making a shared object
set(CMAKE_CXX_FLAGS "-Wall -std=c++11 -pthread")
#set(CMAKE_CXX_FLAGS_RELEASE "-O2 -DNDEBUG")
#set(CMAKE_CXX_FLAGS_DEBUG "-g")

if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
    # -DCMAKE_BUILD_TYPE=Debug
    # -DCMAKE_BUILD_TYPE=Release
    message(STATUS "No build type selected, default to Release")
    set(CMAKE_BUILD_TYPE "Release" CACHE STRING "Build type (default Debug)" FORCE)
endif ()

# opencv set
find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS} ./src/)
#MESSAGE(STATUS "OpenCV_INCLUDE_DIRS = ${OpenCV_INCLUDE_DIRS}")

# base_utils
set(BASE_ROOT 3rdparty/base-utils) # 设置base-utils所在的根目录
add_subdirectory(${BASE_ROOT}/base_utils/ base_build) # 添加子目录到build中
include_directories(${BASE_ROOT}/base_utils/include)
include_directories(${BASE_ROOT}/base_utils/src)
MESSAGE(STATUS "BASE_ROOT = ${BASE_ROOT}")


# TNN set
# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake buil ds it for you.
# Gradle automatically packages shared libraries with your APK.
# build for platform
# set(TNN_BUILD_SHARED OFF CACHE BOOL "" FORCE)
if (CMAKE_SYSTEM_NAME MATCHES "Android")
    set(TNN_OPENCL_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_ARM_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_BUILD_SHARED OFF CACHE BOOL "" FORCE)
    set(TNN_OPENMP_ENABLE ON CACHE BOOL "" FORCE)  # Multi-Thread
    #set(TNN_HUAWEI_NPU_ENABLE OFF CACHE BOOL "" FORCE)
    add_definitions(-DTNN_OPENCL_ENABLE)           # for OpenCL GPU
    add_definitions(-DTNN_ARM_ENABLE)              # for Android CPU
    add_definitions(-DDEBUG_ANDROID_ON)            # for Android Log
    add_definitions(-DPLATFORM_ANDROID)
elseif (CMAKE_SYSTEM_NAME MATCHES "Linux")
    set(TNN_OPENCL_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_CPU_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_X86_ENABLE OFF CACHE BOOL "" FORCE)
    set(TNN_QUANTIZATION_ENABLE OFF CACHE BOOL "" FORCE)
    set(TNN_OPENMP_ENABLE ON CACHE BOOL "" FORCE)  # Multi-Thread
    add_definitions(-DTNN_OPENCL_ENABLE)           # for OpenCL GPU
    add_definitions(-DDEBUG_ON)                    # for WIN/Linux Log
    add_definitions(-DDEBUG_LOG_ON)                # for WIN/Linux Log
    add_definitions(-DDEBUG_IMSHOW_OFF)            # for OpenCV show
    add_definitions(-DPLATFORM_LINUX)
elseif (CMAKE_SYSTEM_NAME MATCHES "Windows")
    set(TNN_OPENCL_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_CPU_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_X86_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_QUANTIZATION_ENABLE OFF CACHE BOOL "" FORCE)
    set(TNN_OPENMP_ENABLE ON CACHE BOOL "" FORCE)  # Multi-Thread
    add_definitions(-DTNN_OPENCL_ENABLE)           # for OpenCL GPU
    add_definitions(-DDEBUG_ON)                    # for WIN/Linux Log
    add_definitions(-DDEBUG_LOG_ON)                # for WIN/Linux Log
    add_definitions(-DDEBUG_IMSHOW_OFF)            # for OpenCV show
    add_definitions(-DPLATFORM_WINDOWS)
endif ()
set(TNN_ROOT 3rdparty/TNN)
include_directories(${TNN_ROOT}/include)
include_directories(${TNN_ROOT}/third_party/opencl/include)
add_subdirectory(${TNN_ROOT}) # 添加外部项目文件夹
set(TNN -Wl,--whole-archive TNN -Wl,--no-whole-archive)# set TNN library
MESSAGE(STATUS "TNN_ROOT = ${TNN_ROOT}")


# Detector
include_directories(src)
set(SRC_LIST
        src/object_detection.cpp
        src/classification.cpp
        src/Interpreter.cpp)
add_library(dmcv SHARED ${SRC_LIST})
target_link_libraries(dmcv ${OpenCV_LIBS} base_utils)
MESSAGE(STATUS "DIR_SRCS = ${SRC_LIST}")

add_executable(Detector src/main.cpp)
#add_executable(Detector src/main_for_detect.cpp)
#add_executable(Detector src/main_for_yolov5.cpp)
target_link_libraries(Detector dmcv ${TNN} -lpthread)

(5) main source code

The implementation of the function main in the main program provides the usage method of facial expression recognition, supports pictures, videos and camera tests

test_image_file(); // test image file

test_video_file(); // test video file

test_camera(); //test camera

//
// Created by Pan on 2020/6/24.
//

#include "object_detection.h"
#include "classification.h"
#include <iostream>
#include <string>
#include <vector>
#include "file_utils.h"
#include "image_utils.h"

using namespace dl;
using namespace vision;
using namespace std;


const int num_thread = 1; // 开启CPU线程数目
DeviceType device = GPU;  // 选择运行设备CPU/GPU
// 人脸检测模型
const char *det_model_file = (char *) "../data/tnn/face/rfb-face-mask-320-320_sim.opt.tnnmodel";
const char *det_proto_file = (char *) "../data/tnn/face/rfb-face-mask-320-320_sim.opt.tnnproto";
ObjectDetectionParam model_param = FACE_MODEL;//模型参数
// 分类模型
const char *cls_model_file = (char *) "../data/tnn/emotion/mobilenet_v2_112_112.tnnmodel";
const char *cls_proto_file = (char *) "../data/tnn/emotion/mobilenet_v2_112_112.tnnproto";
ClassificationParam ClassParam = EMOTION_MODEL;//模型参数

// 设置检测阈值
const float scoreThresh = 0.5;
const float iouThresh = 0.3;
ObjectDetection *detector = new ObjectDetection(det_model_file,
                                                det_proto_file,
                                                model_param,
                                                num_thread,
                                                device);

Classification *classifier = new Classification(cls_model_file,
                                                cls_proto_file,
                                                ClassParam,
                                                num_thread,
                                                device);

/***
 * 测试图片文件
 */
void test_image_file() {
    //测试图片的目录
    string image_dir = "../data/test_image";
    std::vector<string> image_list = get_files_list(image_dir);
    for (string image_path:image_list) {
        cv::Mat bgr_image = cv::imread(image_path);
        bgr_image = image_resize(bgr_image, 960);
        if (bgr_image.empty()) continue;
        FrameInfo resultInfo;
        // 进行人脸检测
        detector->detect(bgr_image, &resultInfo, scoreThresh, iouThresh);
        // 进行图像分类
        classifier->detect(bgr_image, &resultInfo);
        // 可视化检测结果
        classifier->visualizeResult(bgr_image, &resultInfo);
    }
    delete detector;
    detector = nullptr;
    delete classifier;
    classifier = nullptr;
    printf("FINISHED.\n");
}

/***
 * 测试视频文件
 * @return
 */
int test_video_file() {
    //测试视频文件
    string video_file = "../data/video/video-test.mp4";
    cv::VideoCapture cap;
    bool ret = get_video_capture(video_file, cap);
    cv::Mat frame;
    while (ret) {
        cap >> frame;
        if (frame.empty()) break;
        FrameInfo resultInfo;
        // 进行人脸检测
        detector->detect(frame, &resultInfo, scoreThresh, iouThresh);
        // 进行图像分类
        classifier->detect(frame, &resultInfo);
        // 可视化检测结果
        classifier->visualizeResult(frame, &resultInfo, 20);
    }
    cap.release();
    delete detector;
    detector = nullptr;
    delete classifier;
    classifier = nullptr;
    printf("FINISHED.\n");
    return 0;

}


/***
 * 测试摄像头
 * @return
 */
int test_camera() {
    int camera = 0; //摄像头ID号(请修改成自己摄像头ID号)
    cv::VideoCapture cap;
    bool ret = get_video_capture(camera, cap);
    cv::Mat frame;
    while (ret) {
        cap >> frame;
        if (frame.empty()) break;
        FrameInfo resultInfo;
        // 进行人脸检测
        detector->detect(frame, &resultInfo, scoreThresh, iouThresh);
        // 进行图像分类
        classifier->detect(frame, &resultInfo);
        // 可视化检测结果
        classifier->visualizeResult(frame, &resultInfo, 20);
    }
    cap.release();
    delete detector;
    detector = nullptr;
    delete classifier;
    classifier = nullptr;
    printf("FINISHED.\n");
    return 0;

}


int main() {
    test_image_file();   // 测试图片文件
    //test_video_file();   // 测试视频文件
    //test_camera();       //测试摄像头
    return 0;
}

(6) Compile and run the source code

Compile the script, or directly: bash build.sh

#!/usr/bin/env bash
if [ ! -d "build/" ];then
  mkdir "build"
else
  echo "exist build"
fi
cd build
cmake ..
make -j4
sleep 1
./demo

If you want to test the performance of CPU operation, please modify src/main.cpp

DeviceType device = CPU;

If you want to test the performance of GPU running, please modify src/main.cpp (OpenCL needs to be configured)

DeviceType device = GPU;

PS: The pure CPU C++ inference mode is time-consuming and takes a few seconds. After OpenCL acceleration is turned on, the GPU mode takes only a dozen milliseconds, and the performance is greatly improved.

(7) Demo test effect

The results of the C++ version and the Python version are almost the same. The following is the facial expression recognition effect display (where different expressions are represented by different colors)

5. Project source code download

C++ realizes expression recognition project source code download address: Facial expression recognition 4: C++ realizes expression recognition (including source code, which can be detected in real time)

The whole set of project source code content includes:

Provide C/C++ version of the face detection model

Provide C/C++ version of facial expression classification model

C++ source code supports CPU and GPU, and GPU (OpenCL) can be used for real-time detection and recognition (pure CPU inference speed is very slow, model acceleration needs to be configured with OpenCL, GPU inference takes about 15ms)

The project is configured with base-utils and TNN, while OpenCV and OpenCL need to be compiled and installed by themselves

C/C++ Demo supports pictures, videos, and camera tests

Android facial expression recognition APP Demo experience: https://download.csdn.net/download/guyuealian/87575425

Or link: https://pan.baidu.com/s/16OOi-qCENP4WbIeSzO5e9g Extraction code: cs5g

If you need training code for facial expression recognition, please refer to: " Facial Expression Recognition 2: Pytorch Realizes Expression Recognition (Including Expression Recognition Dataset and Training Code)" Facial Expression Recognition 2: Pytorch Realizes Expression Recognition (Including Expression Recognition Dataset and Training Code) Training code)_AI Eat Big Melon Blog-CSDN Blog