Fall detection and recognition 4: C++ implementation of fall detection (including source code, real-time fall detection)

Table of contents

1 Introduction

2. Fall detection model (YOLOv5)

(1) Fall detection model training

(2) Convert Pytorch model to ONNX model

(3) Convert ONNX model to TNN model

3. Deployment on the C++ end of fall detection and recognition

(1) Project structure

(2) Configure the development environment (OpenCV+OpenCL+base-utils+TNN)

(3) Deploy the TNN model

(4) CMake configuration

(5) main source code

(6) Compile and run the source code

4. Fall detection and recognition effect

5. Project source code download

1 Introduction

This is the " C++ implementation of fall detection (including source code, real-time fall detection)" of the project " Fall Detection and Recognition " series ; this article mainly shares the translation of the fall detection model of YOLOv5 after Python training into C/C++. We will develop a simple fall detection C/C++ Demo that can run in real time, which can detect three states of the human body : up (standing), bending (bending, squatting) and down (lying down, falling) . Under the acceleration of OpenCL, the fall detection and recognition demo can achieve real-time detection and recognition effects, basically meeting the performance requirements of the business.

First show a fall detection and recognition effect (three states up, bending and down):

[Respect originality, please indicate the source for reprinting] https://blog.csdn.net/guyuealian/article/details/130250838

For more articles on the "Fall Detection and Recognition" series, please refer to:

Fall detection and recognition 1: Fall detection data set (including download link): https://blog.csdn.net/guyuealian/article/details/130184256
Fall detection and recognition 2: YOLOv5 realizes fall detection (including fall detection data set and training code): https://blog.csdn.net/guyuealian/article/details/130250738
Fall detection and recognition 3: Android implements fall detection (including source code, real-time fall detection): https://blog.csdn.net/guyuealian/article/details/130250824
Fall detection and recognition 4: C++ implementation of fall detection (including source code, real-time fall detection): https://blog.csdn.net/guyuealian/article/details/130250838

2. Fall detection model (YOLOv5)

(1) Fall detection model training

For the training process of fall detection, please refer to Fall Detection and Recognition 2: YOLOv5 Realizes Fall Detection (including fall detection data set and training code)

In order to be able to deploy on the development board or mobile phone platform, I carried out a simple model lightweight on YOLOv5s, and developed a lightweight version of the yolov5s05_416 and yolov5s05_320 models; the lightweight model can achieve real-time detection results on ordinary Android phones , the CPU (4 threads) is about 30ms, and the GPU is about 25ms, which basically meets the performance requirements of the business. The following table gives the calculation and parameter quantities of the yolov5s model, the lightweight models yolov5s05_416 and yolov5s05_320, and their detection accuracy

Model	input-size	params(M)	GFLOPs	mAP_0.5:0.95
yolov5s	640×640	7.2	16.5	0.73693
yolov5s05	416×416	1.7	1.8	0.50567
yolov5s05	320×320	1.7	1.1	0.44821

(2) Convert Pytorch model to ONNX model

After training the yolov5s model, you need to convert the Pytorch model to the ONNX model first, and use onnx-simplifier to simplify the network structure. The Python version has provided the ONNX conversion script. The terminal input command is as follows:

# 转换yolov5s05模型
python export.py --weights "data/model/yolov5s05_320/weights/best.pt" --img-size 320 320

# 转换yolov5s模型
python export.py --weights "data/model/yolov5s_640/weights/best.pt" --img-size 640 640

GitHub： https://github.com/daquexian/onnx-simplifier
Install: pip3 install onnx-simplifier

(3) Convert ONNX model to TNN model

Currently, on the C++ side, there are multiple deployment methods for the CNN model. You can use deployment tools such as TNN, MNN, NCNN, and TensorRT. I use TNN to deploy on the Android side.

TNN conversion tool:

(1) To convert ONNX model to TNN model, please refer to TNN official instructions: TNN/onnx2tnn.md at master Tencent/TNN GitHub

(2) One-click conversion, a must for lazy people: One-click conversion of Caffe, ONNX, TensorFlow to NCNN, MNN, Tengine (there may be version problems, the TNN model converted by this tool may not be compatible, it is recommended to build the source code to convert, September 25, 2022 test available)

After the conversion is successful, two files (*.tnnproto and *.tnnmodel) will be generated, which will be used later after downloading

3. Deployment on the C++ end of fall detection and recognition

The project IDE development tool uses CLion, and the relevant dependent libraries mainly include OpenCV, base-utils, TNN and OpenCL (optional), among which OpenCV must be installed, OpenCL is used for model acceleration, base-utils and TNN have been configured, no need to install;

The project is only tested on Ubuntu 18.04, please configure the development environment by yourself under the Windows system.

(1) Project structure

(2) Configure the development environment (OpenCV+OpenCL+base-utils+TNN)

The project is only tested on Ubuntu18.04, please configure and compile it yourself under Windows system

Install OpenCV: image processing

Image processing (such as reading pictures, image cropping, etc.) needs to be processed using the OpenCV library

Installation tutorial: Install opencv and opencv_contrib on Ubuntu 18.04

The OpenCV library uses the opencv-4.3.0 version, and the opencv_contrib library is not used temporarily, so it is not necessary to install it

Install OpenCL: Model Acceleration

Installation tutorial: Ubuntu16.04 installs OpenCV&OpenCL_xiaozl_284's blog-CSDN blog_clinfo source code download

OpenCL is used for model GPU acceleration. If OpenCL is not used for model inference acceleration, the pure C++ inference model will be extremely slow

base-utils: C++ library

GitHub: https://github.com/PanJinquan/base-utils (No need to install, the project is already configured)

base_utils is a commonly used C++ library for personal development, which integrates commonly used algorithms such as C/C++ OpenCV

TNN: model inference

GitHub: https://github.com/Tencent/TNN (No need to install, the project is already configured)

The high-performance, lightweight neural network inference framework open sourced by Tencent Youtu Lab has many outstanding advantages such as cross-platform, high performance, model compression, and code pruning. On the basis of the original Rapidnet and ncnn frameworks, the TNN framework further strengthens the support and performance optimization of mobile devices. At the same time, it draws on the high performance and good scalability characteristics of the industry's mainstream open source frameworks, and expands the support for background X86 and NV GPU. The TNN on the mobile phone has been implemented in many applications such as mobile QQ, Weishi, and Ptu. The TNN on the server, as the basic acceleration framework of Tencent Cloud's AI, has provided acceleration support for the implementation of many businesses.

(3) Deploy the TNN model

The project model reasoning uses the TNN deployment framework (supporting multi-threaded CPU and GPU accelerated reasoning); the image processing uses the OpenCV library, and the model acceleration uses OpenCL, which can achieve real-time processing on ordinary computer equipment.

If you want to deploy your self-trained fall detection model in this C++ Demo, you can convert the trained Pytorch model to ONNX, then convert it to a TNN model, and then replace the original model with your own TNN model.

(4) CMake configuration

This is CMakeLists.txt, which mainly configures the four libraries of OpenCV+OpenCL+base-utils+TNN , please configure and compile by yourself under Windows system

cmake_minimum_required(VERSION 3.5)
project(Detector)

add_compile_options(-fPIC) # fix Bug: can not be used when making a shared object
set(CMAKE_CXX_FLAGS "-Wall -std=c++11 -pthread")
#set(CMAKE_CXX_FLAGS_RELEASE "-O2 -DNDEBUG")
#set(CMAKE_CXX_FLAGS_DEBUG "-g")

if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
    # -DCMAKE_BUILD_TYPE=Debug
    # -DCMAKE_BUILD_TYPE=Release
    message(STATUS "No build type selected, default to Release")
    set(CMAKE_BUILD_TYPE "Release" CACHE STRING "Build type (default Debug)" FORCE)
endif ()

# opencv set
find_package(OpenCV REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS} ./src/)
#MESSAGE(STATUS "OpenCV_INCLUDE_DIRS = ${OpenCV_INCLUDE_DIRS}")

# base_utils
set(BASE_ROOT 3rdparty/base-utils) # 设置base-utils所在的根目录
add_subdirectory(${BASE_ROOT}/base_utils/ base_build) # 添加子目录到build中
include_directories(${BASE_ROOT}/base_utils/include)
include_directories(${BASE_ROOT}/base_utils/src)
MESSAGE(STATUS "BASE_ROOT = ${BASE_ROOT}")


# TNN set
# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds it for you.
# Gradle automatically packages shared libraries with your APK.
# build for platform
# set(TNN_BUILD_SHARED OFF CACHE BOOL "" FORCE)
if (CMAKE_SYSTEM_NAME MATCHES "Android")
    set(TNN_OPENCL_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_ARM_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_BUILD_SHARED OFF CACHE BOOL "" FORCE)
    set(TNN_OPENMP_ENABLE ON CACHE BOOL "" FORCE)  # Multi-Thread
    #set(TNN_HUAWEI_NPU_ENABLE OFF CACHE BOOL "" FORCE)
    add_definitions(-DTNN_OPENCL_ENABLE)           # for OpenCL GPU
    add_definitions(-DTNN_ARM_ENABLE)              # for Android CPU
    add_definitions(-DDEBUG_ANDROID_ON)            # for Android Log
    add_definitions(-DPLATFORM_ANDROID)
elseif (CMAKE_SYSTEM_NAME MATCHES "Linux")
    set(TNN_OPENCL_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_CPU_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_X86_ENABLE OFF CACHE BOOL "" FORCE)
    set(TNN_QUANTIZATION_ENABLE OFF CACHE BOOL "" FORCE)
    set(TNN_OPENMP_ENABLE ON CACHE BOOL "" FORCE)  # Multi-Thread
    add_definitions(-DTNN_OPENCL_ENABLE)           # for OpenCL GPU
    add_definitions(-DDEBUG_ON)                    # for WIN/Linux Log
    add_definitions(-DDEBUG_LOG_ON)                # for WIN/Linux Log
    add_definitions(-DDEBUG_IMSHOW_OFF)            # for OpenCV show
    add_definitions(-DPLATFORM_LINUX)
elseif (CMAKE_SYSTEM_NAME MATCHES "Windows")
    set(TNN_OPENCL_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_CPU_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_X86_ENABLE ON CACHE BOOL "" FORCE)
    set(TNN_QUANTIZATION_ENABLE OFF CACHE BOOL "" FORCE)
    set(TNN_OPENMP_ENABLE ON CACHE BOOL "" FORCE)  # Multi-Thread
    add_definitions(-DTNN_OPENCL_ENABLE)           # for OpenCL GPU
    add_definitions(-DDEBUG_ON)                    # for WIN/Linux Log
    add_definitions(-DDEBUG_LOG_ON)                # for WIN/Linux Log
    add_definitions(-DDEBUG_IMSHOW_OFF)            # for OpenCV show
    add_definitions(-DPLATFORM_WINDOWS)
endif ()
set(TNN_ROOT 3rdparty/TNN)
include_directories(${TNN_ROOT}/include)
include_directories(${TNN_ROOT}/third_party/opencl/include)
add_subdirectory(${TNN_ROOT}) # 添加外部项目文件夹
set(TNN -Wl,--whole-archive TNN -Wl,--no-whole-archive)# set TNN library
MESSAGE(STATUS "TNN_ROOT = ${TNN_ROOT}")


# Detector
include_directories(src)
set(SRC_LIST
        src/yolov5.cpp
        src/Interpreter.cpp)
add_library(dmcv SHARED ${SRC_LIST})
target_link_libraries(dmcv ${OpenCV_LIBS} base_utils)
MESSAGE(STATUS "DIR_SRCS = ${SRC_LIST}")

#add_executable(Detector src/main.cpp)
#add_executable(Detector src/main_for_detect.cpp)
add_executable(Detector src/main_for_yolov5.cpp)
target_link_libraries(Detector dmcv ${TNN} -lpthread)

(5) main source code

The demo of fall detection is provided in the main program src/main_for_yolov5.cpp:

//
// Created by Pan on 2020/6/24.
//

#include <iostream>
#include <string>
#include <vector>
#include "file_utils.h"
#include "yolov5.h"

using namespace dl;
using namespace vision;
using namespace std;


void test_yolov5_detector() {
    const int num_thread = 1;
    DeviceType device = GPU; // 使用GPU运行，需要配置好OpenCL
    //DeviceType device = CPU; // 使用CPU运行

    // 测试YOLOv5s_640
    string proto_file = "../data/tnn/yolov5/yolov5s_640.sim.tnnproto";
    string model_file = "../data/tnn/yolov5/yolov5s_640.sim.tnnmodel";
    YOLOv5Param model_param = YOLOv5s_640;//模型参数

    // 测试YOLOv5s05_416
    //string proto_file = "../data/tnn/yolov5/yolov5s05_416.sim.tnnproto";
    //string model_file = "../data/tnn/yolov5/yolov5s05_416.sim.tnnmodel";
    //YOLOv5Param model_param = YOLOv5s05_416;//模型参数

    // 测试YOLOv5s05_320
    //string proto_file = "../data/tnn/yolov5/yolov5s05_320.sim.tnnproto";
    //string model_file = "../data/tnn/yolov5/yolov5s05_320.sim.tnnmodel";
    //YOLOv5Param model_param = YOLOv5s05_320;//模型参数

    // 设置检测阈值
    const float scoreThresh = 0.5;
    const float iouThresh = 0.5;
    YOLOv5 *detector = new YOLOv5(model_file,
                                  proto_file,
                                  model_param,
                                  num_thread,
                                  device);

    string image_dir = "../data/test_image/fall";
    //string image_dir = "../data/test_image/person";
    vector<string> image_list = get_files_list(image_dir);
    for (string image_path:image_list) {
        cv::Mat bgr_image = cv::imread(image_path);
        if (bgr_image.empty()) continue;
        FrameInfo resultInfo;
        printf("init frame\n");
        // 开始检测
        detector->detect(bgr_image, &resultInfo, scoreThresh, iouThresh);
        // 可视化代码
        detector->visualizeResult(bgr_image, &resultInfo);
    }
    delete detector;
    detector = nullptr;
    printf("FINISHED.\n");

}

int main() {
    test_yolov5_detector();
    return 0;
}

(6) Compile and run the source code

Compile the script, or directly: bash build.sh

#!/usr/bin/env bash
if [ ! -d "build/" ];then
  mkdir "build"
else
  echo "exist build"
fi
cd build
cmake ..
make -j4
sleep 1
./demo

If you want to test the performance of CPU operation, please modify src/main_for_yolov5.cpp

DeviceType device = CPU;

If you want to test the performance of GPU running, please modify src/main_for_yolov5.cpp (OpenCL needs to be configured)

DeviceType device = GPU; //Use GPU by default

The following screenshots show the performance comparison screenshots with OpenCL acceleration turned on. The pure C++ inference mode takes a few seconds, but after OpenCL acceleration is turned on, the GPU mode takes only a dozen milliseconds, and the performance is greatly improved.

CPU
GPU

4. Fall detection and recognition effect

The following figure shows the fall detection effect of the C++ version:

The following GIF is the fall detection and recognition effect of the Python version. The results of the C++ version and the Python version are almost the same.

5. Project source code download

[Fall detection and recognition C/C++ source code download] Fall detection and recognition 4: C++ realizes fall detection (including source code, real-time fall detection)

The whole set of project source code content includes:

The project provides YOLOv5 fall detection model: including yolov5s model, lightweight model yolov5s05_416 and yolov5s05_320 three fall detection models; it can be detected and recognized in real time on ordinary mobile phones, CPU (4 threads) is about 30ms, GPU is about 25ms; high-precision version yolov5s is included Fall detection model, CPU (4 threads) about 250ms, GPU about 100ms

The C++ source code of the project supports both CPU and GPU. GPU model acceleration needs to be configured with OpenCL, otherwise the speed will be very slow

The project source code does not contain Python training code and Android source code;

If you want to experience the effect of fall detection and recognition, you can download the Android version for testing. The core algorithm of fall detection and recognition in the Android and C++ versions is the same

For fall detection Python training, please refer to: Fall detection and recognition 2: YOLOv5 realizes fall detection (including fall detection data set and training code)
For fall detection Android deployment, please refer to: Fall detection and recognition 3: Android implements fall detection (including source code, real-time fall detection)