Android imports ncnn-android-yolov8-seg: to realize human body recognition and portrait segmentation

1 Introduction

In the last article, we Androidused to OpenCVimplement face recognition. In this article, we use OpenCV+YOLOv8+NCNNto implement the function of portrait segmentation.

First, let’s take a look at the effect. The human body will be recognized here, and it will be outlined with a blue frame, and there will be a label to mark what the recognized object is and what the probability is.
The recognized portrait will be covered with a layer of pink, which actually means that the entire human body outline has been recognized.

Insert image description here

2. What is YOLOv8

YOLOv8is the Ultralyticslatest family of object detection models 2023based on YOLO, offering advanced performance.
To understand YOLOv8, we must first take a look at YOLO （you only look once）the birth history of YOLO. For details, please refer to the childlike father of YOLO, Joseph Redmon Xiaoao CV Jianghu Ji . We will not introduce too much here. We only need to know and YOLObrowse it once. The category and location of objects in the image can be identified and image segmentation can be performed.

3. What is NCNN?

ncnnIt is a high-performance neural network forward computing framework optimized for mobile phones. ncnnThe deployment and use of the mobile terminal are deeply considered from the beginning of the design. No third-party dependencies, cross-platform, and the mobile cpuversion is faster than all currently known open source frameworks. Based on this ncnn, developers can easily transplant deep learning algorithms to mobile phones for efficient execution, develop artificial intelligence APPs, and bring AI to your fingertips. ncnnIt has been used in many Tencent applications, such as: QQ，Qzone，微信，天天 P 图etc.

NCNNCurrently, most CNNnetworks are supported, including YOLO. This means that YOLOalgorithms can be integrated into NCNNthe framework and executed efficiently on mobile phones. Therefore, NCNNand YOLOcan be used in conjunction with each other to achieve faster and more efficient target detection.

4. What is OpenCV

OpenCVIt is a cross-platform computer vision and machine learning software library. It is lightweight and efficient. It consists of a series of Cfunctions and a small number of C++classes. It also provides Python、Ruby、MATLABinterfaces in other languages and implements many common algorithms in image processing and computer vision. In this article, OpenCVit is mainly introduced through the role of image transformation and transmission.

5. Run ncnn-android-yolov8-seg

So, how to use it in Android OpenCV+YOLOv8+NCNN?
First, we can Githubfind this library on: Digital2Slave/ncnn-android-yolov8-seg , which has been used internally OpenCV+YOLOv8+NCNNto implement the portrait segmentation function. Here we can import this project and run Take a look at the effect.

5.1 Import ncnn-android-yolov8-seg

After we download the code of Digital2Slave/ncnn-android-yolov8-seg , we use Android Studio 3.6the import project.

At this time, an error message will be prompted

Caused by: java.lang.NullPointerException
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:787)
	at com.android.build.gradle.internal.ndk.NdkHandler.getPlatformVersion(NdkHandler.java:159)
	at com.android.build.gradle.internal.ndk.NdkHandler.supports64Bits(NdkHandler.java:332)
	at com.android.build.gradle.internal.ndk.NdkHandler.getSupportedAbis(NdkHandler.java:404)
	...

This is because we haven't configured NDKthe path yet

5.2 Configure CMake and NDK paths

local.propertiesAdd the following code

# 设置cmake路径，这里的路径要改成你的实际cmake路径
cmake.dir=C\:\\Developer\\Android_SDK\\cmake\\3.10.2.4988404
# 设置ndk路径，这里的路径要改成你的实际ndk路径
ndk.dir=C\:\\Developer\\Android_SDK\\ndk\\20.0.5594570

ndkThere will be version issues here , the version that needs to be Ndk16used Ndk20, and higher versions Ndkwill have compatibility issues.

If you use a higher version NDK, you need to addfopenmp

5.3 Configure NDK DANDROID_STL

In the code block appof build.gradle, you can add the following codeexternalNativeBuildcmake

arguments "-DANDROID_STL=c++_shared"

The overall code is as follows

externalNativeBuild {
    
    
    cmake {
    
    
        cppFlags "-std=c++11 -frtti -fexceptions"
        abiFilters 'arm64-v8a'
        arguments "-DANDROID_STL=c++_shared"
    }
}

5.4 Solve the error unknown argument

If we synchronize the project again, we can find the following error:

Execution failed for task ':app:externalNativeBuildDebug'.
> Build command failed.
  Error while executing process C:\Developer\Android_SDK\cmake\3.10.2.4988404\bin\cmake.exe with arguments {
    
    --build E:\WorkSpace\Demo\Tnn\New\ncnn-android-yolov8-seg\app\.externalNativeBuild\cmake\debug\arm64-v8a --target yolov8ncnn}
...
clang++.exe: error: unknown argument: '-static-openmp'
ninja: build stopped: subcommand failed.

Here we search globally -static-openmpand delete them all.

Insert image description here
Then recompile it C++ Projects

and click Run, and find that the project runs normally.

6. Connect to OpenCV+YOLOv8+NCNN

Then we can connect it in our own project OpenCV+YOLOv8+NCNN, but after looking at the source code, we can find that ncnn-android-yolov8-segthe camera in this project is used c/c++, but in our project, it is implemented using Javalayers .Camera APIAPI

Insert image description here

If you want to integrate it in your own project ncnn, you need to ncnn-android-yolov8-segextract the core code from it and then connect it to Javait Camera API.

So what needs to be done?

Let’s implement it in the next article

For details, see: Android connects to OpenCV+YOLOv8+NCNN in its own project to achieve portrait segmentation-CSDN Blog