Android ncnn-android-yolov8-seg source code analysis: realizing portrait segmentation

1 Introduction

In the last article, we have already started the portrait segmentation projectncnn-android-yolov8-seg, and we will extract it in subsequent articlesDemo, use the core code in your own project to access the human body recognition and portrait segmentation functions.

Let’s take a look at the effect first. The entire image is the original image from the camera. The upper left corner is the image we obtained after performing portrait recognition and segmentation.(未做镜像处理，所以暂时和原图左右是相反的)

Insert image description here

So how do we implement the portrait segmentation function in our own projects?

Looking at the source code ofncnn-android-yolov8-seg, we can find that the camera in this project is also usedc/c++, but in our project, the camera used. of and then connect it to in your own project, you need to extract the core code in is implemented. If you want to integrate Java layerCamera APIncnnncnn-android-yolov8-segJavaCamera API

So what needs to be done? Next, let’s take a look at its source code.

2. Source code analysis

First, let’s analyze the source code in Digital2Slave/ncnn-android-yolov8-seg Demo

2.1 Load model

Loading the model is inJava_com_tencent_yolov8ncnn_Yolov8Ncnn_loadModel method, the source code is attached here

JNIEXPORT jboolean JNICALL Java_com_tencent_yolov8ncnn_Yolov8Ncnn_loadModel(JNIEnv* env, jobject thiz, jobject assetManager, jint modelid, jint cpugpu)
{
    
    
    if (modelid < 0 || modelid > 6 || cpugpu < 0 || cpugpu > 1)
    {
    
    
        return JNI_FALSE;
    }

    AAssetManager* mgr = AAssetManager_fromJava(env, assetManager);

    __android_log_print(ANDROID_LOG_DEBUG, "ncnn", "loadModel %p", mgr);

    const char* modeltypes[] =
    {
    
    
        "n",
        "s",
    };

    const int target_sizes[] =
    {
    
    
        320,
        320,
    };

    const float mean_vals[][3] =
    {
    
    
        {
    
    103.53f, 116.28f, 123.675f},
        {
    
    103.53f, 116.28f, 123.675f},
    };

    const float norm_vals[][3] =
    {
    
    
        {
    
     1 / 255.f, 1 / 255.f, 1 / 255.f },
        {
    
     1 / 255.f, 1 / 255.f, 1 / 255.f },
    };

    const char* modeltype = modeltypes[(int)modelid];
    int target_size = target_sizes[(int)modelid];
    bool use_gpu = (int)cpugpu == 1;

    // reload
    {
    
    
        ncnn::MutexLockGuard g(lock);

        if (use_gpu && ncnn::get_gpu_count() == 0)
        {
    
    
            // no gpu
            delete g_yolo;
            g_yolo = 0;
        }
        else
        {
    
    
            if (!g_yolo)
                g_yolo = new Yolo;
            g_yolo->load(mgr, modeltype, target_size, mean_vals[(int)modelid], norm_vals[(int)modelid], use_gpu);
        }
    }

    return JNI_TRUE;
}

2.1.1 modelid: select a model

According tomodelid, select a specific model in modeltypes.

const char* modeltypes[] =
{
    
    
    "n",
    "s",
};
const char* modeltype = modeltypes[(int)modelid];

The model categories here correspond to the models in the asserts folder of the project. yolov8 is a model cluster, ranging from small to large: a>yolov8n, yolov8s, yolov8m, yolov8l, yolov8x, etc.
Insert image description here
Usuallyyolov8n is the fastest, please see the table below

2.1.2 cpugpu: Use CPU or GPU

Let’s look at the Java_com_tencent_yolov8ncnn_Yolov8Ncnn_loadModel method.
There is also a parameter cpugpu used to determine the use CPU or GPU, 0 is CPU, 1 is GPU.

bool use_gpu = (int)cpugpu == 1;

2.1.3 Initialize the model

Finally, callg_yolo->load() to initialize the model

g_yolo->load(mgr, modeltype, target_size, mean_vals[(int)modelid], norm_vals[(int)modelid], use_gpu);

2.2 Operating the camera

ncnn-android-yolov8-seg The camera operations in the Demo are all done through NDK’s Camera API

static MyNdkCamera* g_camera = 0;

JNIEXPORT jint JNI_OnLoad(JavaVM* vm, void* reserved)
{
    
    
    __android_log_print(ANDROID_LOG_DEBUG, "ncnn", "JNI_OnLoad");

    g_camera = new MyNdkCamera;

    return JNI_VERSION_1_4;
}

2.2.1 Open the camera

Open the camera and callg_camera->open()

JNIEXPORT jboolean JNICALL Java_com_tencent_yolov8ncnn_Yolov8Ncnn_openCamera(JNIEnv* env, jobject thiz, jint facing)
{
    
    
    if (facing < 0 || facing > 1)
        return JNI_FALSE;

    __android_log_print(ANDROID_LOG_DEBUG, "ncnn", "openCamera %d", facing);

    g_camera->open((int)facing);

    return JNI_TRUE;
}

2.2.2 Turn off the camera

Turning off the camera is calledg_camera->close()

JNIEXPORT jboolean JNICALL Java_com_tencent_yolov8ncnn_Yolov8Ncnn_closeCamera(JNIEnv* env, jobject thiz)
{
    
    
    __android_log_print(ANDROID_LOG_DEBUG, "ncnn", "closeCamera");

    g_camera->close();

    return JNI_TRUE;
}

2.3 Human body detection

Human detection is associated with NDK's Camera, and is completed in the on_image_render method of the camera callback Human detection

2.3.1 on_image_render: Human body detection

Let’s take a look at the source code of on_image_render. It mainly uses g_yolo->detect() for human body detection, g_yolo->draw() to mark the human body position, and uses Frame it out.

void MyNdkCamera::on_image_render(cv::Mat& rgb) const
{
    
    
    // nanodet
    {
    
    
        ncnn::MutexLockGuard g(lock);
        //cv::resize()

        if (g_yolo)
        {
    
    
            __android_log_print(ANDROID_LOG_DEBUG, "myncnn", "g_yolo:true");

            auto start = std::chrono::high_resolution_clock::now();

            std::vector<Object> objects;
            g_yolo->detect(rgb, objects); //人体检测

            start = std::chrono::high_resolution_clock::now();

            g_yolo->draw(rgb, objects); //标注人体位置，用框框出来
        }
        else
        {
    
    
            __android_log_print(ANDROID_LOG_DEBUG, "myncnn", "g_yolo:false");
            draw_unsupported(rgb);
        }
    }

    draw_fps(rgb); //绘制当前多少帧率
}

2.3.2 When is on_image_render called?

So when was theon_image_render method called? Let’s look at the method in ndkcamera.cpp >on_image

You can see that the NV21 image will be cropped and rotated, and then converted into RGB format, and then passed to on_image_render()Method processing

// crop and rotate nv21
cv::Mat nv21_croprotated(roi_h + roi_h / 2, roi_w, CV_8UC1);
{
    
    
    const unsigned char* srcY = nv21 + nv21_roi_y * nv21_width + nv21_roi_x;
    unsigned char* dstY = nv21_croprotated.data;
    ncnn::kanna_rotate_c1(srcY, nv21_roi_w, nv21_roi_h, nv21_width, dstY, roi_w, roi_h, roi_w, rotate_type);

    const unsigned char* srcUV = nv21 + nv21_width * nv21_height + nv21_roi_y * nv21_width / 2 + nv21_roi_x;
    unsigned char* dstUV = nv21_croprotated.data + roi_w * roi_h;
    ncnn::kanna_rotate_c2(srcUV, nv21_roi_w / 2, nv21_roi_h / 2, nv21_width, dstUV, roi_w / 2, roi_h / 2, roi_w, rotate_type);
}

// nv21_croprotated to rgb
cv::Mat rgb(roi_h, roi_w, CV_8UC3);
ncnn::yuv420sp2rgb(nv21_croprotated.data, roi_w, roi_h, rgb.data);

on_image_render(rgb);

2.3.3 Human body detection process

In other words, the NV21 data obtained from the camera will be rotated first, then converted into RGB format, and then handed over to. g_yolo->detect()Perform human body detection and mark the human body position throughg_yolo->draw()

At this point, our core source code is almost analyzed, so how do we integrate this function into our own project?
We will implement it in the next article: Android connects to OpenCV+YOLOv8+NCNN in its own project to achieve portrait segmentation - CSDN Blog.