Rockchip: Face orientation detection based on RK3568

The driver monitoring system is a real-time system that studies the driver's status based on driver facial image processing. First, the facial expression characteristics of people in a fatigue state are mined, and then these qualitative facial expression characteristics are quantified, and facial feature points and characteristic indicators are extracted as the basis for judgment. Then, a recognition method based on these parameters is summarized based on the experimental data, and finally the input is obtained. The received status data is identified and judged.

An infrared camera facing the driver monitors details such as the head, eyes, face, and hands in real time, and can detect the driver's status from eye closure, blinking, gaze direction, yawning, and head movement. Driver fatigue, distraction, and irregular driving are usually detected by detecting facial eyes and other facial features and behaviors, tracking changes at the same time, and extracting symptoms.

As for why infrared cameras are used, it is because the system requires that it can work in all working conditions (including daytime, night, forward light, backlight, etc.) and can adapt to various light source environments, that is, it can also be viewed at night, backlight, etc. Provide high quality. At the same time, when the driver wears a hat, glasses, sunglasses, mask, etc., due to the infrared characteristics, it can penetrate the sunglasses lens, and the blocked eye information can also be imaged normally, which effectively solves the problem of driver's glasses reflecting or wearing sunglasses.

Face detection: The process is divided into face positioning, face recognition and face tracking. The function of face positioning is to detect faces and mark their locations by identifying facial feature points in images; the function of face recognition is to match the facial data detected in new images with stored data; The purpose of tracking is to track on each image frame the face found in the previous image frame.

Head features: It consists of three posture angles. The head tracking system designed based on CNN takes the facial area in the image as input. By combining the detected facial feature points with the default head model, the approximate head posture can be obtained. By further tracking the discovered facial features and finding more features, more data can be obtained to add to the head model, thereby updating the head's geometric properties. While the system is running, this process keeps looping, thus continuously outputting the current posture of the head in the form of a three-dimensional posture angle.

onnx2rknn.py

 
import cv2
import numpy as np
 
from rknn.api import RKNN
import os
 
if __name__ == '__main__':
 
    platform = 'rk3568'
    exp = 'yolov8nseg'
    Width = 640
    Height = 640
    MODEL_PATH = './onnx_models/yolov8n.onnx'
    NEED_BUILD_MODEL = True
    # NEED_BUILD_MODEL = False
    im_file = './bus.jpg'
 
    # Create RKNN object
    rknn = RKNN()
 
    OUT_DIR = "rknn_models"
    RKNN_MODEL_PATH = './{}/{}_kk.rknn'.format(OUT_DIR,exp+'-'+str(Width)+'-'+str(Height))
    if NEED_BUILD_MODEL:
        DATASET = './pose.txt'
        rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]], target_platform="rk3568")
        # Load model
        print('--> Loading model')
        ret = rknn.load_onnx(MODEL_PATH)
        if ret != 0:
            print('load model failed!')
            exit(ret)
        print('done')
 
        # Build model
        print('--> Building model')
        ret = rknn.build(do_quantization=True, dataset=DATASET)
        if ret != 0:
            print('build model failed.')
            exit(ret)
        print('done')
 
        # Export rknn model
        if not os.path.exists(OUT_DIR):
            os.mkdir(OUT_DIR)
        print('--> Export RKNN model: {}'.format(RKNN_MODEL_PATH))
        ret = rknn.export_rknn(RKNN_MODEL_PATH)
        if ret != 0:
            print('Export rknn model failed.')
            exit(ret)
        print('done')
    else:
        ret = rknn.load_rknn(RKNN_MODEL_PATH)
 
    rknn.release()