Face recognition - driving fatigue detection (1) Blink detection

Tip: After the article is written, the table of contents can be automatically generated. How to generate it can refer to the help document on the right

Table of contents

foreword

1. Background

(1) Environment construction

(2) Download open source datasets

(3) Principle of visual fatigue detection

2. Code example

3. Effect demonstration

---

foreword

After consulting the literature, the fatigue detection based on the surface features of the face is mainly divided into three parts, yawning, blinking, and nodding. This experiment starts with data such as face orientation, position, pupil orientation, eye opening and closing degree, blink frequency, pupil contraction rate, etc., and uses these data to calculate the driver's concentration in real time, and analyze whether the driver is fatigued or not. Make timely safety reminders.
 

1. Background

(1) Environment construction

The environment configuration used by the subject: python3.9.13+cuda11.3+anaconda3   

Required libraries:

pip install numpy

pip install matplotlib

pip installl imutils

pip install scipy

pip install dlib

Among them, the dlib download method (this article only provides py3.9 version download)

install first

pip install cmake

pip install boost

Download dlib-19.23.0-cp39-cp39-win_amd64.whl

After downloading, execute it in the corresponding folder (everyone should know this (I put it in the backages folder of the environment directory))

pip install dlib-19.23.0-cp39-cp39-win_amd64.whl

其他版本Download in dlib

(2) Download open source datasets

shape_predictor_68_face_landmarks.dat

(3) Principle of visual fatigue detection

Because people probably have two states when they are tired: Blink: the eyes of normal people blink about 10-15 times per minute,

Each blink is about 0.2-0.4 seconds. If you are tired, the number of blinks will increase and the speed will slow down. yawn: mouth at this time

Will grow and will maintain a certain state. Therefore, the detection of human fatigue can be done from the degree of opening and closing of the eyes, the frequency of blinking, and

and the degree of mouth opening and closing to judge whether a person is tired.

Detection Tool

dlib: A very classic open source library for image processing, shape_predictor_68_face_landmarks.dat is a dat model library for 68 key point detection of faces, using this model library can be used for face detection and simple application.

Blink calculation principle:

(1) Calculate the aspect ratio of the eyes

Basic principle: Calculate the Eye Aspect Ratio, EAR. When the human eye is open, the EAR fluctuates up and down at a certain value. When the human eye is closed, the EAR drops rapidly, and it will be close to zero in theory. At that time, the face detection Models are not yet that accurate. So we think that when the EAR is below a certain threshold, the eyes are closed. In order to detect the number of blinks, it is necessary to set the number of consecutive frames of the same blink. The blinking speed is relatively fast, and the blinking action is generally completed in 1 to 3 frames. Both thresholds should be set according to the actual situation.

(2) If the absolute value (EAR) of the difference between the aspect ratio of the two pairs of eyes in the current frame and the previous frame is greater than 0.2, it is considered to be fatigue

(In the 68-point landmark, you can see that 37-42 is the left eye, and 43-48 is the right eye)

The opening and closing degree of the right eye can be obtained by the following formula (the same is true for the left eye):

code ideas

The first step: use dlib.get_frontal_face_detector() to get the face position detector

Step 2: Use dlib.shape_predictor to get facial feature position detector

Step 3: Get the indices of the left and right eye facial landmarks separately

Step 4: Turn on the cv2 local camera

Step 5: Loop from the video stream, read the picture, expand the dimension of the picture, and convert it to grayscale

Step 6: Use detector(gray, 0) for face position detection

Step 7: Loop the face position information, use predictor(gray, rect) to obtain the information of the facial feature position

Step 8: Convert facial feature information into array format

Step 9: Extract left and right eye coordinates

Step 10: The constructor calculates the EAR values ​​for the left and right eyes, using the average as the final EAR

Step 11: Use cv2.convexHull to obtain the convex hull position, and use drawContours to draw the contour position for drawing operations

Step 12: Carry out the drawing operation, mark the face with a rectangular frame

Step 13: Calculate the average of the scores of the left and right eyes as the final score. If it is less than the threshold, add 1. If it is less than the threshold for 3 consecutive times, it means that an eye blinking activity has been performed

Step 14: Carry out the drawing operation, and identify 68 feature points

Step 15: Perform drawing operation, and use cv2.putText to display the number of blinks

Step 16: Count the total number of blinks greater than 50 times and the screen will show that you are asleep.
 

2. Code example

# -*- coding: utf-8 -*-
# import the necessary packages
from scipy.spatial import distance as dist
from imutils.video import FileVideoStream
from imutils.video import VideoStream
from imutils import face_utils
import numpy as np  # 数据处理的库 numpy
import argparse
import imutils
import time
import dlib
import cv2


def eye_aspect_ratio(eye):
    # 垂直眼标志（X，Y）坐标
    A = dist.euclidean(eye[1], eye[5])  # 计算两个集合之间的欧式距离
    B = dist.euclidean(eye[2], eye[4])
    # 计算水平之间的欧几里得距离
    # 水平眼标志（X，Y）坐标
    C = dist.euclidean(eye[0], eye[3])
    # 眼睛长宽比的计算
    ear = (A + B) / (2.0 * C)
    # 返回眼睛的长宽比
    return ear


# 定义两个常数
# 眼睛长宽比
# 闪烁阈值
EYE_AR_THRESH = 0.2
EYE_AR_CONSEC_FRAMES = 3
# 初始化帧计数器和眨眼总数
COUNTER = 0
TOTAL = 0

# 初始化DLIB的人脸检测器（HOG），然后创建面部标志物预测
print("[INFO] loading facial landmark predictor...")
# 第一步：使用dlib.get_frontal_face_detector() 获得脸部位置检测器
detector = dlib.get_frontal_face_detector()
# 第二步：使用dlib.shape_predictor获得脸部特征位置检测器
predictor = dlib.shape_predictor(
    'shape_predictor_68_face_landmarks.dat')

# 第三步：分别获取左右眼面部标志的索引
(lStart, lEnd) = face_utils.FACIAL_LANDMARKS_IDXS["left_eye"]
(rStart, rEnd) = face_utils.FACIAL_LANDMARKS_IDXS["right_eye"]

# 第四步：打开cv2 本地摄像头
cap = cv2.VideoCapture(0)

# 从视频流循环帧
while True:
    # 第五步：进行循环，读取图片，并对图片做维度扩大，并进灰度化
    ret, frame = cap.read()
    frame = imutils.resize(frame, width=720)
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    # 第六步：使用detector(gray, 0) 进行脸部位置检测
    rects = detector(gray, 0)

    # 第七步：循环脸部位置信息，使用predictor(gray, rect)获得脸部特征位置的信息
    for rect in rects:
        shape = predictor(gray, rect)

        # 第八步：将脸部特征信息转换为数组array的格式
        shape = face_utils.shape_to_np(shape)

        # 第九步：提取左眼和右眼坐标
        leftEye = shape[lStart:lEnd]
        rightEye = shape[rStart:rEnd]

        # 第十步：构造函数计算左右眼的EAR值，使用平均值作为最终的EAR
        leftEAR = eye_aspect_ratio(leftEye)
        rightEAR = eye_aspect_ratio(rightEye)
        ear = (leftEAR + rightEAR) / 2.0

        # 第十一步：使用cv2.convexHull获得凸包位置，使用drawContours画出轮廓位置进行画图操作
        leftEyeHull = cv2.convexHull(leftEye)
        rightEyeHull = cv2.convexHull(rightEye)
        cv2.drawContours(frame, [leftEyeHull], -1, (0, 255, 0), 1)
        cv2.drawContours(frame, [rightEyeHull], -1, (0, 255, 0), 1)

        # 第十二步：进行画图操作，用矩形框标注人脸
        left = rect.left()
        top = rect.top()
        right = rect.right()
        bottom = rect.bottom()
        cv2.rectangle(frame, (left, top), (right, bottom), (0, 255, 0), 3)

        '''
            分别计算左眼和右眼的评分求平均作为最终的评分，如果小于阈值，则加1，如果连续3次都小于阈值，则表示进行了一次眨眼活动
        '''
        # 第十三步：循环，满足条件的，眨眼次数+1
        if ear < EYE_AR_THRESH:  # 眼睛长宽比：0.2
            COUNTER += 1

        else:
            # 如果连续3次都小于阈值，则表示进行了一次眨眼活动
            if COUNTER >= EYE_AR_CONSEC_FRAMES:  # 阈值：3
                TOTAL += 1
            # 重置眼帧计数器
            COUNTER = 0

        # 第十四步：进行画图操作，68个特征点标识
        for (x, y) in shape:
            cv2.circle(frame, (x, y), 1, (0, 0, 255), -1)

        # 第十五步：进行画图操作，同时使用cv2.putText将眨眼次数进行显示
        cv2.putText(frame, "Faces: {}".format(len(rects)), (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 255), 2)
        cv2.putText(frame, "Blinks: {}".format(TOTAL), (150, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 255), 2)
        cv2.putText(frame, "COUNTER: {}".format(COUNTER), (300, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 255), 2)
        cv2.putText(frame, "EAR: {:.2f}".format(ear), (450, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0, 0, 255), 2)

    print('眼睛实时长宽比:{:.2f} '.format(ear))
    if TOTAL >= 50:
        cv2.putText(frame, "SLEEP!!!", (200, 200), cv2.FONT_HERSHEY_SIMPLEX, 0.8, (0, 0, 255), 2)
    cv2.putText(frame, "Press 'q': Quit", (20, 500), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (84, 255, 159), 2)
    # 窗口显示 show with opencv
    cv2.imshow("Frame", frame)

    # if the `q` key was pressed, break from the loop
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# 释放摄像头 release camera
cap.release()
# do a bit of cleanup
cv2.destroyAllWindows()

3. Effect demonstration