Drowsiness detection based on key points of the face

01. Research purpose

According to data from the National Highway Traffic Safety Administration, there are more than 1,550 deaths and 71,000 injuries in fatigue driving accidents every year, but the actual number may be much higher [1]. Therefore, in order to avoid such accidents, we made this system. It predicts the signs of the eyes and mouth by checking whether the person's eyes are closed or yawning to determine whether a person is driving fatigued.

main content

02. Main content

The work of the system can be divided into two parts:

1. Detect or locate the face.

2. Predict the landmarks of important areas in the detected face.

Once the result is predicted, we only use eye landmarks and mouth landmarks to determine the person's eye aspect ratio (EAR) and mouth aspect ratio (MAR) to check whether the person is drowsy. The calculation of EAR and MAR is as follows:

Now that we have the code, let us understand how the code works:

The pre-trained facial landmark detector in the dlib library is used to estimate the position of the 68-(x,y) coordinate mapped to the facial structure [2]. These 68-(x,y) coordinates represent important areas of the face, such as the mouth, left eyebrow, right eyebrow, left eye, right eye, nose, and chin. Among them, we only need the (x, y) coordinates of the left eye, right eye and mouth:

Top left: Visualization of eye landmarks when the eyes are open. Upper right: The eye landmark when the eyes are closed. Bottom: Plot the eye aspect ratio over time. A decrease in the aspect ratio of the eye indicates blinking [3] (Figure 1 by Soukupová and Čech). [4]

Based on the paper Real-Time Eye Blink Detection using Facial Landmarks [5], we can derive an equation reflecting this relationship, called the eye aspect ratio (EAR):

Eye aspect ratio (EAR) formula.

Using this concept, we calculated the mouth aspect ratio:

Use 68-(x,y) coordinates to represent the face

As we can see, the mouth is represented by a set of 20-(x,y) coordinates. Therefore, we have used the coordinates 62, 64, 66, and 68 to calculate the distance between the two in the same way as the EAR calculation.

result

Alarm when a person is drowsy or yawning

In addition, in order to preserve the evidence, we saved the drowsy frame.

The frame is stored in a separate folder as proof

The graph shows the change of EAR and MAR over time

The GitHub link of the source code is available here: https://github.com/fear-the-lord/Drowsiness-Detection

03. References

[1]Drivers are falling asleep behind the wheels. Prevalence of drowsy driving crashes: https://www.nsc.org/road-safety/safety-topics/fatigued-driving

[2]Facial landmarks with dlib, OpenCV and Python: https://www.pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/

[3]Eye blink detection with OpenCV, Python, and dlib: https://www.pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/

[4]Drowsiness Detection with OpenCV: https://www.pyimagesearch.com/2017/05/08/drowsiness-detection-opencv/

[5]Real-Time Eye Blink Detection using Facial Landmarks: http://vision.fe.uni-lj.si/cvww2016/proceedings/papers/05.pdf

Now, each eye is represented by a set of 6-(x,y) coordinates that start from the upper left corner of the eye (as if you are looking at the person) and then rotate clockwise around the rest of the area [3 ]. :

from scipy.spatial import distance as dist
def eye_aspect_ratio(eye):
# Vertical eye landmarks
A = dist.euclidean(eye[1], eye[5])
B = dist.euclidean(eye[2], eye[4])
# Horizontal eye landmarks
C = dist.euclidean(eye[0], eye[3])
# The EAR Equation
EAR = (A + B) / (2.0 * C)
return EAR

01. Research purpose

According to data from the National Highway Traffic Safety Administration, there are more than 1,550 deaths and 71,000 injuries in fatigue driving accidents every year, but the actual number may be much higher [1]. Therefore, in order to avoid such accidents, we made this system. It predicts the signs of the eyes and mouth by checking whether the person's eyes are closed or yawning to determine whether a person is driving fatigued.

main content

02. Main content

The work of the system can be divided into two parts:

1. Detect or locate the face.

2. Predict the landmarks of important areas in the detected face.

Once the result is predicted, we only use eye landmarks and mouth landmarks to determine the person's eye aspect ratio (EAR) and mouth aspect ratio (MAR) to check whether the person is drowsy. The calculation of EAR and MAR is as follows:
```
 
```
from scipy.spatial import distance as dist
def eye_aspect_ratio(eye):
# Vertical eye landmarks
A = dist.euclidean(eye[1], eye[5])
B = dist.euclidean(eye[2], eye[4])
# Horizontal eye landmarks
C = dist.euclidean(eye[0], eye[3])
# The EAR Equation
EAR = (A + B) / (2.0 * C)
return EAR
def mouth_aspect_ratio(mouth):
A = dist.euclidean(mouth[13], mouth[19])
B = dist.euclidean(mouth[14], mouth[18])
C = dist.euclidean(mouth[15], mouth[17])
MAR = (A + B + C) / 3.0
return MAR
# Grab the indexes of the facial landamarks for the left and right eye respectively
(lstart, lend) = face_utils.FACIAL_LANDMARKS_IDXS["left_eye"]
(rstart, rend) = face_utils.FACIAL_LANDMARKS_IDXS["right_eye"]
(mstart, mend) = face_utils.FACIAL_LANDMARKS_IDXS["mouth"]
def mouth_aspect_ratio(mouth):
A = dist.euclidean(mouth[13], mouth[19])
B = dist.euclidean(mouth[14], mouth[18])
C = dist.euclidean(mouth[15], mouth[17])
MAR = (A + B + C) / 3.0
return MAR