Opencv+Mediapipe->Face feature point detection

1. First acquaintance

For facial feature point detection in MediaPipe, you can use the pre-trained models and libraries provided by it. MediaPipe provides a model called "FaceMesh" for real-time detection of 468 feature points of human faces.

The following are the basic steps for facial feature point detection using MediaPipe:

1. Install MediaPipe: First, you need to install the MediaPipe framework.

2. Set up input and output: Determine the source of the input data, which can be a camera, video file, or image. At the same time, set the output to store the detected face landmarks.

3. Create a MediaPipe graph: Using the API of the MediaPipe framework, create a graph to load the "FaceMesh" model and process the input data.

4. Run the graph: By passing the input data to the input node of the MediaPipe graph, and obtaining the result through the output node of the graph, run the graph for face feature point detection.

5. Processing results: Obtain the detected face feature point results from the output node, and perform further analysis or application integration on them.

MediaPipe's "FaceMesh" model can automatically locate and identify key feature points in human face images, such as eyes, eyebrows, nose, mouth, etc. We can choose to use specific feature points according to our needs.

In addition, MediaPipe also provides other functions and models, such as hand detection, pose estimation, etc.

2. Face feature point detection

(1) Installation environment

pip install opencv-python
pip install mediapipe==0.8.3.1

(2) OpenCV loading video

code:

import cv2
import mediapipe as mp
import time

cap = cv2.VideoCapture("Video/6.mp4")  # 加载视频
pTime = 0

while True:
    success, img = cap.read()
    imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

    # 帧数
    cTime = time.time()
    fps = 1 / (cTime - pTime)
    pTime = cTime

    cv2.putText(img, f'FPS:{int(fps)}', (20, 70), cv2.FONT_HERSHEY_PLAIN, 3, (0, 255, 0), 3)
    cv2.imshow("Image", img)
    cv2.waitKey(1)

Effect:

Attach a free video material URL: Pexels

 (3) Feature point detection

code:

import cv2
import mediapipe as mp
import time

cap = cv2.VideoCapture("Video/6.mp4")  # 加载视频
pTime = 0

mpDraw = mp.solutions.drawing_utils  # drawing_utils模块：绘制特征点和边界框
mpFaceMesh = mp.solutions.face_mesh
faceMesh = mpFaceMesh.FaceMesh(max_num_faces=2)  # 初始化FaceMesh模块
drawSpec = mpDraw.DrawingSpec(thickness=1, circle_radius=2)  # 绘图样式

while True:
    success, img = cap.read()
    imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    results = faceMesh.process(imgRGB)  # process()：检测人脸关键点
    # 获取关键点信息
    if results.multi_face_landmarks:
        for faceLms in results.multi_face_landmarks:
            mpDraw.draw_landmarks(img, faceLms, mpFaceMesh.FACE_CONNECTIONS, drawSpec, drawSpec)  # 绘制关键点
            for id, lm in enumerate(faceLms.landmark):
                # print(lm)
                ih, iw, ic = img.shape
                # 关键点坐标
                x, y = int(lm.x * iw), int(lm.y * ih)
                print(id, x, y)
    # 帧数
    cTime = time.time()
    fps = 1 / (cTime - pTime)
    pTime = cTime

    cv2.putText(img, f'FPS:{int(fps)}', (20, 70), cv2.FONT_HERSHEY_PLAIN, 3, (0, 255, 0), 3)
    cv2.imshow("Image", img)
    cv2.waitKey(1)

Effect:

3. Application prospect

Face feature point detection has broad application prospects in the fields of computer vision and artificial intelligence. as follows:

1. Face recognition and authentication: Face feature point detection is the basis for face recognition and authentication. By accurately detecting the key feature points of the face, such as eyes, nose, mouth, etc., a unique feature representation of the face can be established. This can be used in secure access control systems, mobile device unlocking, online payment verification, and more.

2. Expression Analysis and Emotion Recognition: Facial feature point detection can help analyze facial expressions and emotional states. By detecting the positions and changes of key feature points such as eyes, eyebrows, and corners of the mouth, facial expressions such as smiles, anger, and sadness can be judged. This has wide-ranging applications in social media analysis, market research, emotion recognition technology, and more.

3. Face beautification and virtual makeup: Face feature point detection can be used for face beautification and virtual makeup applications. By identifying the key feature points of the face, the facial features can be analyzed and modified, such as changing the size of the eyes, adjusting the shape of the mouth, etc., to achieve the beautification effect of the face. This is very popular in areas such as mobile apps, camera software and virtual try-ons.

4. Face transformation and face fusion: Face feature point detection can be used to achieve the effects of face transformation and face fusion. By detecting the key feature points of multiple faces, the facial features of one person can be applied to the image of another person to realize the conversion and fusion of facial expressions, age, gender, etc. This has wide applications in entertainment applications, movie special effects and face changing software.

5. Video analysis and human-computer interaction: Facial landmark detection also plays an important role in the field of video analysis and human-computer interaction. By detecting and tracking the key feature points of the human face in real time, functions such as facial expression recognition, gesture recognition, and head posture tracking can be realized. This has great potential for things like augmented reality, virtual reality, game development, and user experience improvement.