Preface
AI face-changing refers to the use of deep learning and computer vision to replace or synthesize faces in images or videos. You can replace one person's face with that of another, or composite one person's expression into another person's photo or video. Algorithms are often used for entertainment purposes, such as creating interesting photos or videos on social media, but are also used in film production, special effects production, facial editing tools, and virtual reality. But it can also be misused for deception, dissemination of false information or invasion of privacy.
With the widespread application of AI face-changing technology, this has also caused a lot of attention and worry, because it can be used to create false video content, which may lead to social and political problems. AI face-changing technology will also raise legal and ethical issues, including privacy issues and identity verification issues. Misuse of these technologies can result in damage to an individual's reputation and can also be used to deceive and defraud.
While AI face-changing technology continues to develop and become more advanced, researchers and technology companies are also working hard to develop methods to detect and defend against AI face-changing to deal with abuse and the spread of false information.
Here we combine some commonly used AI face-changing technologies, from face detection to facial key point detection, to AI face-changing, and then use algorithms to perform face repair and super-scoring, so that everyone can better understand the AI face-changing technology. Intelligent algorithms can only be fully understood to achieve better prevention.
1. AI face-changing preprocessing
1. Face detection
To do face-related processing, you first need to obtain the face and coordinates in the image or video, which will be used in face detection-related algorithms. The face detection algorithm provides a rectangular frame for the face position
. The following are some commonly used face detection algorithms:
- Viola-Jones algorithm : The Viola-Jones algorithm is a classic face detection algorithm based on Haar features and cascade classifiers. It has fast detection speed and is usually used in real-time applications.
- Convolutional Neural Network (CNN) : Convolutional neural networks in deep learning have made significant progress in face detection. Many modern face detectors, such as MTCNN (Multi-Task Convolutional Neural Network) and SSD (Single-Shot Multiple Frame Detector), use CNN to detect faces.
- HOG Features and Support Vector Machine (SVM) : HOG (Histogram of Oriented Gradients) features are often used in conjunction with SVM classifiers for face detection. This method performs well when dealing with faces under different lighting and scale conditions.
- Deep learning model : Some face detectors based on deep learning, such as YOLO (You Only Look Once) and Faster R-CNN (Regional Convolutional Neural Network), have been widely used in the field of target detection, including face detection.
- Cascade classifier : Cascade classifier is a method of combining multiple weak classifiers to improve the accuracy of face detection. This method is adopted by the Viola-Jones algorithm, as well as some other AdaBoost-based methods.
- 3D face detection : In addition to 2D face detection, there are also algorithms for detecting and recognizing 3D faces, which often use depth sensors or multi-view cameras.
Usually contour point detection is also called Face Alignment. The AI face-changing implementation in this article is based on 68 facial contour points (Landmarks). Face Alignment must rely on a face detector, namely Face Detection. In other words, the basis of face swapping or other refined face operations introduced in this article is Face Detection + Face Alignment. However, these two technologies do not require us to develop them ourselves. dlib already provides third-party libraries that are effective and easy to use.
2. Key feature points of human face
Key feature points (face-landmarks) in face images, such as eyes, nose, mouth, cheeks, etc. These feature points are commonly used in facial recognition, expression analysis, facial pose estimation and other applications. The following are some commonly used facial key point detection algorithms:
Dlib : Dlib is a popular C++ library that provides efficient facial key point detection functions. It uses a cascade classifier based on HOG features and uses regression methods to predict key point locations. Dlib also provides Python bindings, so it can be easily used in Python environments.
OpenCV : OpenCV is a widely used computer vision library that includes several functions for facial keypoint detection. OpenCV's facial landmark detectors are typically based on cascaded classifiers and shape models.
Face++ : Face++ is a commercial face recognition platform that provides facial key point detection services. It uses deep learning techniques, including convolutional neural networks (CNN), to detect and locate facial keypoints.
dlib + pre-trained model : In addition to the Dlib library itself, you can also use pre-trained deep learning models, such as models based on TensorFlow or PyTorch, for facial key point detection. These models are typically trained on large-scale datasets and are able to detect facial keypoints more accurately.
MediaPipe Face Detection : Google's MediaPipe framework provides a solution for real-time facial keypoint detection. It uses a deep learning model to detect faces and estimate keypoint locations, which can be used in real-time applications and deployed on mobile devices.
68-Point Facial Landmarks Model : This is a common facial key point detection model. It can detect 68 key points of the face, including eyes, mouth, nose, cheeks and other parts. Such models are often built using deep learning methods.
2. InsightFace algorithm
1. Introduction to algorithm
The InsightFace algorithm is a deep learning model for face analysis and recognition tasks. It mainly focuses on face recognition and face verification. InsightFace is an integrated Python library for 2D and 3D face analysis. InsightFace efficiently implements a variety of state-of-the-art face recognition, face detection, and face alignment algorithms and is optimized for training and deployment. It supports a range of backbone architectures, including IResNet, RetinaNet, MobileFaceNet, InceptionResNet_v2 and DenseNet. In addition to models, it can use facial datasets such as MS1M, VGG2, and CASIA-WebFace. Basic analysis of the InsightFace algorithm:
Backbone Networks : InsightFace uses a deep convolutional neural network (CNN) as the backbone network to extract facial features from the input image. Typically, network structures such as ResNet and MobileNet are used as backbone networks, but InsightFace also supports other network structures.
ArcFace Loss : A distinctive feature of InsightFace is the use of the ArcFace loss function during the training process. The ArcFace loss function is designed to enhance the discriminative ability of feature embeddings, making it suitable for face recognition tasks. It does this by introducing margins between categories in the embedding space to better distinguish different identities.
Efficient Training : InsightFace is known for its efficient training process, including large-scale face recognition and data enhancement technologies. These methods help improve model performance, especially when dealing with large-scale data sets.
Pretrained Models : InsightFace provides pretrained models, allowing developers to easily start face recognition tasks without having to train the model from scratch. These pre-trained models are trained on large-scale face datasets and can be fine-tuned for specific applications.
Open Source : InsightFace is an open source framework, which means developers can access its code base and customize it to their needs. This has resulted in a thriving research and development community that continuously contributes to its development.
Various Applications : InsightFace can be used not only for basic face recognition, but also for a wider range of tasks, such as sentiment analysis, age estimation, gender classification and facial attribute recognition.
2. Algorithm advantages
InsightFace has many advantages that make it popular for face recognition and related tasks. The following is a summary of InsightFace’s main benefits:
High accuracy : InsightFace is known for its powerful deep learning model and ArcFace loss function, which enables it to achieve high-precision recognition results in face recognition and face verification tasks.
Efficiency : InsightFace employs effective training techniques, including large-scale face recognition and data augmentation, which help improve model performance and reduce training time.
Pre-trained models : InsightFace provides pre-trained models, allowing developers to quickly start face recognition projects without having to train the model from scratch. This improves development efficiency.
Flexibility : InsightFace is an open source framework that developers can customize according to their needs. This means it can be adapted to a variety of face analysis tasks and application areas.
Multi-purpose : In addition to basic face recognition and verification, InsightFace can also be used for a variety of other applications, such as sentiment analysis, age estimation, gender classification, and facial attribute recognition.
Large-scale support : InsightFace has the ability to process large-scale data sets, so it is suitable for projects that require large amounts of training data, such as the application of face recognition in large identity databases.
Community support : Because it is an open source project, InsightFace has an active community that continuously contributes code, documentation, and improvements to keep it active and updated.
Emphasis on privacy and security : InsightFace users can enforce security and privacy measures themselves to ensure compliance and data protection.
3. Algorithm Overview
InsightFace provides training data, network settings and loss design for deep recognition. The training data includes the standardized MS1M, VGG2 and CASIA-Webface datasets, which have been packaged in MXNet binary format. The network backbone includes ResNet, MobilefaceNet, MobileNet, InceptionResNet_v2, DenseNet, and DPN. Loss functions include Softmax, SphereFace, CosineFace, ArcFace and Triplet (Euclidean/Angular) Loss.
InsightFace's approach to ArcFace was originally described in an arXiv technical report. By using this repository, LFW 99.80%+ and Megaface 98%+ can be easily implemented with a single model.
3. Use InsightFace to implement face attribute analysis
import os
import cv2
import insightface
from sklearn import preprocessing
import numpy as np
import onnxruntime
from sklearn.metrics.pairwise import cosine_similarity, paired_distances
import gradio as gr
class DisposeFace:
def __init__(self,face_db,gpu_id=0, threshold=1.24,
det_thresh=0.50, det_size=(640, 640)):
self.gpu_id = gpu_id
self.threshold = threshold
self.det_thresh = det_thresh
self.det_size = det_size
self.face_db = face_db
# 初始化模型
providers = onnxruntime.get_available_providers()
self.face_model = insightface.app.FaceAnalysis(
name="buffalo_l", root="checkpoints", allowed_modules=None, providers=providers)
self.face_model.prepare(ctx_id=self.gpu_id, det_thresh=self.det_thresh, det_size=self.det_size)
# 人脸库的人脸特征
self.faces_embedding = list()
# # 加载人脸库中的人脸
self.load_db_faces(self.face_db)
def load_model(self):
return self.face_model
def load_db_faces(self,face_db_path):
if not os.path.exists(face_db_path):
os.makedirs(face_db_path) #创建人脸数据库
for root,dirs,files in os.walk(face_db_path):
for file in files:
input_image = cv2.imdecode(np.fromfile(os.path.join(root, file), dtype=np.uint8), 1)
user_name = file.split(".")[0]
face = self.face_model.get(input_image)[0]
embedding = np.array(face.embedding).reshape((1, -1))
embedding = preprocessing.normalize(embedding)
self.faces_embedding.append({
"user_name": user_name,"feature": embedding})
def detect_face(self,cv_src):
faces = self.face_model.get(cv_src)
faces_results = list()
for face in faces:
result = dict()
#获取人脸属性
result["bbox"] = np.array(face.bbox).astype(np.int32).tolist()
result["kps"] = np.array(face.kps).astype(np.int32).tolist()
result["landmark_3d_68"] = np.array(face.landmark_3d_68).astype(np.int32).tolist()
result["landmark_2d_106"] = np.array(face.landmark_2d_106).astype(np.int32).tolist()
result["pose"] = np.array(face.pose).astype(np.int32).tolist()
result["age"] = face.age
gender = 'man'
if face.gender == 0:
gender = 'female'
result["gender"] = gender
#人脸识别
embedding = np.array(face.embedding).reshape((1, -1))
embedding = preprocessing.normalize(embedding)
result["embedding"] = embedding
faces_results.append(result)
return faces_results
@staticmethod
def feature_compare(feature1, feature2, threshold):
diff = np.subtract(feature1, feature2)
dist = np.sum(np.square(diff), 1)
if dist < threshold:
return True
else:
return False
def face_recognition(self,cv_src):
faces = self.face_model.get(cv_src)
face_results = list()
for face in faces:
# 开始人脸识别
embedding = np.array(face.embedding).reshape((1, -1))
embedding = preprocessing.normalize(embedding)
user_name = "unknown"
for com_face in self.faces_embedding:
result = self.feature_compare(embedding,com_face["feature"],self.threshold)
if result:
user_name = com_face["user_name"]
face_results.append(user_name)
return face_results
def recognition(self,cv_face1,cv_face2):
faces1 = self.face_model.get(cv_face1)
faces2 = self.face_model.get(cv_face2)
if len(faces1) != 1 and len(faces2) != 1:
return "No face detected or multiple faces detected!"
embedding1 = np.array(faces1[0].embedding).reshape