Table of contents
foreword
This project uses the functions of machine learning, numerical calculation, graph model algorithm and image processing provided by Dlib, based on the pre-training model of face recognition, aiming to realize the function of changing faces in photos.
Dlib is a powerful open source library that provides a variety of machine learning and computer vision algorithms. In this project, we will use the face recognition pre-trained model in Dlib to identify and locate the face area in the photo.
By using the face recognition algorithm, we can obtain the face feature points and facial contours in the two photos. Then, using these feature points and contour information, we can perform face alignment and registration operations.
During the registration process, we will match facial landmarks and facial contours in one photo to corresponding regions in another photo. By aligning the facial feature points of the two photos, we can achieve the face-swapping effect of the two photos.
The application of this project is very interesting and creative. It can be used in entertainment, art creation, virtual reality and other fields, providing users with a fun and imaginative space.
This project is for personal learning use only, please do not apply it to any commercial or bad fields.
overall design
This part includes the overall structure diagram of the system and the system flow chart.
System overall structure diagram
The overall structure of the system is shown in the figure.
System flow chart
The system flow is shown in the figure.
operating environment
This section includes the Python environment and the installation of related library packages.
Python environment
Python 3.6 and above configuration is required, and Anaconda is downloaded in the Windows environment to complete the configuration of the environment required by Python.
Related library package installation
The library files required to complete the project include OpenCV, dlib, numpy, sys, PIL, thikter, matplotlib.
dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software that solves real-world problems in C++.
OpenCV is a cross-platform library released under the BSD license (open source) and can run on Linux, Windows, Android and Mac OS operating systems. It is lightweight and efficient-consisting of a series of C functions and a small number of C++ classes, and provides language interfaces such as Python, Ruby, and MATLAB to implement many general algorithms in image processing and computer vision.
Among them, the Dlib library file needs to be downloaded from https://pypi.org/project/dlib/19.1.0/#files , and installed by using the following command in the command prompt:dlib-19.1.0-cp35-cp35m-win_amd64.whl
pip install dlib-19.1.0-cp35-cp35m-win_amd64.whl
For other required library files, just use pip install in the command prompt.
module implementation
This project includes 7 modules: preparing data, extracting facial markers, adjusting face alignment, blending images, correcting colors, conversion functions, and interactive interface design. The function introduction and related codes of each module are given below.
1. Prepare data
dlib.get_frontal_face_detector()It is a face detector that detects whether there is a face in the picture and returns a rectangular list of face detectors.
The dlib.shape_predictor(PREDICTOR_PATH) feature extractor takes the bounding box provided by the face detector as input to the algorithm and returns a face keypoint predictor.
Using the official pre-training model to help developers save time, download and build a face feature extractor composed of 68 feature points from the Dlib sourceforge library: http://sourceforge.net/projects/dclib/files/dlib/v18 .10/shape_predictor_68_face_landmarks.dat.bz2
The relevant code is as follows:
PREDICTOR_PATH = "D:/HuanLian/model/shape_predictor_68_face_landmarks.dat"
FACE_POINTS = list(range(17, 68))
MOUTH_POINTS = list(range(48, 61))
RIGHT_BROW_POINTS = list(range(17, 22))
LEFT_BROW_POINTS = list(range(22, 27))
RIGHT_EYE_POINTS = list(range(36, 42))
LEFT_EYE_POINTS = list(range(42, 48))
NOSE_POINTS = list(range(27, 35))
JAW_POINTS = list(range(0, 17))
#划分68个点的每个点意味着什么部位，如第27~35的点(xi,yi)就是鼻子
ALIGN_POINTS = (LEFT_BROW_POINTS + RIGHT_EYE_POINTS + LEFT_EYE_POINTS + RIGHT_BROW_POINTS + NOSE_POINTS + MOUTH_POINTS)
#对齐图片的点，即五官
OVERLAY_POINTS = [LEFT_EYE_POINTS + RIGHT_EYE_POINTS + LEFT_BROW_POINTS + RIGHT_BROW_POINTS,NOSE_POINTS + MOUTH_POINTS,]
detector = dlib.get_frontal_face_detector()
#人脸检测器
predictor = dlib.shape_predictor(PREDICTOR_PATH)
#特征提取器
2. Extract facial markers
After pre-training to obtain the feature extractor, the user inputs the face feature points of two images. The function converts one image into a numpy array and returns a 68*2 element matrix. The input image has 68 feature points for a face, and each feature The points correspond to the x,y coordinates of each row.
class TooManyFaces(Exception):#设置检测到太多脸的类
pass
class NoFaces(Exception):#设置没有检测到脸的类
Pass
def get_landmarks(im):#获取人脸特征点，将图像转化成numpy数组，返回68*2元素矩阵
#输入图像的每个特征点对应每行的x、y坐标
rects = detector(im, 1)#每个矩形列表在图像中对应一个脸
#rects表示人脸框的位置信息
if len(rects) > 1: #如果识别的人脸数大于一个，引发TooManyFaces异常
raise TooManyFaces
if len(rects) == 0:#如果图片没人脸，引发NoFaces异常
raise NoFaces
return numpy.matrix([[p.x, p.y] for p in predictor(im, rects[0]).parts()])
#为加快计算，把得到的特征点转换成numpy矩阵
def read_im_and_landmarks(fname):#从计算机中读取用户所选的图片并提取特征点
im = cv2.imread(fname, cv2.IMREAD_COLOR)#opencv读取图片并显示
im = cv2.resize(im, (im.shape[1] * SCALE_FACTOR,
im.shape[0] * SCALE_FACTOR))
s = get_landmarks(im)
return im, s
3. Adjust face alignment
The Procrustes analysis method is to normalize the two shapes. Mathematically speaking, Platts analysis is to use the least square method to find the affine transformation from shape A to shape B.
Two marker matrices have now been obtained, with each row having a set of coordinates corresponding to a particular facial feature. The next step is to figure out how to rotate, translate and scale the first vector so that they fit as closely as possible in the second vector's points. The 2nd image can be overlaid on the 1st image with the same transformation.
Finding T, s and R, the result of the expression (1) is minimized:
E = min ∑ i = 1 68 ∥ s R p i t + T − q i T ∥ 2 E=\min \sum_{i=1}^{68} \| s Rp_i^t +T-q_i^T \|^2 E=mini=1∑68∥sRpit+T−qiT∥2
The meanings of the parameters are as follows:
s: scaling factor.
R: Rotation matrices (matrixes that change direction when multiplying a vector, but do not change magnitude and maintain chirality).
pit p_i^tpit: Extract the coordinate vectors of 68 points from the picture to be matched.
T: translation vector.
qit q_i^tqit:: The reference image extracts the coordinate vectors of 68 points.
The relevant code is as follows:
def transformation_from_points(points1, points2):
#普氏分析(Procrustes analysis)调整脸部，相同变换使两张照片面部特征相对距离尽可能小
#输入的自变量是两张图片特征点矩阵
points1 = points1.astype(numpy.float64) #将输入矩阵转化为浮点数
points2 = points2.astype(numpy.float64)
c1 = numpy.mean(points1, axis=0) #按列求均值，即求样本点在图像中的均值
c2 = numpy.mean(points2, axis=0)
points1 -= c1
#对所有形状的大小进行归一化，将每个样本点减去对应均值，处理消除平移T的影响
points2 -= c2
s1 = numpy.std(points1)#求出每个点的标准差
s2 = numpy.std(points2)
points1 /= s1
points2 /= s2 #每一个点集除以它的标准偏差，这一步处理消除缩放系数s的影响
U, S, Vt = numpy.linalg.svd(points1.T * points2) #可以求解出R
R = (U * Vt).T
return numpy.vstack([numpy.hstack(((s2 / s1) * R, c2.T - (s2 / s1) * R * c1.T)), numpy.matrix([0., 0., 1.])])
The essence is to scale, rotate, and translate the first picture so that the first and second pictures overlap as much as possible to obtain an affine transformation matrix. Plug in OpenCV's cv2.warpAffine function to map image 2 to image 1.
4. Blend images
Use a mask to represent different areas, the pixel value of the area belonging to the face is 1, and the value of the pixel of the area not belonging to the face is 0. When extracting, directly multiply the original image by the mask to get the face, and the pixel value of the rest area is 0; if the original image is multiplied by 1−mask, the face area is 0, and the rest area is reserved. The above two results are added together to achieve a preliminary face change.
1) convex hull
During processing, it is necessary to find the convex hull surrounding an object in the image. A convex hull is similar to a polygon. A convex set that surrounds the outermost layer of an object is the intersection of all convex sets that can surround this object. As shown in the figure, the convex set enclosed by the circle lines is the convex hull.
2) Realization of convex hull
In OpenCV, the convex hull of a series of points is obtained through the function convexHull. For example, for a contour composed of points, the convex hull of the contour can be obtained by the convexHull function:
def draw_convex_hull(im, points, color):#绘制凸包
points = cv2.convexHull(points)
#寻找图像的凸包，points就是输入的一组点
cv2.fillConvexPoly(im, points, color=color)
#cv2.fillConvexPoly()函数可以填充凸多边形,由凸包得到的轮廓点作为顶点
3) Mask implementation
def get_face_mask(im, landmarks):#为一张图像和一个标记矩阵生成一个遮罩
#画出了两个凸多边形：一个是眼睛周围的区域，一个是鼻子和嘴部周围的区域
im = numpy.zeros(im.shape[:2], dtype=numpy.float64)
#numpy.zeros返回给定形状和类型的新数组，用0填充
#img.shape[:2] 取彩色图片的高、宽
for group in OVERLAY_POINTS:
#OVERLAY_POINTS 定义为[LEFT_EYE_POINTS + RIGHT_EYE_POINTS + LEFT_BROW_POINTS + RIGHT_BROW_POINTS , NOSE_POINTS + MOUTH_POINTS,]
#分为[眼睛周围、鼻子和嘴]两个区域，是第二张图片中要覆盖第一张图片的点
draw_convex_hull(im,landmarks[group],color=1)
#对图片中68特征点集里是OVERLAY_POINTS的点进行凸包绘制
im = numpy.array([im, im, im]).transpose((1, 2, 0))
im = (cv2.GaussianBlur(im, (FEATHER_AMOUNT, FEATHER_AMOUNT),0)>0)*1.0
#高斯滤波
im = cv2.GaussianBlur(im, (FEATHER_AMOUNT, FEATHER_AMOUNT), 0)
return im
4) Image combination
A mask is generated for both images at the same time, so that the mask of image 2 can be transformed into the coordinate space of image 1. The idea of combining the two masks into one is to ensure that image 1 is masked while revealing the properties of image 2.
def warp_im(im, M, dshape):#得到了转换矩阵M后，使用它进行映射，OpenCV的cv2.warpAffine函数，将图像2映射到图像1
output_im = numpy.zeros(dshape, dtype=im.dtype)
#返回给定形状和类型的新数组，用0填充
#将图像二映射到图像一
cv2.warpAffine(im, #输入图像
M[:2],#仿射矩阵
(dshape[1], dshape[0]),
dst=output_im,#输出图像
borderMode=cv2.BORDER_TRANSPARENT,#边缘像素模式
flags=cv2.WARP_INVERSE_MAP)#插值方法的组合
return output_im
5. Correct color
Due to the influence of factors such as background lighting or skin color of the two images, the effect of direct fusion of images is not ideal. The idea is to use Gaussian Blur to help correct the color. Specific operation: Use a template to scan each pixel in the image, and use the weighted average gray value of the pixels in the neighborhood determined by the template to replace the value of the pixel in the center of the template.
1) Gaussian blur
For the part with high pixel energy, the weighted average method is used to recalculate the pixel value and become a value with lower energy. For the image, after the high-frequency part passes through the low-pass filter, the entire image becomes low-frequency, causing the image to be blurred, which is called Gaussian blur.
2) Import pictures and specific implementation
OpenCV provides the GaussianBlur() function to perform Gaussian filtering on graphics.
def correct_colours(im1, im2, landmarks1):
#修正两幅图像之间不同肤色和光线造成的覆盖区域边缘不连续
blur_amount = COLOUR_CORRECT_BLUR_FRAC * numpy.linalg.norm(
numpy.mean(landmarks1[LEFT_EYE_POINTS], axis=0) -
numpy.mean(landmarks1[RIGHT_EYE_POINTS],axis=0))
#numpy.mean求左右眼点集均值，其中axis=0，压缩行，对各列求均值，返回1*n矩阵
#从左眼点集、右眼点集分别得到一个代表左眼和右眼的点，两者相减是左右眼横，纵相对距离
#用numpy.linalg.norm得到矩阵所有元素平方和开根号，勾股定理，得到了两眼之间的距离
#COLOUR_CORRECT_BLUR_FRAC*两眼距离作为高斯内核大小
#内核太小，第一个图像的面部特征将显示在第二个图像中
#内核过大，内核之外区域像素被覆盖并发生变色，COLOUR_CORRECT_BLUR_FRAC设置为0.6
blur_amount = int(blur_amount)
if blur_amount % 2 == 0:
blur_amount += 1 #高斯内核大小不能是偶数
im1_blur = cv2.GaussianBlur(im1, (blur_amount, blur_amount), 0)
#用模板扫描图像中的每一个像素，确定的邻域内像素的加权平均灰度值去替代模板中心像素点的值
#高斯矩阵的长与宽是高斯内核（blur_amount）的大小，标准差取0，返回高斯滤波后的图像
im2_blur = cv2.GaussianBlur(im2, (blur_amount, blur_amount), 0)
im2_blur += (128 * (im2_blur <= 1.0)).astype(im2_blur.dtype)
#防止除零，将高斯滤波后im2元素的数据类型返回，强制类型转换成数据类型
return (im2.astype(numpy.float64) * im1_blur.astype(numpy.float64) /
im2_blur.astype(numpy.float64))
#试图改变图像2的颜色来匹配图像1，通过用im2*im1/im2的高斯模糊
6. Conversion function
The functions defined above are the conversion functions, and the relevant codes are as follows:
def main():
global image1,image2#将两张照片设置为全局变量，方便在各个函数中直接使用
im1, landmarks1 = read_im_and_landmarks(image1)#提取图片1的特征点
im2, landmarks2 = read_im_and_landmarks(image2)#提取图片2的特征点
M = transformation_from_points(landmarks1[ALIGN_POINTS],
landmarks2[ALIGN_POINTS])
#普氏分析(Procrustes analysis)调整脸部，相同变换使两张照片面部特征的相对距离尽可能小
mask = get_face_mask(im2, landmarks2)#为一张图像和一个标记矩阵生成一个遮罩
warped_mask = warp_im(mask, M, im1.shape)
#把图像2遮罩通过仿射矩阵M映射到图像1上
combined_mask = numpy.max([get_face_mask(im1, landmarks1), warped_mask], axis=0)
warped_im2 = warp_im(im2, M, im1.shape)#把图像2映射到遮好的图像1上
warped_corrected_im2 = correct_colours(im1, warped_im2, landmarks1)#校正颜色
output_im = im1 * (1.0 - combined_mask) + warped_corrected_im2 * combined_mask
cv2.imwrite('output.jpg', output_im) #输出换脸后的照片
7. Interactive interface design
#从计算机中选择原图1并展示
def show_original1_pic():
global image1
image1 = askopenfilename(title='选择文件') #将选择的图片作为待换脸图片
print(image1)
Img = PIL.Image.open(r'{}'.format(image1))
Img = Img.resize((270,270),PIL.Image.ANTIALIAS)
img_png_original = ImageTk.PhotoImage(Img)
label_Img_original1.config(image=img_png_original)
label_Img_original1.image = img_png_original #保持参考点
cv_orinial1.create_image(5, 5,anchor='nw', image=img_png_original)
#从计算机中选择原图2并展示
def show_original2_pic():
global image2
image2 = askopenfilename(title='选择文件')#将选择的图片作为待换脸图片
print(image2)
Img = PIL.Image.open(r'{}'.format(image2))
Img = Img.resize((270,270),PIL.Image.ANTIALIAS)
img_png_original = ImageTk.PhotoImage(Img)
label_Img_original2.config(image=img_png_original)
label_Img_original2.image = img_png_original #keep a reference
cv_orinial2.create_image(5, 5,anchor='nw', image=img_png_original)
#设置按钮，打开图片1并将其作为待替换图片
Button(root, text = "打开图片1", command = show_original1_pic).place(x=50,y=120)
#设置按钮，打开图片2并将其作为待替换图片
Button(root, text = "打开图片2", command = show_original2_pic).place(x=50,y=200)
#进行换脸
Button(root, text = "换脸！！", command = main).place(x=50,y=280)
#配置界面的一些细节，如边框、线条等
Label(root,text = "图片1",font=10).place(x=280,y=120)
cv_orinial1 = Canvas(root,bg = 'white',width=270,height=270)
cv_orinial1.create_rectangle(8,8,260,260,width=1,outline='red')
cv_orinial1.place(x=180,y=150)
label_Img_original1 = Label(root)
label_Img_original1.place(x=180,y=150)
Label(root,text="图片2",font=10).place(x=600,y=120)
cv_orinial2 = Canvas(root,bg = 'white',width=270,height=270)
cv_orinial2.create_rectangle(8,8,260,260,width=1,outline='red')
cv_orinial2.place(x=500,y=150)
label_Img_original2 = Label(root)
label_Img_original2.place(x=500,y=150)
root.mainloop()
System test
The application interface is shown in the figure. There is a column of buttons on the left, the first two buttons are to open picture 1. The first step is to click to enter the browsing page and browse the files in the computer by yourself; the second step is to select the face-changing photo; the third step is to display the photo you just selected in the two boxes on the right under picture 1. In the same way, the function of opening picture 2 is the same as that of opening picture 1; the fourth step, click the face change button at the bottom left to change the face. After clicking the face change, a face-changed picture will be generated in the current file directory.
Project source code download
See my blog resource download page for details
Other information download
If you want to continue to learn about artificial intelligence-related learning routes and knowledge systems, welcome to read my other blog " Heavy | Complete artificial intelligence AI learning-basic knowledge learning route, all materials can be downloaded directly from the network disk without paying attention to routines "
This blog refers to Github's well-known open source platform, AI technology platform and experts in related fields: Datawhale, ApacheCN, AI Youdao and Dr. Huang Haiguang, etc. There are about 100G related materials, and I hope to help all friends.