Before starting, take a look at the renderings (declare in advance: the pictures are from the Internet), because the face contrast is a bit big, so it’s a bit spicy.
Swap the roles in the picture, and then look at the effect of face replacement after conversion:
Emm, how do you say the result, the effect is still good (the replacement seam produced will not be so distorted and abrupt), but it feels that the new face is deformed.
1. Introduction to Face Swap technology
Well, the following will introduce the face replacement technology in detail and implement it with OpenCV; so far, if you haven’t read the previous articles to
achieve face recognition and 68 feature point extraction of faces, perhaps this Python library can help your turn!
Using OpenCV-Python to perform face Delaunay triangulation (one of the core face detection technologies)
recommends that you understand it in advance, because this article will use facial feature point extraction and Delaunay triangulation.
Face replacement technology is relatively difficult. One of the reasons is that there is a large difference in face shape between people (texture, face shape, regional unevenness, etc.), which makes the face area and the surrounding skin tissue different after replacement. Distorted, unnatural visual effects,
As for the final face replacement images given in the following two pictures, the main reasons why the results are very spicy are as follows:
1. Age differences, facial wrinkles and texture areas are quite different;
2. Ethnic differences; one is Asian yellow race, the other is white, the degree of sunken eyes and facial structure are quite different;
3. Gender differences; there will still be slight differences between the face shapes of men and women;
2. Face replacement technology needs to solve difficulties
1. The size of the face area is inconsistent. For example, the size of a fatter person and a thinner person are obviously different. The direct replacement operation is obviously inappropriate. We need to unify the face shape in advance to have subsequent operations;
2. After the face is replaced, there will be obvious color differences between the replaced face area and the surrounding skin tissue. Problems such as lighting make the replacement gap more abrupt. As shown in the figure below, if this problem is not resolved, the final processed image will be very different. "false";
3. There is a problem with the camera angle of the face. Some pictures show the front face and some show the profile face;
4. For the final skin problem, the texture of the replaced face area needs to be kept consistent with the surrounding tissues, and the final change is not much different;
Due to limited technology, this article uses OpenCV to solve only problems 1 and 2, and the solutions to problems 3 and 4 can continue to dig the following.
3. OpenCV realizes face recognition;
1. Feature point extraction, find Convexhull
Use the dlib program package to extract facial feature points, and outline the facial area according to the feature points **find Convexhull (convex hull)** (here only the contour of the face needs to be outlined, no feature points in the middle of the face are needed) ), please refer to the extraction of facial feature points:
realize face recognition, 68 facial feature points extraction, maybe this Python library can help you!
2. Delaunay triangulation
Use the Convexhull (convex hull) calculated in step 1 to perform Delaunay triangulation. The result is as shown in the figure below. For the specific operation of triangulation, please refer to:
Use OpenCV-Python to perform face Delaunay triangulation (one of the core technologies of face detection) )
3. Affine transformation
Calculate the affine transformation matrix for each triangle area in 2 and apply it to the face area to finally achieve preliminary alignment:
4. Seamless Cloning (seamless cloning)
In step 3, the gaps at the highlighted edges are obvious, and the distortion is large. Here, we finally use OpenCV's Seamless Cloning function for post-processing. Here we need a face Mask (with the help of fillConvexPoly function), a comparison image, and the processing image
output = cv2.seamlessClone(np.uint8(img1Warped),img2,mask,center,cv2.NORMAL_CLONE)
Looks pretty good, right
4. Small summary
There are many techniques used in this article, involving dlib feature point extraction, Subdiv2D calculation Delaunay triangulation, find Convexhull calculation polygon area convex hull, fillConvexpoly polygon area filling and other technologies
This article is also considered an advanced application of OpenCV. For those who have just learned about it, it takes some time to fully grasp it. It is still recommended to follow the code to understand the basic process sequence of Coding. Due to space reasons, the core code is placed below:
def warpTriangle(img1,img2,t1,t2):
# Find bounding rectangle for each triangle
r1 = cv2.boundingRect(np.float32([t1]))
r2 = cv2.boundingRect(np.float32([t2]))
# Offset points by left top corner of respective rectangles
t1Rect =[]
t2Rect = []
t2RectInt = []
for i in range(0,3):
t1Rect.append(((t1[i][0]-r1[0]),(t1[i][1]-r1[1])))
t2Rect.append(((t2[i][0]-r2[0]),(t2[i][1]-r2[1])))
t2RectInt.append(((t2[i][0] - r2[0]),(t2[i][1]-r2[1])))
# Get mask by filling triangle
mask = np.zeros((r2[3],r2[2],3),dtype = np.float32)
cv2.fillConvexPoly(mask,np.int32(t2RectInt),(1.0,1.0,1.0),16,0)
# Apply warpImage to small rectangular patches
img1Rect = img1[r1[1]:r1[1]+r1[3],r1[0]:r1[0]+r1[2]]
size = (r2[2],r2[3])
img2Rect = applyAffineTransform(img1Rect,t1Rect,t2Rect,size)
img2Rect = img2Rect*mask
# Copy triangular region of the rectangular patch to the output image
img2[r2[1]:r2[1]+r2[3],r2[0]:r2[0]+r2[2]] = img2[r2[1]:r2[1]+r2[3],r2[0]:r2[0]+r2[2]] *((1.0,1.0,1.0)-mask)
img2[r2[1]:r2[1] +r2[3],r2[0]:r2[0]+r2[2]] = img2[r2[1]:r2[1]+r2[3],r2[0]:r2[0]+r2[2]] +img2Rect
The code files involved in the article have been uploaded to Github, follow the official account : Mr. Z, click the key, and you can get it by replying to the keyword Face Swap in the background .