Meitu Freckle Removal Algorithm in Practice——Freckle Elimination

Meitu Freckle Removal Algorithm in Practice——Freckle Elimination

This article is mainly based on the idea of ​​this blog post . Since the original blog post did not release the complete code, I try to implement the full version of the code of the face freckle algorithm based on the author's idea. This code uses some tricks for specific images, not a general algorithm, just do refer to.
The algorithm flow is mainly as follows:

1. Grayscale
2. contrast enhancement
3. Thresholding
4. Gradient maxima search
5. connected domain analysis
6. skin exclusion
7. image restoration inpaint

The pictures used in this code are as follows, downloaded directly from this blog post , our goal is to remove the freckles on the neck and shoulders of the task.

Import the library used

import cv2
import matplotlib.pyplot as plt
import numpy as np
import skimage
from skimage import morphology
from skimage.io import imread

Read the image, convert it to grayscale, and then perform contrast enhancement

img = cv2.imread('./face.png')
src = img.copy()
src = cv2.cvtColor(src, cv2.COLOR_BGR2GRAY) # 转为灰度图
con = skimage.exposure.adjust_sigmoid(src,0.4) # 0.4这个阈值就比较trick了。。。

Edge detection using sobel operator

x_gray = cv2.Sobel(con, cv2.CV_32F, 1, 0, ksize=3)
y_gray = cv2.Sobel(con, cv2.CV_32F, 0, 1, ksize=3)
x_gray = cv2.convertScaleAbs(x_gray)
y_gray = cv2.convertScaleAbs(y_gray)
dst = cv2.add(x_gray, y_gray, dtype=cv2.CV_16S)
dst = cv2.convertScaleAbs(dst)

plt.imshow(dst)

After visualizing dst, it is found that the edges have been successfully identified, including the edges of freckles and faces. Then do binary processing on the gradient, select a threshold, set it to 255 if it is greater than the threshold, and set it to 0 if it is less than the threshold

a = dst.copy()
thres = 40
a[a>thres]=255
a[a<=thres]=0
plt.show(a)

At this point, the outline of the freckles has been clearly displayed in front of us, but some shadows on the left face have also been extracted, and we must find a way to remove them.

In order to use the connected domain to remove the shadow of the face, first use the closed operation to process the binary image, connect the scattered points of the shadow into one piece, and use the connected domain analysis to remove the connected domain whose area is greater than a certain threshold

labels = ((morphology.closing(a, skimage.morphology.disk(7))>0)*255).astype(np.uint8)
num_labels,labels,stats,centers = cv2.connectedComponentsWithStats(labels, connectivity=4,ltype=cv2.CV_32S)
for t in range(1, num_labels, 1):
        x, y, w, h, area = stats[t]
        if area>300:
            index = np.where(labels==t)
            labels[index[0], index[1]] = 0

mask = (labels>0).astype(np.uint8)*255
plt.imshow(mask)

Using closed operation + connected domain processing , most of the edges of the human body are successfully removed, leaving a mask of freckles, as shown in the figure below

Sometimes the freckle is relatively large, and there is a hole in the middle of the outline. Here, a function is used to fill the hole in the freckle outline to make it a solid mask

def remove_holes_in_region(mask_scar):
    cnts, _ = cv2.findContours(mask_scar.astype(np.uint8), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
    for cnt in cnts:
        cnt = cnt.transpose(1,0,2) # (1,N,2) N:points number
        cv2.fillPoly(mask_scar, cnt, 1)
    return mask_scar
mask = remove_holes_in_region(mask)

Finally, use the cv2.inpaint function to input the original image img and the freckle mask we got to patch the image

dst_TELEA = cv2.inpaint(img,mask,3,cv2.INPAINT_TELEA)
plt.imshow(dst)

The effect is pretty good! Probably realized the function of removing freckles

Summarize

It cannot be said that this algorithm is not good, it can be said that it is completely useless

quote

Idea Source Blog
Meitu Patent

---

Note: If you want to reprint, please contact the author