face3d: Python tools for processing 3D face

git code: https://github.com/yfeng95/face3d
paper list: PaperWithCode

Based on the BFM model, estimating the parameters of 3DMM can realize linear face representation. This method can be used for face generation, pose detection and rendering based on key points. recommend! ! !

face3d: Python tools for processing 3D face
1. Review
Summarize

Face 1 generated by 3DMM model: normal expression
insert image description here
Face 2 generated by 3DMM model: smiling expression

3DMM模型是如何运行的？其原理是怎样的？如何实现三维人脸定制生成呢？
To answer the above questions, it is necessary to figure out what information 3DMM provides? How to edit this information to achieve a specific face generation.

1. Review

The solution process in Face3d can be summarized as follows:

Initialize α, β is 0;
Use the gold standard algorithm to get an affine matrix {P_A}, decompose to get ${s,R,t_{2d}}$
$ds,R,t_{2d}$ calculated in (2) $s, R, t_{2}$ Bring it into the energy equation to solve β;
Substitute the α obtained in (2) and (3) into the energy equation to solve α;
Update the values of α and β, and repeat (2)-(4) for iterative update

We have covered the first two steps of the algorithm in the previous series. Here we continue to analyze how to solve α, β

1.1 Calculate $ds,R,t_{2d} calculated in (2)$ Bringing into the energy equation, the solution of β

Code Analysis
In the previous article, we solved the affine matrix $P_A through the gold standard algorithm$ and decompose it into $s,R,t_{2d}$ , this part will continue to analyze the source code according to the solution steps:

for i in range(max_iter):
        X = shapeMU + shapePC.dot(sp) + expPC.dot(ep)
        X = np.reshape(X, [int(len(X)/3), 3]).T
        
        #----- estimate pose
        P = mesh.transform.estimate_affine_matrix_3d22d(X.T, x.T)
        s, R, t = mesh.transform.P2sRt(P)
        rx, ry, rz = mesh.transform.matrix2angle(R)
        # print('Iter:{}; estimated pose: s {}, rx {}, ry {}, rz {}, t1 {}, t2 {}'.format(i, s, rx, ry, rz, t[0], t[1]))

        #----- estimate shape
        # expression
        shape = shapePC.dot(sp)
        shape = np.reshape(shape, [int(len(shape)/3), 3]).T
        ep = estimate_expression(x, shapeMU, expPC, model['expEV'][:n_ep,:], shape, s, R, t[:2], lamb = 0.002)

        # shape
        expression = expPC.dot(ep)
        expression = np.reshape(expression, [int(len(expression)/3), 3]).T
        sp = estimate_shape(x, shapeMU, shapePC, model['shapeEV'][:n_sp,:], expression, s, R, t[:2], lamb = 0.004)

    return sp, ep, s, R, t

For the formula:
insert image description here
where the shape part is $\sum_{i=1}^{m}\alpha_iS_i$ ,
shape = shapePC.dot(sp)
define the addition of shape by .
The format of the shape is (159645, 1), and then
shape = np.reshape(shape, [int(len(shape)/3), 3]).T
the XYZ coordinates of the shape are separated to convert to (53215, 3) format.

The next piece of code
ep = estimate_expression(x, shapeMU, expPC, model['expEV'][:n_ep,:], shape, s, R, t[:2], lamb = 0.002)
where ep is β, the source code of estimate_expression is as follows:

def estimate_expression(x, shapeMU, expPC, expEV, shape, s, R, t2d, lamb = 2000):
    '''
    Args:
        x: (2, n). image points (to be fitted)
        shapeMU: (3n, 1)
        expPC: (3n, n_ep)
        expEV: (n_ep, 1)
        shape: (3, n)
        s: scale
        R: (3, 3). rotation matrix
        t2d: (2,). 2d translation
        lambda: regulation coefficient

    Returns:
        exp_para: (n_ep, 1) shape parameters(coefficients)
    '''
    x = x.copy()
    assert(shapeMU.shape[0] == expPC.shape[0])
    assert(shapeMU.shape[0] == x.shape[1]*3)

    dof = expPC.shape[1]

    n = x.shape[1]
    sigma = expEV
    t2d = np.array(t2d)
    P = np.array([[1, 0, 0], [0, 1, 0]], dtype = np.float32)
    A = s*P.dot(R) #(2,3)

    # --- calc pc
    pc_3d = np.resize(expPC.T, [dof, n, 3]) 
    pc_3d = np.reshape(pc_3d, [dof*n, 3]) # (29n,3)
    pc_2d = pc_3d.dot(A.T) #(29n,2)
    pc = np.reshape(pc_2d, [dof, -1]).T # 2n x 29

    # --- calc b
    # shapeMU
    mu_3d = np.resize(shapeMU, [n, 3]).T # 3 x n
    # expression
    shape_3d = shape
    # 
    b = A.dot(mu_3d + shape_3d) + np.tile(t2d[:, np.newaxis], [1, n]) # 2 x n
    b = np.reshape(b.T, [-1, 1]) # 2n x 1

    # --- solve
    equation_left = np.dot(pc.T, pc) + lamb * np.diagflat(1/sigma**2)
    x = np.reshape(x.T, [-1, 1])
    equation_right = np.dot(pc.T, x - b)

    exp_para = np.dot(np.linalg.inv(equation_left), equation_right)
    
    return exp_para

Next, we analyze the estimate_expression function:

data processing

 x = x.copy()
    assert(shapeMU.shape[0] == expPC.shape[0])
    assert(shapeMU.shape[0] == x.shape[1]*3)

    dof = expPC.shape[1]

    n = x.shape[1]
    sigma = expEV
    t2d = np.array(t2d)
    P = np.array([[1, 0, 0], [0, 1, 0]], dtype = np.float32)

First, confirm that the format of the input is correct:
assert(shapeMU.shape[0] == expPC.shape[0])
assert(shapeMU.shape[0] == x.shape[1]*3)
then the expression principal component expPC format input at this time is (159645,29)
let dof=29
dof = expPC.shape[1]
let n=68
n = x.shape[1]
and sigma=expEV that is the variance of the expression principal component σ
sigma = expEV
$t_{2d}$ Convert to array array
t2d = np.array(t2d)
P, which is the orthogonal projection matrix $P_{orth}$
P = np.array([[1, 0, 0], [0, 1, 0]], dtype = np.float32)

	A = s*P.dot(R) #(2,3)

    # --- calc pc
    pc_3d = np.resize(expPC.T, [dof, n, 3]) 
    pc_3d = np.reshape(pc_3d, [dof*n, 3]) # (29n,3)
    pc_2d = pc_3d.dot(A.T) #(29n,2)
    pc = np.reshape(pc_2d, [dof, -1]).T # 2n x 29

    # --- calc b
    # shapeMU
    mu_3d = np.resize(shapeMU, [n, 3]).T # 3 x n
    # expression
    shape_3d = shape
    # 
    b = A.dot(mu_3d + shape_3d) + np.tile(t2d[:, np.newaxis], [1, n]) # 2 x n
    b = np.reshape(b.T, [-1, 1]) # 2n x 1

Known formulas
insert image description here
define and calculate A, pc and b:

Definition A:
A = s*P.dot(R)
Calculation of pc:
The calculation of pc here is equivalent to the following formula

to convert the expression principal component expPC into a new matrix pc_3d of (29, 68, 3)
pc_3d = np.resize(expPC.T, [dof, n, 3])

Note: The expPC here has been
expPC = model['expPC'][valid_ind, :n_ep]
calculated and only contains the expression principal components of the feature points. The format is (68 3,29)
to convert pc_3d to the format of (29 68,3):
pc_3d = np.reshape(pc_3d, [dof*n, 3])
calculate $pc_{2d}= pc_{3d}\cdot A^T$ , pc_2d format is (29*68, 2):
pc_2d = pc_3d.dot(A.T)

Get pc after expanding pc_2d:
pc = np.reshape(pc_2d, [dof, -1]).T

The formula for defining b
b is as follows:

Due to the format of the matrix, some transformations must be performed first:
the shapeMU here also contains only 68 feature points.
The shapeMU with the format (68*3,1) is converted to the format (3, 68):
mu_3d = np.resize(shapeMU, [n, 3]).T
Here shape = shapePC.dot(sp), that is, $\sum_{i=1}^{m}a_iS_i$ ：
shape_3d = shape

At this point, b can be calculated according to the formula. The obtained b format is (2,68)
b = A.dot(mu_3d + shape_3d) + np.tile(t2d[:, np.newaxis], [1, n])
and then b is converted to the format (68*2,1)
b = np.reshape(bT, [-1, 1])

Calculate β
after completing the definition and calculation of A, pc and b $X_{projection}$ The formula of can be written as:
insert image description here
the formula brought into pc can be written as:

将 $X_{projection}$ Bring the formula of β into the energy equation:
insert image description here
get the derivative of β, and get the value of β when the derivative is zero.
The derivation of the L2 norm can be obtained by using the formula:

Obtain:

Simplify to obtain:

Next is the code for obtaining β:

equation_left = np.dot(pc.T, pc) + lamb * np.diagflat(1/sigma**2)
x = np.reshape(x.T, [-1, 1])
equation_right = np.dot(pc.T, x - b)
exp_para = np.dot(np.linalg.inv(equation_left), equation_right)

1.2 (4). Substitute the β obtained in (2) and (3) into the energy equation to solve α

Similarly, the code to obtain α is as follows:

expression = expPC.dot(ep)
expression = np.reshape(expression, [int(len(expression)/3), 3]).T
sp = estimate_shape(x, shapeMU, shapePC, model['shapeEV'][:n_sp,:], expression, s, R, t[:2], lamb = 0.004)

The algorithm process is the same as calculating β, but the β that is brought in is the new value after the above calculation.

1.3 (5) Update the values of α and β, and repeat (2)-(4) for iterative update

At this point, the code of the loop iteration part comes to an end. After multiple iterations (the number of iterations given in the program is three times), the required sp, ep, s, R, t are obtained.

back to routine

Go back to bfm.fit and
fitted_sp, fitted_ep, s, R, t = fit.fit_points(x, X_ind, self.model, n_sp = self.n_shape_para, n_ep = self.n_exp_para, max_iter = max_iter)
continue to execute downwards:

def fit(self, x, X_ind, max_iter = 4, isShow = False):
        ''' fit 3dmm & pose parameters
        Args:
            x: (n, 2) image points
            X_ind: (n,) corresponding Model vertex indices
            max_iter: iteration
            isShow: whether to reserve middle results for show
        Returns:
            fitted_sp: (n_sp, 1). shape parameters
            fitted_ep: (n_ep, 1). exp parameters
            s, angles, t
        '''
        if isShow:
            fitted_sp, fitted_ep, s, R, t = fit.fit_points_for_show(x, X_ind, self.model, n_sp = self.n_shape_para, n_ep = self.n_exp_para, max_iter = max_iter)
            angles = np.zeros((R.shape[0], 3))
            for i in range(R.shape[0]):
                angles[i] = mesh.transform.matrix2angle(R[i])
        else:
            fitted_sp, fitted_ep, s, R, t = fit.fit_points(x, X_ind, self.model, n_sp = self.n_shape_para, n_ep = self.n_exp_para, max_iter = max_iter)
            angles = mesh.transform.matrix2angle(R)
        return fitted_sp, fitted_ep, s, angles, t

angles = mesh.transform.matrix2angle(R)
Returns fitted_sp, fitted_ep, s, angles, t after converting the rotation matrix to XYZ angles .

Go back to the 3DMM routine.
fitted_sp, fitted_ep, fitted_s, fitted_angles, fitted_t = bfm.fit(x, X_ind, max_iter = 3)
After the execution is completed, continue to execute:

x = projected_vertices[bfm.kpt_ind, :2] # 2d keypoint, which can be detected from image
X_ind = bfm.kpt_ind # index of keypoints in 3DMM. fixed.

# fit
fitted_sp, fitted_ep, fitted_s, fitted_angles, fitted_t = bfm.fit(x, X_ind, max_iter = 3)

# verify fitted parameters
fitted_vertices = bfm.generate_vertices(fitted_sp, fitted_ep)
transformed_vertices = bfm.transform(fitted_vertices, fitted_s, fitted_angles, fitted_t)

image_vertices = mesh.transform.to_image(transformed_vertices, h, w)
fitted_image = mesh.render.render_colors(image_vertices, bfm.triangles, colors, h, w)

The next step is to calculate $S_{newModel}$
fitted_vertices = bfm.generate_vertices(fitted_sp, fitted_ep)
by substituting the calculated α and β
insert image description here
$S_{n e wM o d e l}$ , the corresponding source code is as follows:

 def generate_vertices(self, shape_para, exp_para):
        '''
        Args:
            shape_para: (n_shape_para, 1)
            exp_para: (n_exp_para, 1) 
        Returns:
            vertices: (nver, 3)
        '''
        vertices = self.model['shapeMU'] + \
                   self.model['shapePC'].dot(shape_para) + \
                   self.model['expPC'].dot(exp_para)
        vertices = np.reshape(vertices, [int(3), int(len(vertices)/3)], 'F').T

        return vertices

算出 $S_{newModel}$ Then perform a similar transformation on the 3D model:
transformed_vertices = bfm.transform(fitted_vertices, fitted_s, fitted_angles, fitted_t)

即 $S_{tranformed}=s\cdot R\cdot S_{newModel}+t_{3d}$

 def transform(self, vertices, s, angles, t3d):
        R = mesh.transform.angle2matrix(angles)
        return mesh.transform.similarity_transform(vertices, s, R, t3d)

Code after:

image_vertices = mesh.transform.to_image(transformed_vertices, h, w)
fitted_image = mesh.render.render_colors(image_vertices, bfm.triangles, colors, h, w)

They are converting the 3D model into a 2D image format and coloring the model with the built-in color information. I won’t explain too much here.

Result display
The generated new face image is saved to the results/3dmm directory

# ------------- print & show 
print('pose, groudtruth: \n', s, angles[0], angles[1], angles[2], t[0], t[1])
print('pose, fitted: \n', fitted_s, fitted_angles[0], fitted_angles[1], fitted_angles[2], fitted_t[0], fitted_t[1])

save_folder = 'results/3dmm'
if not os.path.exists(save_folder):
    os.mkdir(save_folder)

io.imsave('{}/generated.jpg'.format(save_folder), image)
io.imsave('{}/fitted.jpg'.format(save_folder), fitted_image)

You can also generate a gif to show the process of feature point fitting:

# fit

fitted_sp, fitted_ep, fitted_s, fitted_angles, fitted_t = bfm.fit(x, X_ind, max_iter = 3, isShow = True)

# verify fitted parameters
for i in range(fitted_sp.shape[0]):
	fitted_vertices = bfm.generate_vertices(fitted_sp[i], fitted_ep[i])
	transformed_vertices = bfm.transform(fitted_vertices, fitted_s[i], fitted_angles[i], fitted_t[i])

	image_vertices = mesh.transform.to_image(transformed_vertices, h, w)
	fitted_image = mesh.render.render_colors(image_vertices, bfm.triangles, colors, h, w)
	io.imsave('{}/show_{:0>2d}.jpg'.format(save_folder, i), fitted_image)

options = '-delay 20 -loop 0 -layers optimize' # gif. need ImageMagick.
subprocess.call('convert {} {}/show_*.jpg {}'.format(options, save_folder, save_folder + '/3dmm.gif'), shell=True)
subprocess.call('rm {}/show_*.jpg'.format(save_folder), shell=True)

Here is the result display:

generated.jpg
fitted.jpg
3dmm.gif

insert image description here
Of course, the new random model generated by the program will look different each time it is executed.

Summarize

Here we mainly introduce the process of solving α and β based on the reverse process of the 3DMM model. Many details still need to be compared with the code to understand the formula and principle, and combine the formula to guide the code reading.

3D Face Practice: Face Generation, Rendering and 3D Reconstruction Based on Face3D <3>

face3d: Python tools for processing 3D face

Table of contents

1. Review

1.1 Calculate $ds,R,t_{2d} calculated in (2)$ Bringing into the energy equation, the solution of β

1.2 (4). Substitute the β obtained in (2) and (3) into the energy equation to solve α

1.3 (5) Update the values of α and β, and repeat (2)-(4) for iterative update

Summarize

Guess you like

3D Face Practice: Face Generation, Rendering and 3D Reconstruction Based on Face3D <3>

face3d: Python tools for processing 3D face

Table of contents

1. Review

1.1 Calculate s , R , t 2 ds,R,t_{2d} calculated in (2)s,R,t2 dBringing into the energy equation, the solution of β

1.2 (4). Substitute the β obtained in (2) and (3) into the energy equation to solve α

1.3 (5) Update the values ​​of α and β, and repeat (2)-(4) for iterative update

Summarize

Guess you like

1.1 Calculate $ds,R,t_{2d} calculated in (2)$ Bringing into the energy equation, the solution of β

1.3 (5) Update the values of α and β, and repeat (2)-(4) for iterative update