Chen Cao, Hongzhi Wu, Yanlin Weng, Tianjia Shao, Kun Zhou, "Real-time Facial Animation with Image-based Dynamic Avatars", ACM Trans. on Graphics (SIGGRAPH), 2016.
构建和使用基于图像的动态化身的pipeline：（1）image acquisition, （2）avatar construction （3） real-time animation
Overview of our pipeline. From captured images of the user that sparsely sample over head poses and expressions, we build a head blendshape and a morphable hair model to represent the corresponding coarse geometry. Other components such as eyes and teeth are expressed as billboards. During runtime, an actor can drive our avatar using a single-camera-based face tracker. The captured images are warped and blended under the guidance of the coarse geometry to generate real-time facial animation with fine-scale details.
从稀疏采样用户头部姿态和表情的捕获图像中，我们构建了头部混合形状和可变形毛发模型来表示相应的粗几何形状。其他部件，如眼睛和牙齿，则表示为广告牌。在运行期间，一个演员可以使用一个基于单摄像头的面部跟踪器来驱动我们的化身。在粗几何的指导下，对采集到的图像进行扭曲和混合，生成具有精细尺度细节的实时人脸动画。
4. Image-based Avatar Construction
4.1 Image Preprocessing
Segmentation:The first step is to segment the images into several layers: head, hair (including headwear), eyes, teeth, body and background.
Landmark Labeling: label a few facial landmarks for each image,which indicate the 2D positions of a set of facial features
4.2 Face Blendshapes
Initial Modeling by Tensor Fitting. For each image, we compute the initial face model by interpolating the 3-mode tensor C of FaceWarehouse
Geometry Refinement by Mesh Deformation. optimize the positions of all mesh vertices
Blendshape Generation
4.3 Morphable Hair Model
Single-Image Depth Estimation
Inter-image Depth Optimization
Correspondence Computation
Morphable Model Generation
4.4 Handling Eyes, Teeth and Body
build the eye and body billboards based on the image with neutral expression under frontal view (i.e., I1), and the tooth billboards from the image with the teeth expression
5 Real-time Animation
5.1 Geometry Construction: describe how to construct the coarse geometry for face and hair, from the rigid head transformation and the expression coefficients
5.2 Image Warping & Blending:warp the captured images of face and hair regions to the current frame, guided by the corresponding coarse geometry