While existing methods for 3D face reconstruction from in-the-wild images
excel at recovering the overall face shape, they commonly miss subtle, extreme,
asymmetric, or rarely observed expressions. We improve upon these methods with
SMIRK (Spatial Modeling for Image-based Reconstruction of Kinesics), which
faithfully reconstructs expressive 3D faces from images. We identify two key
limitations in existing methods: shortcomings in their self-supervised training
formulation, and a lack of expression diversity in the training images. For
training, most methods employ differentiable rendering to compare a predicted
face mesh with the input image, along with a plethora of additional loss
functions. This differentiable rendering loss not only has to provide
supervision to optimize for 3D face geometry, camera, albedo, and lighting,
which is an ill-posed optimization problem, but the domain gap between
rendering and input image further hinders the learning process. Instead, SMIRK
replaces the differentiable rendering with a neural rendering module that,
given the rendered predicted mesh geometry, and sparsely sampled pixels of the
input image, generates a face image. As the neural rendering gets color
information from sampled image pixels, supervising with neural rendering-based
reconstruction loss can focus solely on the geometry. Further, it enables us to
generate images of the input identity with varying expressions while training.
These are then utilized as input to the reconstruction model and used as
supervision with ground truth geometry. This effectively augments the training
data and enhances the generalization for diverse expressions. Our qualitative,
quantitative and particularly our perceptual evaluations demonstrate that SMIRK
achieves the new state-of-the art performance on accurate expression
reconstruction. Project webpage: this https URL

从实景图像中精确重建具有表情的 3D 人脸，通过改进现有方法（SMIRK），引入神经渲染模块并增加表情多样性来优化训练数据，从而显著提高表情重建的性能。

分析 - 合成途径实现的三维面部表情

3D Facial Expressions through Analysis-by-Neural-Synthesis

Face alignment and 3D face reconstruction are traditionally accomplished as
separated tasks. By exploring the strong correlation between 2D landmarks and
3D shapes, in contrast, we propose a joint face alignment and 3D face
reconstruction method to simultaneously solve these two problems for 2D face
images of arbitrary poses and expressions. This method, based on a summation
model of 3D faces and cascaded regression in 2D and 3D shape spaces,
iteratively and alternately applies two cascaded regressors, one for updating
2D landmarks and the other for 3D shape. The 3D shape and the landmarks are
correlated via a 3D-to-2D mapping matrix, which is updated in each iteration to
refine the location and visibility of 2D landmarks. Unlike existing methods,
the proposed method can fully automatically generate both
pose-and-expression-normalized (PEN) and expressive 3D faces and localize both
visible and invisible 2D landmarks. Based on the PEN 3D faces, we devise a
method to enhance face recognition accuracy across poses and expressions. Both
linear and nonlinear implementations of the proposed method are presented and
evaluated in this paper. Extensive experiments show that the proposed method
can achieve the state-of-the-art accuracy in both face alignment and 3D face
reconstruction, and benefit face recognition owing to its reconstructed PEN 3D
face.

提出了一种新的联合人脸对齐和三维人脸重建方法，该方法可以自动生成姿态和表情归一化的三维人脸，并能恢复可见和不可见的二维人脸关键点，从而可以改善跨姿态和表情的人脸识别准确性。实验结果表明，该方法在人脸对齐和三维人脸重建方面均达到了最先进的精度。