The remarkable progress in 3D face reconstruction has resulted in high-detail
and photorealistic facial representations. Recently, Diffusion Models have
revolutionized the capabilities of generative methods by achieving far better
performance than GANs. In this work, we present FitDiff, a diffusion-based 3D
facial avatar generative model. This model accurately generates relightable
facial avatars, utilizing an identity embedding extracted from an "in-the-wild"
2D facial image. Our multi-modal diffusion model concurrently outputs facial
reflectance maps (diffuse and specular albedo and normals) and shapes,
showcasing great generalization capabilities. It is solely trained on an
annotated subset of a public facial dataset, paired with 3D reconstructions. We
revisit the typical 3D facial fitting approach by guiding a reverse diffusion
process using perceptual and face recognition losses. Being the first LDM
conditioned on face recognition embeddings, FitDiff reconstructs relightable
human avatars, that can be used as-is in common rendering engines, starting
only from an unconstrained facial image, and achieving state-of-the-art
performance.

在这项研究中，我们提出了 FitDiff，一种基于扩散的三维人脸化身生成模型，它可准确生成可光照补充的人脸化身，利用从 “野外” 二维人脸图像中提取的身份嵌入。该多模态扩散模型同时输出面部反射映射（漫反射和镜面反射率及法线）和形状，展示了强大的泛化能力。它仅在公共人脸数据集的注释子集上进行训练，与三维重建配对。我们通过使用感知和人脸识别损失来指导反向扩散过程，重新审视了典型的三维面部拟合方法。FitDiff 是第一个以面部识别嵌入为条件的 LDM，它能够重构可光照补充的人类化身，可以直接在常见的渲染引擎中使用，仅从无约束的人脸图像开始，并达到了业界的最佳性能。