Face reenactment is challenging due to the need to establish dense
correspondence between various face representations for motion transfer. Recent
studies have utilized Neural Radiance Field (NeRF) as fundamental
representation, which further enhanced the performance of multi-view face
reenactment in photo-realism and 3D consistency. However, establishing dense
correspondence between different face NeRFs is non-trivial, because implicit
representations lack ground-truth correspondence annotations like mesh-based 3D
parametric models (e.g., 3DMM) with index-aligned vertexes. Although aligning
3DMM space with NeRF-based face representations can realize motion control, it
is sub-optimal for their limited face-only modeling and low identity fidelity.
Therefore, we are inspired to ask: Can we learn the dense correspondence
between different NeRF-based face representations without a 3D parametric model
prior? To address this challenge, we propose a novel framework, which adopts
tri-planes as fundamental NeRF representation and decomposes face tri-planes
into three components: canonical tri-planes, identity deformations, and motion.
In terms of motion control, our key contribution is proposing a Plane
Dictionary (PlaneDict) module, which efficiently maps the motion conditions to
a linear weighted addition of learnable orthogonal plane bases. To the best of
our knowledge, our framework is the first method that achieves one-shot
multi-view face reenactment without a 3D parametric model prior. Extensive
experiments demonstrate that we produce better results in fine-grained motion
control and identity preservation than previous methods.

我们提出了一种新颖的框架，使用三面体平面作为基本 NeRF 表示，并将面的三个部分分解为：规范三面体平面、身份形变和运动。在运动控制方面，我们的主要贡献是提出了一个平面字典（PlaneDict）模块，它将运动条件高效地映射到可学习的正交平面基的线性加权相加。据我们所知，我们的框架是第一个在没有 3D 参数模型先验的情况下实现一次性多视角面部重现的方法。大量实验表明，我们在细粒度运动控制和身份保持方面产生了比以前的方法更好的结果。

基于 NeRF 的人脸再现的密集对应关系学习

Learning Dense Correspondence for NeRF-Based Face Reenactment

We present a method that reconstructs and animates a 3D head avatar from a
single-view portrait image. Existing methods either involve time-consuming
optimization for a specific person with multiple images, or they struggle to
synthesize intricate appearance details beyond the facial region. To address
these limitations, we propose a framework that not only generalizes to unseen
identities based on a single-view image without requiring person-specific
optimization, but also captures characteristic details within and beyond the
face area (e.g. hairstyle, accessories, etc.). At the core of our method are
three branches that produce three tri-planes representing the coarse 3D
geometry, detailed appearance of a source image, as well as the expression of a
target image. By applying volumetric rendering to the combination of the three
tri-planes followed by a super-resolution module, our method yields a high
fidelity image of the desired identity, expression and pose. Once trained, our
model enables efficient 3D head avatar reconstruction and animation via a
single forward pass through a network. Experiments show that the proposed
approach generalizes well to unseen validation datasets, surpassing SOTA
baseline methods by a large margin on head avatar reconstruction and animation.

该研究提出了一种通过对单视角幅面图像进行重建和动画化的三维头像，通过三个三角面分别捕捉源图像的粗略三维几何，详细外观以及目标图像的表情实现，再通过超分辨率模块的渲染进行填充并通过单向传递网络的高效性进行动画。实验表明，该方法在重建和动画方面优于当前最先进的基线方法，并且对新的验证数据集有很好的泛化能力。