Audio-driven talking head synthesis is a promising topic with wide
applications in digital human, film making and virtual reality. Recent
NeRF-based approaches have shown superiority in quality and fidelity compared
to previous studies. However, when it comes to few-shot talking head
generation, a practical scenario where only few seconds of talking video is
available for one identity, two limitations emerge: 1) they either have no base
model, which serves as a facial prior for fast convergence, or ignore the
importance of audio when building the prior; 2) most of them overlook the
degree of correlation between different face regions and audio, e.g., mouth is
audio related, while ear is audio independent. In this paper, we present Audio
Enhanced Neural Radiance Field (AE-NeRF) to tackle the above issues, which can
generate realistic portraits of a new speaker with fewshot dataset.
Specifically, we introduce an Audio Aware Aggregation module into the feature
fusion stage of the reference scheme, where the weight is determined by the
similarity of audio between reference and target image. Then, an Audio-Aligned
Face Generation strategy is proposed to model the audio related and audio
independent regions respectively, with a dual-NeRF framework. Extensive
experiments have shown AE-NeRF surpasses the state-of-the-art on image
fidelity, audio-lip synchronization, and generalization ability, even in
limited training set or training iterations.

本文提出了一种名为 AE-NeRF 的音频增强神经辐射场模型，该模型能够生成具有少量样本的新演讲者的逼真肖像。实验证明，AE-NeRF 在图像保真度、音频嘴唇同步和泛化能力方面超过了现有技术，即使在有限的训练数据或训练迭代次数下也能取得优异结果。

AE-NeRF: 音频增强的神经辐射场 用于少样本说话头合成

AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head  Synthesis

Generating talking person portraits with arbitrary speech audio is a crucial
problem in the field of digital human and metaverse. A modern talking face
generation method is expected to achieve the goals of generalized audio-lip
synchronization, good video quality, and high system efficiency. Recently,
neural radiance field (NeRF) has become a popular rendering technique in this
field since it could achieve high-fidelity and 3D-consistent talking face
generation with a few-minute-long training video. However, there still exist
several challenges for NeRF-based methods: 1) as for the lip synchronization,
it is hard to generate a long facial motion sequence of high temporal
consistency and audio-lip accuracy; 2) as for the video quality, due to the
limited data used to train the renderer, it is vulnerable to out-of-domain
input condition and produce bad rendering results occasionally; 3) as for the
system efficiency, the slow training and inference speed of the vanilla NeRF
severely obstruct its usage in real-world applications. In this paper, we
propose GeneFace++ to handle these challenges by 1) utilizing the pitch contour
as an auxiliary feature and introducing a temporal loss in the facial motion
prediction process; 2) proposing a landmark locally linear embedding method to
regulate the outliers in the predicted motion sequence to avoid robustness
issues; 3) designing a computationally efficient NeRF-based motion-to-video
renderer to achieves fast training and real-time inference. With these
settings, GeneFace++ becomes the first NeRF-based method that achieves stable
and real-time talking face generation with generalized audio-lip
synchronization. Extensive experiments show that our method outperforms
state-of-the-art baselines in terms of subjective and objective evaluation.
Video samples are available at this https URL .

GeneFace++ 是目前第一个实现了稳定和实时的通用音频 - 唇形同步对话人肖像生成，并通过采用辅助特征、引入时间损失、设计计算效率高的 NeRF 模型等方法处理了唇形同步、视频质量和系统效率等方面的挑战。