Generating talking person portraits with arbitrary speech audio is a crucial
problem in the field of digital human and metaverse. A modern talking face
generation method is expected to achieve the goals of generalized audio-lip
synchronization, good video quality, and high system efficiency. Recently,
neural radiance field (NeRF) has become a popular rendering technique in this
field since it could achieve high-fidelity and 3D-consistent talking face
generation with a few-minute-long training video. However, there still exist
several challenges for NeRF-based methods: 1) as for the lip synchronization,
it is hard to generate a long facial motion sequence of high temporal
consistency and audio-lip accuracy; 2) as for the video quality, due to the
limited data used to train the renderer, it is vulnerable to out-of-domain
input condition and produce bad rendering results occasionally; 3) as for the
system efficiency, the slow training and inference speed of the vanilla NeRF
severely obstruct its usage in real-world applications. In this paper, we
propose GeneFace++ to handle these challenges by 1) utilizing the pitch contour
as an auxiliary feature and introducing a temporal loss in the facial motion
prediction process; 2) proposing a landmark locally linear embedding method to
regulate the outliers in the predicted motion sequence to avoid robustness
issues; 3) designing a computationally efficient NeRF-based motion-to-video
renderer to achieves fast training and real-time inference. With these
settings, GeneFace++ becomes the first NeRF-based method that achieves stable
and real-time talking face generation with generalized audio-lip
synchronization. Extensive experiments show that our method outperforms
state-of-the-art baselines in terms of subjective and objective evaluation.
Video samples are available at this https URL .

GeneFace++ 是目前第一个实现了稳定和实时的通用音频 - 唇形同步对话人肖像生成，并通过采用辅助特征、引入时间损失、设计计算效率高的 NeRF 模型等方法处理了唇形同步、视频质量和系统效率等方面的挑战。