Generating photo-realistic video portrait with arbitrary speech audio is a
crucial problem in film-making and virtual reality. Recently, several works
explore the usage of neural radiance field in this task to improve 3D realness
and image fidelity. However, the generalizability of previous NeRF-based
methods to out-of-domain audio is limited by the small scale of training data.
In this work, we propose GeneFace, a generalized and high-fidelity NeRF-based
talking face generation method, which can generate natural results
corresponding to various out-of-domain audio. Specifically, we learn a
variaitional motion generator on a large lip-reading corpus, and introduce a
domain adaptative post-net to calibrate the result. Moreover, we learn a
NeRF-based renderer conditioned on the predicted facial motion. A head-aware
torso-NeRF is proposed to eliminate the head-torso separation problem.
Extensive experiments show that our method achieves more generalized and
high-fidelity talking face generation compared to previous methods.

研究提出了一种新的通用高保真 NeRF-based 说话人脸生成方法 GeneFace，并且在大规模口型阅读语料库的基础上，学习了一个变分运动生成器，并引入了一个域自适应后置网络来校准结果，实现了通用外域音频的自然结果创造。