Animating high-fidelity video portrait with speech audio is crucial for
virtual reality and digital entertainment. While most previous studies rely on
accurate explicit structural information, recent works explore the implicit
scene representation of Neural Radiance Fields (NeRF) for realistic generation.
In order to capture the inconsistent motions as well as the semantic difference
between human head and torso, some work models them via two individual sets of
NeRF, leading to unnatural results. In this work, we propose Semantic-aware
Speaking Portrait NeRF (SSP-NeRF), which creates delicate audio-driven
portraits using one unified set of NeRF. The proposed model can handle the
detailed local facial semantics and the global head-torso relationship through
two semantic-aware modules. Specifically, we first propose a Semantic-Aware
Dynamic Ray Sampling module with an additional parsing branch that facilitates
audio-driven volume rendering. Moreover, to enable portrait rendering in one
unified neural radiance field, a Torso Deformation module is designed to
stabilize the large-scale non-rigid torso motions. Extensive evaluations
demonstrate that our proposed approach renders more realistic video portraits
compared to previous methods. Project page:
this https URL

该研究提出了一种语音驱动的 Semantic-aware Speaking Portrait NeRF (SSP-NeRF) 模型，该模型可以通过两个语义感知模块处理细节本地面部语义和全局头颈关系，从而通过一个统一的神经辐射场呈现面部表情，且通过进行大规模非刚性的 Torso 变形来稳定人物的大比例非刚性运动。