While recent advances in deep neural networks have made it possible to render
high-quality images, generating photo-realistic and personalized talking head
remains challenging. With given audio, the key to tackling this task is
synchronizing lip movement and simultaneously generating personalized
attributes like head movement and eye blink. In this work, we observe that the
input audio is highly correlated to lip motion while less correlated to other
personalized attributes (e.g., head movements). Inspired by this, we propose a
novel framework based on neural radiance field to pursue high-fidelity and
personalized talking head generation. Specifically, neural radiance field takes
lip movements features and personalized attributes as two disentangled
conditions, where lip movements are directly predicted from the audio inputs to
achieve lip-synchronized generation. In the meanwhile, personalized attributes
are sampled from a probabilistic model, where we design a Transformer-based
variational autoencoder sampled from Gaussian Process to learn plausible and
natural-looking head pose and eye blink. Experiments on several benchmarks
demonstrate that our method achieves significantly better results than
state-of-the-art methods.

本文提出了一种基于神经辐射场的框架来生成高保真、个性化的说话人头像，并利用大量数据观察到输入音频与唇部运动高度相关，提出一种条件分解方法，其中唇部运动直接从音频输入中预测来实现同步生成，同时从高斯过程中设计一个变分自编码器来学习可能的、自然的头部姿势和眨眼的个性化属性。多个基准测试表明，该方法实现了远优于现有方法的成果。