While accurate lip synchronization has been achieved for arbitrary-subject
audio-driven talking face generation, the problem of how to efficiently drive
the head pose remains. Previous methods rely on pre-estimated structural
information such as landmarks and 3D parameters, aiming to generate
personalized rhythmic movements. However, the inaccuracy of such estimated
information under extreme conditions would lead to degradation problems. In
this paper, we propose a clean yet effective framework to generate
pose-controllable talking faces. We operate on raw face images, using only a
single photo as an identity reference. The key is to modularize audio-visual
representations by devising an implicit low-dimension pose code. Substantially,
both speech content and head pose information lie in a joint non-identity
embedding space. While speech content information can be defined by learning
the intrinsic synchronization between audio-visual modalities, we identify that
a pose code will be complementarily learned in a modulated convolution-based
reconstruction framework.
Extensive experiments show that our method generates accurately lip-synced
talking faces whose poses are controllable by other videos. Moreover, our model
has multiple advanced capabilities including extreme view robustness and
talking face frontalization. Code, models, and demo videos are available at
this https URL

本文提出了一种简洁而有效的框架来生成姿势可控的对话脸，通过使用隐式低维姿势代码对原始面部图像进行操作，实现语音和头部姿势信息的联合非身份嵌入空间，通过调制卷积重建框架，在极端视角稳健的情况下生成准确的唇形同步会话，并具有多种先进功能，例如对话面部前视。