Achieving disentangled control over multiple facial motions and accommodating
diverse input modalities greatly enhances the application and entertainment of
the talking head generation. This necessitates a deep exploration of the
decoupling space for facial features, ensuring that they a) operate
independently without mutual interference and b) can be preserved to share with
different modal input, both aspects often neglected in existing methods. To
address this gap, this paper proposes a novel Efficient Disentanglement
framework for Talking head generation (EDTalk). Our framework enables
individual manipulation of mouth shape, head pose, and emotional expression,
conditioned on video or audio inputs. Specifically, we employ three lightweight
modules to decompose the facial dynamics into three distinct latent spaces
representing mouth, pose, and expression, respectively. Each space is
characterized by a set of learnable bases whose linear combinations define
specific motions. To ensure independence and accelerate training, we enforce
orthogonality among bases and devise an efficient training strategy to allocate
motion responsibilities to each space without relying on external knowledge.
The learned bases are then stored in corresponding banks, enabling shared
visual priors with audio input. Furthermore, considering the properties of each
space, we propose an Audio-to-Motion module for audio-driven talking head
synthesis. Experiments are conducted to demonstrate the effectiveness of
EDTalk. We recommend watching the project website:
this https URL

本研究提出了一个用于言语生成的高效解开耦合框架 (EDTalk)，该框架具有分解面部动态的能力，允许对嘴型、头部姿势和情绪表达进行个别操作，并能根据视频或音频输入进行条件设置。我们通过三个轻量级模块将面部动态分解为分别代表嘴部、姿势和表情的三个独立潜在空间，并利用可学习的基向量对每个空间内的特定运动进行定义。我们通过在基向量之间施加正交性约束并设计了高效的训练策略来加速训练过程，同时保证独立性。学习到的基向量存储在相应的存储器中，以实现与音频输入的共享先验知识。另外，鉴于每个空间的特性，我们还提出了一个音频到运动模块，用于音频驱动的言语生成。实验证明了 EDTalk 的有效性。