In this paper, we abstract the process of people hearing speech, extracting
meaningful cues, and creating various dynamically audio-consistent talking
faces, termed Listening and Imagining, into the task of high-fidelity diverse
talking faces generation from a single audio. Specifically, it involves two
critical challenges: one is to effectively decouple identity, content, and
emotion from entangled audio, and the other is to maintain intra-video
diversity and inter-video consistency. To tackle the issues, we first dig out
the intricate relationships among facial factors and simplify the decoupling
process, tailoring a Progressive Audio Disentanglement for accurate facial
geometry and semantics learning, where each stage incorporates a customized
training module responsible for a specific factor. Secondly, to achieve
visually diverse and audio-synchronized animation solely from input audio
within a single model, we introduce the Controllable Coherent Frame generation,
which involves the flexible integration of three trainable adapters with frozen
Latent Diffusion Models (LDMs) to focus on maintaining facial geometry and
semantics, as well as texture and temporal coherence between frames. In this
way, we inherit high-quality diverse generation from LDMs while significantly
improving their controllability at a low training cost. Extensive experiments
demonstrate the flexibility and effectiveness of our method in handling this
paradigm. The codes will be released at
this https URL

从单一音频生成具有高保真度的多样对话人脸，通过音频解耦、面部几何和语义学习、可控一致帧生成实现视觉多样性和音频同步动画。