Generating emotional talking faces is a practical yet challenging endeavor.
To create a lifelike avatar, we draw upon two critical insights from a human
perspective: 1) The connection between audio and the non-deterministic facial
dynamics, encompassing expressions, blinks, poses, should exhibit synchronous
and one-to-many mapping. 2) Vibrant expressions are often accompanied by
emotion-aware high-definition (HD) textures and finely detailed teeth. However,
both aspects are frequently overlooked by existing methods. To this end, this
paper proposes using normalizing Flow and Vector-Quantization modeling to
produce emotional talking faces that satisfy both insights concurrently
(FlowVQTalker). Specifically, we develop a flow-based coefficient generator
that encodes the dynamics of facial emotion into a multi-emotion-class latent
space represented as a mixture distribution. The generation process commences
with random sampling from the modeled distribution, guided by the accompanying
audio, enabling both lip-synchronization and the uncertain nonverbal facial
cues generation. Furthermore, our designed vector-quantization image generator
treats the creation of expressive facial images as a code query task, utilizing
a learned codebook to provide rich, high-quality textures that enhance the
emotional perception of the results. Extensive experiments are conducted to
showcase the effectiveness of our approach.

利用归一化流和向量量化建模生成既满足面部动态的同步性和多对多映射关系，又包含高清纹理和细节牙齿的情感说话脸，具体方法为通过流式系数生成器对面部情绪进行编码，利用伴随音频引导随机采样生成唇同步和不确定的非言语面部线索，并借助学习的码本提供富有表现力、高质量的纹理，从而增强结果的情感感知。

FlowVQTalker：通过正则化流和量化实现高质量的情感说话人脸生成

FlowVQTalker: High-Quality Emotional Talking Face Generation through  Normalizing Flow and Quantization

The generation of emotional talking faces from a single portrait image
remains a significant challenge. The simultaneous achievement of expressive
emotional talking and accurate lip-sync is particularly difficult, as
expressiveness is often compromised for the accuracy of lip-sync. As widely
adopted by many prior works, the LSTM network often fails to capture the
subtleties and variations of emotional expressions. To address these
challenges, we introduce DREAM-Talk, a two-stage diffusion-based audio-driven
framework, tailored for generating diverse expressions and accurate lip-sync
concurrently. In the first stage, we propose EmoDiff, a novel diffusion module
that generates diverse highly dynamic emotional expressions and head poses in
accordance with the audio and the referenced emotion style. Given the strong
correlation between lip motion and audio, we then refine the dynamics with
enhanced lip-sync accuracy using audio features and emotion style. To this end,
we deploy a video-to-video rendering module to transfer the expressions and lip
motions from our proxy 3D avatar to an arbitrary portrait. Both quantitatively
and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of
expressiveness, lip-sync accuracy and perceptual quality.

DREAM-Talk 是一个两阶段扩散式音频驱动框架，通过 EmoDiff 生成多样的动态情感表达和姿态，并使用音频特征和情感风格来提升唇部同步准确性，通过视频到视频渲染模块将情感和唇部运动从虚拟 3D 头像传输到任意肖像，在表达性、唇部同步准确性和感知质量方面超过现有方法。