Recently, emotional talking face generation has received considerable
attention. However, existing methods only adopt one-hot coding, image, or audio
as emotion conditions, thus lacking flexible control in practical applications
and failing to handle unseen emotion styles due to limited semantics. They
either ignore the one-shot setting or the quality of generated faces. In this
paper, we propose a more flexible and generalized framework. Specifically, we
supplement the emotion style in text prompts and use an Aligned Multi-modal
Emotion encoder to embed the text, image, and audio emotion modality into a
unified space, which inherits rich semantic prior from CLIP. Consequently,
effective multi-modal emotion space learning helps our method support arbitrary
emotion modality during testing and could generalize to unseen emotion styles.
Besides, an Emotion-aware Audio-to-3DMM Convertor is proposed to connect the
emotion condition and the audio sequence to structural representation. A
followed style-based High-fidelity Emotional Face generator is designed to
generate arbitrary high-resolution realistic identities. Our texture generator
hierarchically learns flow fields and animated faces in a residual manner.
Extensive experiments demonstrate the flexibility and generalization of our
method in emotion control and the effectiveness of high-quality face synthesis.

本研究提出了一种更灵活、更通用的基于文本、图像和语音情感模态的多模情感编码器，以及一个面向情感的音频到 3DMM 转换器和一个高保真情感脸生成器，通过充分利用深度神经网络，实现了情感控制的灵活性和可扩展性，进而提高了合成图像的质量和细节。