The generation of emotional talking faces from a single portrait image
remains a significant challenge. The simultaneous achievement of expressive
emotional talking and accurate lip-sync is particularly difficult, as
expressiveness is often compromised for the accuracy of lip-sync. As widely
adopted by many prior works, the LSTM network often fails to capture the
subtleties and variations of emotional expressions. To address these
challenges, we introduce DREAM-Talk, a two-stage diffusion-based audio-driven
framework, tailored for generating diverse expressions and accurate lip-sync
concurrently. In the first stage, we propose EmoDiff, a novel diffusion module
that generates diverse highly dynamic emotional expressions and head poses in
accordance with the audio and the referenced emotion style. Given the strong
correlation between lip motion and audio, we then refine the dynamics with
enhanced lip-sync accuracy using audio features and emotion style. To this end,
we deploy a video-to-video rendering module to transfer the expressions and lip
motions from our proxy 3D avatar to an arbitrary portrait. Both quantitatively
and qualitatively, DREAM-Talk outperforms state-of-the-art methods in terms of
expressiveness, lip-sync accuracy and perceptual quality.

DREAM-Talk 是一个两阶段扩散式音频驱动框架，通过 EmoDiff 生成多样的动态情感表达和姿态，并使用音频特征和情感风格来提升唇部同步准确性，通过视频到视频渲染模块将情感和唇部运动从虚拟 3D 头像传输到任意肖像，在表达性、唇部同步准确性和感知质量方面超过现有方法。

DREAM-Talk: 基于扩散的逼真情感音频驱动的单张图像说话脸生成方法

DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for  Single Image Talking Face Generation

Previous methods on knowledge base question generation (KBQG) primarily focus
on enhancing the quality of a single generated question. Recognizing the
remarkable paraphrasing ability of humans, we contend that diverse texts should
convey the same semantics through varied expressions. The above insights make
diversifying question generation an intriguing task, where the first challenge
is evaluation metrics for diversity. Current metrics inadequately assess the
above diversity since they calculate the ratio of unique n-grams in the
generated question itself, which leans more towards measuring duplication
rather than true diversity. Accordingly, we devise a new diversity evaluation
metric, which measures the diversity among top-k generated questions for each
instance while ensuring their relevance to the ground truth. Clearly, the
second challenge is how to enhance diversifying question generation. To address
this challenge, we introduce a dual model framework interwoven by two selection
strategies to generate diverse questions leveraging external natural questions.
The main idea of our dual framework is to extract more diverse expressions and
integrate them into the generation model to enhance diversifying question
generation. Extensive experiments on widely used benchmarks for KBQG
demonstrate that our proposed approach generates highly diverse questions and
improves the performance of question answering tasks.

我们提出了一种新的多样性评估度量，用于评估每个实例的前 K 个生成问题的多样性，并确保它们与事实相关。同时，我们引入了一个双模型框架，通过两种选择策略将外部自然问题与生成模型相结合，以生成多样化的问题。实验结果表明，我们的方法生成了高度多样化的问题，并改善了问题回答任务的性能。