Speech-driven animation has gained significant traction in recent years, with
current methods achieving near-photorealistic results. However, the field
remains underexplored regarding non-verbal communication despite evidence
demonstrating its importance in human interaction. In particular, generating
laughter sequences presents a unique challenge due to the intricacy and nuances
of this behaviour. This paper aims to bridge this gap by proposing a novel
model capable of generating realistic laughter sequences, given a still
portrait and an audio clip containing laughter. We highlight the failure cases
of traditional facial animation methods and leverage recent advances in
diffusion models to produce convincing laughter videos. We train our model on a
diverse set of laughter datasets and introduce an evaluation metric
specifically designed for laughter. When compared with previous speech-driven
approaches, our model achieves state-of-the-art performance across all metrics,
even when these are re-trained for laughter generation.

本文提出了一种新颖的模型，可以在给定一个静止肖像和一个包含笑声的音频剪辑的情况下生成逼真的笑声序列，该模型使用扩散模型的最新进展，解决了传统面部动画方法的失败案例，针对笑声引入了一种评估指标，获得了所有指标的最先进表现。

笑声重要：基于扩散模型的笑脸生成介绍

Laughing Matters: Introducing Laughing-Face Generation using Diffusion  Models

This paper presents FaceXHuBERT, a text-less speech-driven 3D facial
animation generation method that allows to capture personalized and subtle cues
in speech (e.g. identity, emotion and hesitation). It is also very robust to
background noise and can handle audio recorded in a variety of situations (e.g.
multiple people speaking). Recent approaches employ end-to-end deep learning
taking into account both audio and text as input to generate facial animation
for the whole face. However, scarcity of publicly available expressive audio-3D
facial animation datasets poses a major bottleneck. The resulting animations
still have issues regarding accurate lip-synching, expressivity,
person-specific information and generalizability. We effectively employ
self-supervised pretrained HuBERT model in the training process that allows us
to incorporate both lexical and non-lexical information in the audio without
using a large lexicon. Additionally, guiding the training with a binary emotion
condition and speaker identity distinguishes the tiniest subtle facial motion.
We carried out extensive objective and subjective evaluation in comparison to
ground-truth and state-of-the-art work. A perceptual user study demonstrates
that our approach produces superior results with respect to the realism of the
animation 78% of the time in comparison to the state-of-the-art. In addition,
our method is 4 times faster eliminating the use of complex sequential models
such as transformers. We strongly recommend watching the supplementary video
before reading the paper. We also provide the implementation and evaluation
codes with a GitHub repository link.

该研究提出了一种基于语音和自监督预训练 HuBERT 模型的 3D 面部动画生成方法，可以捕捉到语音中的身份、情感和犹豫等个性化和微妙的提示，同时具有很强的抗干扰能力和适应性。